Demo of data retrieval using opendata.swiss API and a standard HTTP Server like Amazon AWS S3 as storage with HEAD request.
- Python 3.x
- pandas
- requests
See requirements.txt for the virtual environment (venv). Recreate the environment with
$ python3 -m venv .venv
$ . .venv/bin/activate
$ pip install -r requirements.txt
Edit harvester.py global variable BASEURL to suit your need. If you encounter issues with the proxy settings, deactivate it.
If you plan to use the local server to test, start a HTTP server where you have your mock API (eidgenossische-wahlen-2023 file) and resources (*.json)
$ python httpserver.py
Start the harvester demo
$ python harvester.py
- Try to update files and see what happens, either locally or on the remote storage.
- Try to change the resource URLs, either locally or on opendata.swiss
The HEAD Request works only on HTTP servers or end points where the Last-Modified header is returned. This won't work with services like the FSO DAM that is a Tomcat web app and returns data directly from a DB without Last-Modified. If you see any link looking like https://dam-api.bfs.admin.ch/hub/api/dam/assets/26965404/master, it won't be applicable