Harvester is an open-source application that downloads data from various public sources and pushes data to ekosystem.slovensko.digital.
Supported data sources
Currently, Harvester supports the following data-sources:
Currently, there are data sources in ekosystem.slovensko.digital which are not ingested by Harvester. The corresponding code hasn't been open-sourced yet.
Harvester is a Ruby-on-Rails application. However, there is no web process present. The application runs as a backend worker/scheduler and uses the Rails framework to enforce coding rules for the project.
To get the Harvester dependencies running (db, etc),
we have prepared a Docker Compose file for you.
You can start the dependencies with
To run and test the application while developing:
- Prepare the environment with
- Ensure that all tests pass:
- Start the worker with
foreman start worker
- Run a selected job with rake
- Check that the corresponding database tables have been filled in
Feel free to contribute other datasets / scrapers or improvements. See issues if you want to help, request a feature or file a bug. Join us on our public slack channel if you want to talk live.