This is a task-based IATI data downloader. All downloaded data gets pushed to Amazon S3.
You can run this wherever, but here are some instructions to run on Heroku. It assumes you have a Heroku account, and have the heroku CLI installed.
- Create a new app:
heroku create [app name]
- Create a Redis instance (for managing the task queue)
heroku addons:create redistogo:nano
- add some AWS environment variables (for hosting the files)
heroku config:set AWS_ACCESS_KEY_ID=[AWS access key] heroku config:set AWS_SECRET_ACCESS_KEY=[AWS secret key] heroku config:set S3_BUCKET_NAME=[name of S3 bucket]
- push the app to heroku
git push heroku master
- Then scale the number of workers. The more you do, the faster it is (but the more it costs!)
heroku scale worker=[number of workers]
You can start a registry refresh using:
heroku run python run.py enqueue
You could add this as a cron job, or make it restart on completion so it crawls continuously.
You can check how many tasks are remaining in the queue using:
heroku run python run.py status
You can requeue failed jobs using:
heroku run rq requeue --queue default -u \$REDISTOGO_URL --all