Skip to content
Codebase for Zoltar forecast repository
Branch: master
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
.idea
forecast_app
forecast_repo
templates/rest_framework
utils
.env
.gitignore
LICENSE
Pipfile
Pipfile.lock
Procfile
er-diagram.jpg
manage.py
readme.md
truths-2010-through-2018.csv

readme.md

Zoltar forecast archive project

This is a Django project to implement a repository of forecast challenges. See the internal Forecast repository notes document for a more detailed description. The internal reichlab Slack channel for this is #forecast-repository. The GitHub location is https://github.com/reichlab/forecast-repository .

Email-based notification requirements

Zoltar uses Anymail to abstract access to the transactional email server that's used for notifications. (Currently we only have notifications about file uploads.) Anymail can be used for a number of services, as configured in settings. Currently we use SendinBlue.

Configuration: The environment variable SENDINBLUE_API_KEY must be set, e.g., for Heroku:

heroku config:set \
  SENDINBLUE_API_KEY=<YOUR_SENDINBLUE_API_KEY>

AWS S3 configuration

Zoltar uses S3 for temporary storage of uploaded files (forecasts, truth, and templates). You'll need to set three S3-related environment variables, either locally or, for Heroku:

heroku config:set \
  S3_BUCKET_PREFIX=<reichlab_bucket_prefix> \
  AWS_ACCESS_KEY_ID=<YOUR_ACCESS_KEY> \
  AWS_SECRET_ACCESS_KEY=<YOUR_SECRET_KEY>

These keys must enable read, write, and list operations on a bucket named S3_BUCKET_PREFIX + object type in that account. (See cloud_file.py for details re: our bucket naming convention.) For development that account was configured thus:

  • (IAM) Zoltar app user:
    • no groups
    • Permissions > Permissions policies > Attached directly:
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
                "s3:PutObject",
                "s3:GetObject",
                "s3:ListBucket",
                "s3:DeleteObject"
            ],
            "Resource": [
                "arn:aws:s3:::mc.zoltarapp.sandbox",
                "arn:aws:s3:::mc.zoltarapp.sandbox/*"
            ]
        }
    ]
}
  • (S3) Zoltar upload bucket:
    • Permissions > Access control list: default (root)
    • Permissions > Bucket policy: none (controlled above at the user level)

Requirements (see Pipfile)

To install required packages:

$ pipenv --three
$ cd <readme.md's dir>/forecast-repository
$ pipenv install

Pipfile was created via:

pipenv install django
pipenv install click
pipenv install requests
pipenv install jsonfield
pipenv install psycopg2-binary
pipenv install dj-database-url
pipenv install gunicorn
pipenv install whitenoise
pipenv install djangorestframework
pipenv install Pillow
pipenv install pymmwr
pipenv install pyyaml
pipenv install djangorestframework-csv
pipenv install django-debug-toolbar
pipenv install rq
pipenv install django-rq
pipenv install boto3
pipenv install djangorestframework-jwt
pipenv install more-itertools
pipenv install django-anymail[sendgrid,sendinblue]

Utils

The files under utils/ are currently project-specific ones. They should probably be moved.

RQ infrastructure

Zoltar uses an asynchronous messaging queue to support executing long-running tasks outside the web dyno, which keeps the latter responsive and prevents Heroku's 30 second timeouts. We use RQ for this, which requires a Redis server along with one or more worker dynos. Here's the setup to run locally:

  1. Start Redis:
redis-server
  1. Start an rq worker:
cd ~/IdeaProjects/django-redis-play
pipenv shell
export PATH="/Applications/Postgres.app/Contents/Versions/9.6/bin:${PATH}" ; export DJANGO_SETTINGS_MODULE=forecast_repo.settings.local_sqlite3 ; export PYTHONPATH=.
python3 manage.py rqworker
  1. Optionally start monitor (rq info or rqstats):
cd ~/IdeaProjects/django-redis-play
pipenv shell
rq info --interval 1

# alternatively:
export PATH="/Applications/Postgres.app/Contents/Versions/9.6/bin:${PATH}" ; export DJANGO_SETTINGS_MODULE=forecast_repo.settings.local_sqlite3 ; export PYTHONPATH=.
python3 manage.py rqstats --interval 1
  1. Start the web app
cd ~/IdeaProjects/django-redis-play
pipenv shell
export PATH="/Applications/Postgres.app/Contents/Versions/9.6/bin:${PATH}" ; export DJANGO_SETTINGS_MODULE=forecast_repo.settings.local_sqlite3 ; export PYTHONPATH=.
python3 manage.py runserver --settings=forecast_repo.settings.local_sqlite3
  1. Execute the functions that insert onto the queue, e.g.,
cd ~/IdeaProjects/django-redis-play
pipenv shell
export PATH="/Applications/Postgres.app/Contents/Versions/9.6/bin:${PATH}" ; export DJANGO_SETTINGS_MODULE=forecast_repo.settings.local_sqlite3 ; export PYTHONPATH=.
python3 utils/row_count_util.py update
  1. Optionally monitor the progress in the web app

Running the tests

$ cd <readme.md's dir>/forecast-repository
$ pipenv shell
$ cd forecast_app/tests
$ python3 ../../manage.py test --verbosity 2 --settings=forecast_repo.settings.local_sqlite3

Django project layout

This project's settings scheme follows the "split settings.py into separate files in their own 'settings' module" approach. Since we plan on deploying to Heroku, there is no production.py. Regardless, every app needs to set the DJANGO_SETTINGS_MODULE environment variable accordingly, e.g., one of the following:

$ export DJANGO_SETTINGS_MODULE="forecast_repo.settings.local_sqlite3"
$ ./manage.py migrate --settings=forecast_repo.settings.local_sqlite3
$ heroku config:set DJANGO_SETTINGS_MODULE=forecast_repo.settings.local_sqlite3
gunicorn -w 4 -b 127.0.0.1:8001 --settings=forecast_repo.settings.local_sqlite3

Heroku deployment

The site is currently hosted by Heroku at https://reichlab-forecast-repository.herokuapp.com/ . Follow these steps to update it:

login

$ cd ~/IdeaProjects/forecast-repository
$ pipenv shell
$ heroku login

optional: dump local db then copy to remote

$ PGPASSWORD=password
$ pg_dump -Fc --no-acl --no-owner -h localhost -U username forecast_repo > /tmp/mc-1219-forecast_repo.dump
  • upload to somewhere publicly accessible, e.g., Amazon S3
$ heroku pg:backups:restore 'https://s3.us-east-2.amazonaws.com/yourbucket/yourdatabase.dump' DATABASE_URL
$ heroku run python manage.py createsuperuser --settings=forecast_repo.settings.heroku_production

push code

$ git push heroku master

start web dyno

$ heroku ps:scale web=1
$ heroku open
You can’t perform that action at this time.