A Django app providing a REST API and dashboards for the HSCIC's GP prescribing data
Python JavaScript HTML CSS Shell PLSQL
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Failed to load latest commit information.
contrib Add some missing setup documentation Nov 29, 2016
deploy Redirect clean_up_bq_test_data logs Jun 13, 2018
openprescribing Merge pull request #930 from ebmdatalab/remove-august-analyse-limit Jul 13, 2018
requirements Make spending types an Enum Jun 7, 2018
scripts Remove debugging stuff from docker setup Jun 18, 2018
.codeclimate.yml Merge branch 'master' into better-js-handling Jun 18, 2018
.csslintrc ClodeClimate config files Jun 20, 2016
.dockerignore Support running tests in docker May 19, 2016
.eslintignore ClodeClimate config files Jun 20, 2016
.eslintrc.json Rename eslintrc to follow convention Dec 13, 2017
.gitignore Merge pull request #835 from ebmdatalab/add-data-flow-visualisation May 17, 2018
.pre-commit-config.yaml Add some more pre-commit hooks Dec 5, 2017
.travis.yml Pin the version of coveralls we use Jul 16, 2018
CONTRIBUTING.md Update CONTRIBUTING.md Mar 6, 2017
Dockerfile Upgrade node to fix npm install errors Jun 18, 2018
Dockerfile-test Update docker environment for local tests Mar 21, 2018
LICENSE.txt Initial commit Sep 2, 2015
README.md Merge pull request #689 from ebmdatalab/better-js-handling Jun 18, 2018
TRACKING.md Remove half-finished sentence Nov 9, 2016
docker-compose.yml Attempt to integrate Coveralls for test coverage Jul 12, 2018
environment-docker Add environment-docker Apr 18, 2018
environment-sample Move mailgun key to environment May 9, 2018
environment-test Move mailgun key to environment May 9, 2018
fabfile.py Don't try to copy crontab when deploying staging Jun 12, 2018
google-credentials.json.enc Use bigquery for measure calculations Sep 27, 2016
requirements.txt Initial commit Sep 2, 2015

README.md

Build Status Code Climate

Open Prescribing

Website code for https://openprescribing.net - a Django application that provides a REST API and dashboards for the NHS Digitals's GP-level prescribing data and NHS BSA's Detailed Prescribing Information Report.

Information about data sources used on OpenPrescribing can be found here

Set up the application

You can install the application dependencies either on bare metal, or using docker.

Using docker

Install docker and docker-compose per the instructions (you need at least Compose 1.9.0+ and Docker Engine of version 1.12.0+.)

In the project root, run

docker-compose run test

This will pull down the relevant images, and run the tests. In order for all tests to run successfully, you will also need to decrypt the credentials file openprescribing/google-credentials.json.

In our CI system, we also run checks against the production environment, which you can reproduce with

docker-compose run test-production

To open a shell (from where you can run migrations, start a server, etc), run

docker-compose run --service-ports dev

The project code is mounted as a volume within the docker container, at /code/openprescribing. Note that the container runs as the root user, so any files you create from that console will be owned by root.

The first time you run docker-compose it creates a persistent volume for the postgres container. Therefore, if you ever need to change the database configuration, you'll need to blow away the volume with:

docker-compose stop
docker-compose rm -f all

Any time you change the npm or pip dependencies, you should rebuild the docker image used by the tests to improve runtime performance of travis.

# Base docker image, for production
docker build -t ebmdatalab/openprescribing-base .
# Same as base, but with local-only pip dependencies
docker build -t ebmdatalab/openprescribing-test -f Dockerfile-test .
docker login  # details in `pass`; only have to do this once on your machin
# push the images to hub.docker.io
docker push ebmdatalab/openprescribing-base
docker push ebmdatalab/openprescribing-test

Running the application from within Docker

To be able to access the django instance running inside Docker from outside the container, docker must be told to publish the port on which Django will listen:

docker-compose run --service-ports dev

This will give a shell, at which you can start Django, specifying the 0.0.0.0 interface so that it will accept connections from all IP addesses (not just localhost):

 python manage.py runserver_plus 0.0.0.0:8000 --settings=openprescribing.settings.local

The application should then be accessible at http://localhost:8000/ from a web- on the host computer.

On bare metal

This should be enough to get a dev sandbox running; some brief notes about production environment follow.

Set up a virtualenv

If you're using virtualenvwrapper:

mkvirtualenv openprescribing
cd openprescribing && add2virtualenv `pwd`
workon openprescribing

Install dependencies

Install library dependencies (current as of Debian Jessie):

sudo apt-get install nodejs binutils libproj-dev gdal-bin libgeoip1 libgeos-c1 git-core vim sudo screen supervisor libpq-dev python-dev python-pip python-virtualenv python-gdal postgis emacs nginx build-essential libssl-dev libffi-dev unattended-upgrades libblas-dev liblapack-dev libatlas-base-dev gfortran libxml2-dev libxslt1-dev

Ensure pip and setuptools are up to date:

pip install -U pip setuptools

Install Python dependencies in development:

pip install -r requirements/local.txt --process-dependency-links

Or in production:

pip install -r requirements.txt --process-dependency-links

And then install JavaScript dependencies. Make sure you have the latest version of nodejs:

cd openprescribing/media/js
npm install -g browserify
npm install -g jshint
npm install -g less
npm install

To generate monthly alert emails (and run the tests for those) you'll need a phantomjs binary located at /usr/local/bin/phantomjs. Get it from here.

Create database and env variables

Set up a Postgres 9.5 database (required for jsonb type), with PostGIS extensions, and create a superuser for the database.

createuser -s <myuser>
createdb -O <myuser> <dbname>
psql -d <dbname> -c "CREATE EXTENSION postgis;"

Copy environment-sample to environment, and set the DB_* environment variables.

Set the CF_API_EMAIL and CF_API_KEY for Cloudflare (this is only required for automated deploys, see below).

You will want MAILGUN_WEBHOOK_USER and MAILGUN_WEBHOOK_PASS if you want to process Mailgun webhook callbacks (see TRACKING.md) to match the username/password configured in Mailgun. For example, if the webhook is

http://bobby:123@openprescribing.net/anymail/mailgun/tracking/

Then set MAILGUN_WEBHOOK_USER to bobby and MAILGUN_WEBHOOK_PASS to 123.

Production notes

Keeping environment variables all in one place can be a faff. In production, we've solved this by adding them all in /etc/profile.d/openprescribing.sh and then adding source /etc/profile.d/openprescribing.sh to <virtualenv>/bin/activate.

The script at contrib/bin/gunicorn_start is responsible for starting up a backend server. We proxy to this from nginx using a configuration like that at contrib/nginx/. We control the gunicorn process using supervisor, with a script like that at contrib/supervisor/.

Set up the database

Run migrations:

python manage.py migrate

Sandbox data

You can copy everything from the production server, if you want, but the full set of prescribing data is enormous. To get a sample of that, you can run something like the following on production (this example grabs 1% of the September 2016 data):

copy (
SELECT
  *
FROM
  frontend_prescription_201609 TABLESAMPLE SYSTEM (1)) TO '/tmp/sample' WITH binary;

You can then import this data locally using pg_restore:

copy frontend_prescription_201609 from '/tmp/sample' WITH BINARY;

This does require you to have all associated tables (BNF codes, practices etc) up-to-date.

Run tests

Run Django and JavaScript tests:

make test

If required, you can run individual Django tests as follows:

python manage.py test frontend.tests.test_api_views

We support IE8 and above. We have a free account for testing across multiple browsers, thanks to BrowserStack.

image

Note that tests are run using the settings in openprescribing/settings/test.py; this happens automatically

Functional tests

Functional tests are run using Selenium; the default in your sandbox is to do so via a Firefox driver, so you will need Firefox installed. Running the functional tests will therefore result in a browser being launched.

If you want to run the tests headless (i.e. without launching a browser), you have two choices:

Run the functional tests using Xvbf

To do this, you'll need to install pyvirtualdisplay and Xvbf. This is, apparently, quite hard to do on OS X.

If you don't install Xvbf, you'll see the tests launch a browser and operate it.

You can run just the functional tests with

TEST_SUITE=functional make test

And the inverse is:

TEST_SUITE=nonfunctional make test

Run the functional tests in Saucelabs

In our CI environment we use Saucelabs to run the functional tests in various browsers. If you are connected to the internet, you can run these tests against Saucelabs by installing Sauce Connect, and running:

# Start the Sauce Connect proxy
./sc -u $SAUCE_USERNAME -k $SAUCE_ACCESS_KEY

# Run the tests using Saucelabs
USE_SAUCELABS=1 BROWSER="firefox:47:Windows 2012" make test

You can work out the browser string to use for your desired platform / browser combination using this Saucelabs tool. You can find the combinations we use for our Travis CI in .travis.yml.

Skip the functional tests

TEST_SUITE=nonfunctional make test

Run the application

python manage.py runserver_plus --settings=openprescribing.settings.local

You should now have a Django application running with no data inside it.

Load the HSCIC data

Check out the openprescribing-data repo (which contains data for the app, and scripts to update that data):

git clone git@github.com:ebmdatalab/openprescribing-data.git

Follow the documentation there to import data.

Editing JS and CSS

Source JavaScript is in /media, compiled JavaScript is in /static.

During development, run the watch task to see changes appear in the compiled JavaScript and CSS.

cd openprescribing/media/js
npm run watch

The client-side code makes extensive use of data from the API. To test client-side code against production data, you can set an environment variable to use an API host other than the default:

API_HOST=https://openprescribing.net npm run watch

And run tests with:

npm run test

This build task generates production-ready minified JavaScript, CSS etc, and is executed as part of the fabric deploy process:

npm run build

If you add new javascript source files, update the modules array at media/js/build.js.

Deployment

Deployment is carried out using fabric.

Your public key must be added to authorized_keys for the hello user, and SSH forwarding should work (this possibly means running ssh-agent add <private-key> on your workstation - see this helpful debugging guide if you are having problems.

Running fab deploy:production will:

  • Check if there are any changes to deploy
  • Install npm and pip as required (you will need sudo access to do this)
  • Update the repo on the server
  • Install any new pip and npm dependencies
  • Build JS and CSS artefacts
  • Run pending migations (only for production environment)
  • Reload the server gracefully
  • Clear the cloudflare cache
  • Log a deploy to deploy-log.json in the deployment directory on the server

You can also deploy to staging:

fab deploy:staging

Or deploy a specific branch to staging:

fab deploy:staging,branch=my_amazing_branch

If the fabfile detects no undeployed changes, it will refuse to run. You can force it to do so (for example, to make it rebuild assets), with:

fab deploy:production,force_build=true

Or for staging:

fab deploy:staging,force_build=true,branch=deployment

Development

Various hooks can be run before committing. Pre-commit hooks are managed by https://github.com/pre-commit/pre-commit-hooks.

To install the hooks, run once:

pre-commit install

Details of the hooks are in .pre-commit-config.yaml

Philosophy

This project follows design practices from Two Scoops of Django.