Website code for https://openprescribing.net - a Django application that provides a REST API and dashboards for the NHS Digitals's GP-level prescribing data and NHS BSA's Detailed Prescribing Information Report.
Information about data sources used on OpenPrescribing can be found here
Set up the application
You can install the application dependencies either on bare metal, or using docker.
the instructions (you need
at least Compose 1.9.0+ and Docker Engine of version 1.12.0+.)
In the project root, run
docker-compose run test
This will pull down the relevant images, and run the tests. In order for all tests to run successfully, you will also need to decrypt the credentials file openprescribing/google-credentials.json.
In our CI system, we also run checks against the production environment, which you can reproduce with
docker-compose run test-production
To open a shell (from where you can run migrations, start a server, etc), run
docker-compose run --service-ports dev
The project code is mounted as a volume within the docker container,
/code/openprescribing. Note that the container runs as the
user, so any files you create from that console will be owned by
The first time you run
docker-compose it creates a persistent volume
for the postgres container. Therefore, if you ever need to change the
database configuration, you'll need to blow away the volume with:
docker-compose stop docker-compose rm -f all
Any time you change the npm or pip dependencies, you should rebuild the docker image used by the tests to improve runtime performance of travis.
# Base docker image, for production docker build -t ebmdatalab/openprescribing-base . # Same as base, but with local-only pip dependencies docker build -t ebmdatalab/openprescribing-test -f Dockerfile-test . docker login # details in `pass`; only have to do this once on your machin # push the images to hub.docker.io docker push ebmdatalab/openprescribing-base docker push ebmdatalab/openprescribing-test
Running the application from within Docker
To be able to access the django instance running inside Docker from outside the container, docker must be told to publish the port on which Django will listen:
docker-compose run --service-ports dev
This will give a shell, at which you can start Django, specifying the
0.0.0.0 interface so that it will accept connections from all IP addesses (not just localhost):
python manage.py runserver_plus 0.0.0.0:8000 --settings=openprescribing.settings.local
The application should then be accessible at
http://localhost:8000/ from a web-
on the host computer.
On bare metal
This should be enough to get a dev sandbox running; some brief notes about production environment follow.
Set up a virtualenv
If you're using virtualenvwrapper:
mkvirtualenv openprescribing cd openprescribing && add2virtualenv `pwd` workon openprescribing
Install library dependencies (current as of Debian Jessie):
sudo apt-get install nodejs binutils libproj-dev gdal-bin libgeoip1 libgeos-c1 git-core vim sudo screen supervisor libpq-dev python-dev python-pip python-virtualenv python-gdal postgis emacs nginx build-essential libssl-dev libffi-dev unattended-upgrades libblas-dev liblapack-dev libatlas-base-dev gfortran libxml2-dev libxslt1-dev
Ensure pip and setuptools are up to date:
pip install -U pip setuptools
Install Python dependencies in development:
pip install -r requirements/local.txt --process-dependency-links
Or in production:
pip install -r requirements.txt --process-dependency-links
cd openprescribing/media/js npm install -g browserify npm install -g jshint npm install -g less npm install
To generate monthly alert emails (and run the tests for those) you'll
phantomjs binary located at
it from here.
Create database and env variables
Set up a Postgres 9.5 database (required for
jsonb type), with
PostGIS extensions, and create a superuser for the database.
createuser -s <myuser> createdb -O <myuser> <dbname> psql -d <dbname> -c "CREATE EXTENSION postgis;"
environment, and set the
DB_* environment variables.
CF_API_KEY for Cloudflare (this is only required for automated deploys, see below).
You will want
MAILGUN_WEBHOOK_PASS if you want to process Mailgun webhook callbacks (see
TRACKING.md) to match the username/password configured in Mailgun. For example, if the webhook is
Keeping environment variables all in one place can be a faff. In
production, we've solved this by adding them all in
/etc/profile.d/openprescribing.sh and then adding
source /etc/profile.d/openprescribing.sh to
The script at
contrib/bin/gunicorn_start is responsible for starting up a backend server. We proxy to this from nginx using a configuration like that at
contrib/nginx/. We control the gunicorn process using
supervisor, with a script like that at
Set up the database
python manage.py migrate
You can copy everything from the production server, if you want, but the full set of prescribing data is enormous. To get a sample of that, you can run something like the following on production (this example grabs 1% of the September 2016 data):
copy ( SELECT * FROM frontend_prescription_201609 TABLESAMPLE SYSTEM (1)) TO '/tmp/sample' WITH binary;
You can then import this data locally using
copy frontend_prescription_201609 from '/tmp/sample' WITH BINARY;
This does require you to have all associated tables (BNF codes, practices etc) up-to-date.
If required, you can run individual Django tests as follows:
python manage.py test frontend.tests.test_api_views
We support IE8 and above. We have a free account for testing across multiple browsers, thanks to BrowserStack.
Note that tests are run using the settings in
openprescribing/settings/test.py; this happens automatically
Functional tests are run using Selenium; the default in your sandbox is to do so via a Firefox driver, so you will need Firefox installed. Running the functional tests will therefore result in a browser being launched.
If you want to run the tests headless (i.e. without launching a browser), you have two choices:
Run the functional tests using Xvbf
To do this, you'll need to install pyvirtualdisplay and Xvbf. This is, apparently, quite hard to do on OS X.
If you don't install Xvbf, you'll see the tests launch a browser and operate it.
You can run just the functional tests with
TEST_SUITE=functional make test
And the inverse is:
TEST_SUITE=nonfunctional make test
Run the functional tests in Saucelabs
In our CI environment we use Saucelabs to run the functional tests in various browsers. If you are connected to the internet, you can run these tests against Saucelabs by installing Sauce Connect, and running:
# Start the Sauce Connect proxy ./sc -u $SAUCE_USERNAME -k $SAUCE_ACCESS_KEY # Run the tests using Saucelabs USE_SAUCELABS=1 BROWSER="firefox:47:Windows 2012" make test
Skip the functional tests
TEST_SUITE=nonfunctional make test
Run the application
python manage.py runserver_plus --settings=openprescribing.settings.local
You should now have a Django application running with no data inside it.
Load the HSCIC data
Check out the
openprescribing-data repo (which contains data for the
app, and scripts to update that data):
git clone firstname.lastname@example.org:ebmdatalab/openprescribing-data.git
Follow the documentation there to import data.
Editing JS and CSS
During development, run the
watch task to see changes appear in the
cd openprescribing/media/js npm run watch
The client-side code makes extensive use of data from the API. To test client-side code against production data, you can set an environment variable to use an API host other than the default:
API_HOST=https://openprescribing.net npm run watch
And run tests with:
npm run test
npm run build
modules array at
Deployment is carried out using
Your public key must be added to
authorized_keys for the
user, and SSH forwarding should work (this possibly means running
ssh-agent add <private-key> on your workstation - see
this helpful debugging guide
if you are having problems.
fab deploy:production will:
- Check if there are any changes to deploy
pipas required (you will need sudo access to do this)
- Update the repo on the server
- Install any new pip and npm dependencies
- Build JS and CSS artefacts
- Run pending migations (only for production environment)
- Reload the server gracefully
- Clear the cloudflare cache
- Log a deploy to
deploy-log.jsonin the deployment directory on the server
You can also deploy to staging:
Or deploy a specific branch to staging:
If the fabfile detects no undeployed changes, it will refuse to run. You can force it to do so (for example, to make it rebuild assets), with:
Or for staging:
Various hooks can be run before committing. Pre-commit hooks are managed by https://github.com/pre-commit/pre-commit-hooks.
To install the hooks, run once:
Details of the hooks are in .pre-commit-config.yaml
This project follows design practices from Two Scoops of Django.