Skip to content
A scrape of data from the UK Engineering and Physical Sciences Research Council "Grants on the Web" service.
Python JavaScript Shell
Find file
Pull request Compare This branch is 6 commits ahead of nickstenning:master.
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Failed to load latest commit information.
epsrc
schema
.gitignore
README.mkd
export.sql
migrate.sh
openspending-model.js
setup.py

README.mkd

EPSRC Grant portfolio

As of 2012-05-28, the EPSRC "Grants on the Web" database contains grants dating back to 1985:

  • 43,317 grants
  • £10,935,828,250 of committed funding

This repository contains tools for scraping and manipulating this data into a useful format.

Quick access

Instead of using the code in this repository, why not download the database generated by it? A listing of these can be found at CKAN.

Setup

To get up and running, you will need python and pip. Instructions for installing python are beyond the scope of this document, but you should be able to get pip if you don't have it by running:

$ easy_install pip

If you use virtualenv, you might want to create an environment to play around in:

$ virtualenv --distribute pyenv
$ source pyenv/bin/activate

Now install the scraper and its dependencies:

$ pip install -e .

Create the database (or migrate your existing database to the current schema version) by running:

$ ./migrate.sh epsrc.db 4

Usage

This scraper operates on a local mirror of the EPSRC GOW website. You must first make this mirror (this will take a long time to complete, often several days):

$ wget -c -m gow.epsrc.ac.uk

Then you can use the epsrc-scrape tool to parse this data:

$ epsrc-scrape epsrc.db ./gow.epsrc.ac.uk

Each grant has a unique grant reference, e.g. GR/J50118/01. We can check that the example grant above made it into the database with the following command

$ sqlite3 -line epsrc.db "select * from grants where id='GR/J50118/01'"
                       id = GR/J50118/01
                    title = USE OF CATALYTIC MEMBRANES FOR IN SITU REACTION AND SEPARATION
                    value = 114608
principal_investigator_id = 40792
            department_id = 5273
               created_at = 2011-06-16 12:36:35.164790
              modified_at = 2011-06-16 12:36:35.164800

Warnings

Be careful with the data this scraper generates. Care has been taken to sanitize the data, but as with any scraper, there may be inconsistencies or missing entries. Please let me know if you find any problems.

Something went wrong with that request. Please try again.