Code and data for myeu.uk - find out what the EU has done for your area
Switch branches/tags
Nothing to show
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Failed to load latest commit information.
data Remove nhs data for release Oct 7, 2018
frontend
pipelines Don't fail task if no officers returned for company Aug 14, 2018
prototype Add first try at NHS staffing data Sep 16, 2018
.editorconfig
.eslintrc Get linting working Aug 12, 2018
.gitattributes Add second pass at England ESIF data Aug 12, 2018
.gitignore
.prettierignore
LICENSE.md Tidy up readme Sep 9, 2018
README.md Add rollbar for error monitoring Oct 7, 2018
package-lock.json
package.json Add license Sep 9, 2018
requirements.txt Add second pass at cordis fp7 data Aug 11, 2018

README.md

My.EU

Data Processing

Requirements

We recommend using Python 3 with anaconda. There are also non-conda dependencies to install with pip:

pip install -r requirements.txt

To avoid committing the output of IPython notebooks, which can be very large and make diffs hard to read, please install nbstripout as a git filter.

There is a Makefile in the data directory that handles running the IPython notebooks and updating the appropriate output files.

cd data
make

Input Datasets

Most datasets are included in this repo, but some are quite large, so they must be downloaded separately, as follows.

You only need to download these if you want to work on the relevant processing; the data needed for development on the app are in the data/*/output folders, especially data/map/output.

CAP

From these sources:

Files must be stored here:

  • data/cap/input/2014_All_CAP_Search_Results_Data_P14_1.xls
  • data/cap/input/2015/DAERA-Table 1.csv
  • data/cap/input/2015/RPA-Table 1.csv
  • data/cap/input/2015/RPA2-Table 1.csv
  • data/cap/input/2015/SGRPID-Table 1.csv
  • data/cap/input/2015/WG-Table 1.csv
  • data/cap/input/2016_All_CAP_Search_Results_Data_P14.xlsx
  • data/cap/input/2017_All_CAP_Search_Results_Data_P14.xlsx

CORDIS

From these sources:

Files must be stored here:

  • data/cordis/input/fp7/cordis-fp7briefs.xlsx
  • data/cordis/input/fp7/cordis-fp7organizations.xlsx
  • data/cordis/input/fp7/cordis-fp7projects.xlsx
  • data/cordis/input/h2020/cordis-h2020organizations.xlsx
  • data/cordis/input/h2020/cordis-h2020projects.xlsx

Erasmus

From this page:

Files must be stored here:

  • data/erasmus/input/CoopIndustrialisedCountries_Projects_Overview_2018-09-10.xls
  • data/erasmus/input/ErasmusMundus_Projects_Overview_2018-09-10.xls
  • data/erasmus/input/ErasmusPlus_JeanMonnet_Projects_Overview_2018-09-10.xls
  • data/erasmus/input/ErasmusPlus_KA1_2014_LearningMobilityOfIndividuals_Projects_Overview_2018-09-11.xls
  • data/erasmus/input/ErasmusPlus_KA1_2015_LearningMobilityOfIndividuals_Projects_Overview_2018-09-11.xls
  • data/erasmus/input/ErasmusPlus_KA1_2016_LearningMobilityOfIndividuals_Projects_Overview_2018-09-11.xls
  • data/erasmus/input/ErasmusPlus_KA1_2017_LearningMobilityOfIndividuals_Projects_Overview_2018-09-10.xls
  • data/erasmus/input/ErasmusPlus_KA1_2018_LearningMobilityOfIndividuals_Projects_Overview_2018-09-10.xls
  • data/erasmus/input/ErasmusPlus_KA2_CooperationForInnovationAndTheExchangeOfGoodPractices_Projects_Overview_2018-09-10.xls
  • data/erasmus/input/ErasmusPlus_KA3_SupportForPolicyReform_Projects_Overview_2018-09-10.xls
  • data/erasmus/input/ErasmusPlus_Sports_Projects_Overview_2018-09-10.xls
  • data/erasmus/input/LifeLongLearning_Projects_Overview_2018-09-10.xls
  • data/erasmus/input/Sports_Projects_Overview_2018-09-10.xls
  • data/erasmus/input/Tempus_Projects_Overview_2018-09-10.xls
  • data/erasmus/input/YouthInAction_Projects_Overview_2018-09-10.xls

App Development

The latest application is in the frontend directory. It is a static website built with webpack.

Requirements

Install node js version 8.11 or higher.

The application uses Google Maps, so you will need to create a Google Cloud Platform account for development. This requires putting in a credit card, but development usage will generally be in the free tier.

  1. Go to the API list and ensure that both the Google Maps JavaScript API and the Places API are enabled.

  2. Go to the Credentials list and generate an API key for development.

Environment

For development, you need to put your development Google Maps API key in an environment variable:

DEVELOPMENT_MY_EU_API_KEY=YourKeyHere

(To deploy to production, you need a PRODUCTION_MY_EU_API_KEY variable set to a production API key; see notes on deployment below.)

Running the Application

cd frontend
npm install
npm run dev

This will start the application using the webpack-serve development server on http://localhost:8080. Changes to the application's source files in frontend/src will in most cases automatically reload the page.

You can also preview the as it will be built in production (but with a development Google Maps API key, so the map will load) with

cd frontend
PRODUCTION_MY_EU_API_KEY=$DEVELOPMENT_MY_EU_API_KEY npm run build
npm run serve
rm -rf dist # don't leak your development key!

Code Style

There is an editorconfig in the root that defines encoding, tabs vs spaces, etc.. Please use it. The best way to do this is to get an editorconfig plugin for your editor.

The prettier code formatter is set up to format the JavaScript code consistently. Please use it. The best way to do it is to get a plugin for your editor, but you can also run

npm install # in the repo root
npm run prettier

to format all the code.

There is also eslint for JavaScript linting. Again, the best thing to do is turn on the plugin for your editor, but you can also run

npm run eslint

to check all the code.

Deployment

The static website is deployed to Google Cloud Storage behind CloudFlare for SSL. To deploy:

  1. You need to be granted write access to the Google Cloud Storage bucket that hosts the website.

  2. You need to get the PRODUCTION_MY_EU_API_KEY, which restricts the referrer to www.myeu.uk (and does not allow localhost or other domains).

  3. To deploy with error monitoring, you need to get the ROLLBAR_ACCESS_TOKEN (post_client_item) and ROLLBAR_DEPLOY_ACCESS_TOKEN (post_server_item) keys for our rollbar project.

Then run the deploy script to deploy:

cd frontend
./deploy.sh

License

Please see LICENSE.md.