Open Elections core code

Core repo for election results data acquisition, transformation and output.

OpenElections core is developed and tested using Python 2.7.*. The package might not work with older or newer Python distributions.

Getting started as a developer

You'll:

• set up a virtual environment
• fork/clone this repository, install dependencies
• add any optional configuration details you need (e.g. Mongo or AWS)

Setting up a virtual environment

You should use virtualenv and virtualenvwrapper to work on Open Elections inside a virtualized development environment.

The easiest way is to install these tools system-wide with pip (you may need to use sudo):

$ pip install virtualenv
$ pip install virtualenvwrapper

Then, to make a virtual environment called openelex for open elections work:

$ mkvirtualenv openelex

This will automatically activate the openelex environment. To turn it on for future sessions:

$ workon openelex

Fork and set up this project

Fork this repo by hitting the "Fork" button above, and clone your fork to your computer:

$ git clone git@github.com:[my_github_user]/core.git openelex-core
$ cd openelex-core

Turn on your virtual environment from the previous step, if you haven't already:

$ workon openelex

Then install the Python dependencies:

$ pip install -r requirements.txt
$ pip install -r requirements-dev.txt

Create settings.py from the template

$ cp settings.py.tmplt openelex/settings.py

Setting up 'invoke'

OpenElections uses invoke to run tasks (similar to Ruby's rake).

First, make sure you're in the root of the repository you've cloned.

Add the openelex directory to your $PYTHONPATH, so that invoke can see our tasks. This will append to your shell's login script (replace .bashrc with whatever your shell uses, if needed).

echo "export PYTHONPATH=$PYTHONPATH:`pwd`/openelex" >> ~/.bashrc

That will run automatically for future terminal sessions. To activate it for the current session:

source ~/.bashrc

All invoke commands must be run from the project root.

Test it out by running invoke --list, you should see something like:

$ invoke --list
Available tasks:

    fetch
    archive.delete
    archive.save
    cache.clear
    cache.files
    datasource.elections
    datasource.filename_url_pairs
    datasource.mappings
    datasource.target_urls
    load.run
    transform.list
    transform.run
    validate.list
    validate.run

Running commands looks something like this:

$ invoke cache.clear --state=NY
0 files deleted
0 files still in cache

You can also get help on particular commands, e.g. invoke --help cache.clear.

Configuring services (optional)

openelex/settings.py can be configured for MongoDB and AWS. You only need to set these configs if you plan to archive files on your own S3 account, or write data loaders.

To configure S3 to cache your raw results, update these values in openelex/settings.py:

AWS_ACCESS_KEY_ID = ''
AWS_SECRET_ACCESS_KEY =''

Install MongoDB (optional)

You only need to install MongoDB if you plan to write data loaders.

To store your data in MongoDB, you need only install Mongo. The default configuration should auto-create the databases and tables you need, as you need them.

Load party and office metadata (optional)

You only need to do this if you plan to write data loaders or transforms.

$ cd openelex
$ invoke load_metadata.run --collection=office
$ invoke load_metadata.run --collection=party