DEPRECATED (We're working on the `disclosure-backend-static` repo instead) A Django app to download, extract and load campaign finance and lobbying activity data from the California Secretary of State's CAL-ACCESS database as well as local Netfile jurisdictions
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Failed to load latest commit information.

This project is deprecated and no longer maintained. We are consolidating efforts into a static generated website approach using disclosure-backend-static and odca-jekyll.

Build Status Coverage Status

California Civic Lab Disclosure Backend


This is the back-end Django application for scraping ballot measure funding data (from NetFile and CalAccess), pushing it to our database, and exposing the data to our front-end apps via a RESTful API.

Helpful links:

  • How to contribute - links to the overall app technical design and status, as well as information on how to contribute ocde.
  • Adding your city - How to test and add your city's disclosure information to this app.
  • Example Dataset - This is a sample Form 460 for Oakland. If you want to dive into the data, please check this out!

See below for server setup.

Mock-ups for data tables that this app intends to support are here:

Setting up the app.

If you've worked with Django and python before, these steps should be familiar to you. We're going to create an environment with Python 2.7.9 for the project

Software Installation

  1. Clone disclosure-backend (or your fork of it) to your own local copy.

  2. Install python or anaconda or conda (If using OSX (Mac) python is already installed, if using Linux, install anaconda, if using Windows, you can choose between conda and python but we recommend python)

  1. Install pip (if using Anaconda, pip is already installed.)

    sudo easy_install pip
  2. Create an environment for this project:

  • For non-Anaconda Python distribution

    sudo pip install virtualenv
    virtualenv env
    source env/bin/activate
  • For Anaconda (we'll make an environment called ODB with Python 2.7.9 in it)

    conda create --name ODB python=2.7.9
    source activate ODB

You will have to activate this environment (or virtualenv) every time you want to start working. You activate using:

source env/bin/activate


source activate ODB
  1. Install mysql and other system dependencies


 brew install pkg-config
 pkg-config --cflags-only-I libcgraph

 brew install mysql
 brew install libssl
 brew install graphviz
  • If brew install libssl does not work don't worry about it.
  • When prompted for a password, remember it because you'll need it.
  1. Install project requirements with:
    pip install -r requirements.txt
    pip install -r requirements_dev.txt
  • Did you encounter an error that says something like fatal error: 'graphviz/cgraph.h' file not found ? Run

    cd /usr/local/Cellar/graphviz/2.38.0/include/graphviz/
    mkdir graphviz
    cp cgraph.h graphviz/

    Then try

    pip install -r requirements_dev.txt

Database setup

  1. Create the database
mysql -u root
mysql> create database opendisclosure;
mysql> create database calaccess_raw;
mysql> \q
  1. Run the server setup script
python setuptestserver

This will run database migrations, add a superuser (username: admin, password: admin), and other setup steps.

OSX: If you get the following error django.core.exceptions.ImproperlyConfigured: Error loading MySQLdb module: dlopen(, 2): Library not loaded: libssl.1.0.0.dylib

Then, you need to add openssl to your DYLD_LIBRARY_PATH:

  1. Go to /usr/local/Cellar/openssl/, and locate your directory (e.g. 1.0.2d_1)
  2. Add the following to your ~/.bash_profile:
export DYLD_LIBRARY_PATH=$DYLD_LIBRARY_PATH:/usr/local/Cellar/openssl/1.0.2d_1/lib

Validate your install

Run the tests

python test

It should load and clean some files in a few seconds.

Note: if this fails with an SSL error and you are using conda/miniconda, use virtualenv instead. See this link for details about the conda issue.

Run with test data

A basic data check to make sure things are working:

python downloadcalaccessrawdata --use-test-data

Download data

Zipcode/metro data

python downloadzipcodedata


Netfile contains campaign finance data for a number of jurisdictions. Not all jurisdictions will have data.

# Download netfile data and load into calaccess_raw.NETFILE_CAL201_TRANSACTION
python downloadnetfilerawdata
# Process NETFILE_CAL201_TRANSACTION into opendisclosure
python xformnetfilerawdata

Run the server

To run for the purposes of development, accessing Django's admin interface:

python runserver

Then go to to log in and see data.

Deploying the app

For deployment to the official website:

ssh /usr/local/bin/deploy-backend