Permalink
Switch branches/tags
Nothing to show
Find file Copy path
Fetching contributors…
Cannot retrieve contributors at this time
28 lines (19 sloc) 1.65 KB

Open States Infrastructure

Open States has grown to be a rather complex project over the years. The purpose of this documentation is to help explain how Open States works end-to-end to better help contributors and project members grasp the entirety of the system.

Overview

A great deal of Open States' infrastructure falls within the billy project. Here's roughly how it works:

  • A scraper is written that utilizes billy.scrape's Python helpers.
  • When invoked via billy-update, this scraper writes JSON files to disk.
  • These JSON files are then imported by billy.importers into MongoDB.
  • A denormalization step takes place allowing us to have aggregate info, billy.reports.

At this point the data has been scraped, validated, post-processed, imported, and we've generated some statistics like how many bills were updated, etc.

From here, the data is served out via a Django project. billy also helps with this, in the form of three applications:

billy.web.public
Essentially the site you see when you browse OpenStates.org.
billy.web.api
Open States API v1
billy.web.admin

This is a custom-built admin, truthfully just a series of views that project maintainers have found useful over time. There are some tools for manual data entry/reconciliation as well as error reporting.

Note: Now that the project is more community-driven and doesn't have a full-time staff, this approach is somewhat outdated and we'll be looking to improve access so that non-staff can help with some of the tasks included in the admin.