elidickinson / fiftystates forked from sunlightlabs/fiftystates

download and parse legislation from all 50 state legislatures

This URL has Read+Write access

name age
history
message
file .gitignore Mon Apr 13 02:45:14 -0700 2009 add cache dir to gitignore Signed-off-by: mind... [mikejs]
file LICENSING Fri Feb 27 16:15:01 -0800 2009 updated readme [jamesturk]
file README.rst Thu Apr 09 08:19:28 -0700 2009 Added 'mechanize' dependency to rbutils and upd... [Mr Man]
directory data/ Fri Mar 27 15:42:55 -0700 2009 more documentation updates. document how to us... [Caged]
directory scripts/ Loading commit data...
README.rst

About the Project

The goal here is to build scrapers and parsers in order to get as much state legislative data as possible in one place.

For details on the reasons for the project and goals behind the project see the project announcement.

For an editable overview of each state's progress visit the Sunlight Labs Wiki.

Project Goals

  1. Collect URLs of State Legislature and Legislative Information Pages [done]

  2. Grab legislators and legislation
    1. Build scrapers and obtain data files for legislation in each of the fifty states
    2. create sponsor relationship between legislators and legislation
  3. Grab votes
    1. Build scrapers and obtain data files for legislator votes on legislation
    2. create voting relationship between legislators and legislation

  4. Build tools on top of data

To encourage as many contributions as possible we aren't saying "write in Python" or anything, but we do need the code to follow a few guidelines.

For details on how scripts should be written and how they should run see scripts/README.rst. For details on how data should be stored see data/README.rst.

Usage (proposed)

Valid options:
--year: a year or years the parser should attempt --all: Attempt to parse years from 1969-2009 --upper: Parse upper chamber --lower: Parse lower chamber
The vision is that the flow will look something like this:
$ ./scripts/nc/get_legislation --year=2009 --upper

Contributing

If you are interested in contributing the recommended procedure is to check on the Sunlight Labs Wiki and in the repository to see where your state is. The next step is generally to announce your interest on the Sunlight Labs Mailing List (this is where you can ask questions and make suggestions regarding the project).

Managing a State

Once you have claimed a state on the wiki and mailing list you should probably maintain your own fork of the project on github.

Please avoid making changes to files in other states/etc. on your state branch. Stick to editing files in the scripts/your_state directory and where necessary in any relevant utils directories.

Whenever your state script works as it should announce it on the mailing list and someone will merge your changes into the core.

Licensing

We feel that in order to protect everyone's intellectual property it makes the most sense to keep the code under the AGPL license. In a nutshell this means that you are not permitted to make changes to this source without releasing your code as well, this is intended to prevent abuse but we are sensitive to license choice on a project of this scale so if you have strong objections please raise them on the mailing list. See LICENSING for the full terms of the AGPLv3.

Requirements

Python (2.4+)

  • BeautifulSoup

Ruby (1.8.*+)

  • hpricot (gem install hpricot)
  • fastercsv (gem install fastercsv)
  • mechanize (gem install mechanize)

A note on new requirements

Because there are potentially fifty plus contributors to this project, a real effort should be made to keep the requirements of running the full suite to a minimum.

In other words, if you can write your parser in language X and Y, choose language X if language X is already a requirement. If you are using a language (say Python) and you need to parse HTML and there is already an HTML parsing library (say BeautifulSoup) favor that over some other library (unless absolutely necessary).