Join GitHub today
GitHub is home to over 20 million developers working together to host and review code, manage projects, and build software together.
Fetching latest commit…
Cannot retrieve the latest commit at this time.
These are the Perl screen-scrapers that build the GovTrack.us legislative database. The files are licensed under the terms of the GNU AGPL, and I am very serious about the terms of the license. You may not modify these files without sharing your modifications. ------------------------------------------------------------- Getting The Source Code ----------------------- The scripts will only work on a Linux-like computer. Check out the source code into a directory called "gather": svn co svn://occams.info/govtrack/gather/us gather Setting Up Perl --------------- To use these scripts, you'll need a ton of Perl modules installed. You can look at the use directives to see which. A quick way to install them all would be to run as root: cpan `grep "^use" * |sed -e "s/.*:use \(\S*\)[; ].*/\\1/"|sort|uniq` You'll need the (non-Perl) GD image library development files installed first. Some other modules may still need to be installed (DateTime::Locale, DBD::mysql, perhaps others). Running the script (see below) will tell you if it is missing any dependancies, which can then be installed with cpan. If you are using Cygwin, see here http://search.cpan.org/dist/DBD-mysql/lib/DBD/mysql/INSTALL.pod#Windows/CygWin http://mail.python.org/pipermail/python-dev/2003-July/036932.html Setting Up A Database --------------------- You'll also need to set up access to a database. You can either run a MySQL database locally or access the GovTrack database remotely. In the latter case, you won't be able to do any write operations to the database, which means some things won't work, and it will also be considerably slower. To set up a local MySQL database: create a database named "govtrack" set its default encoding to utf-8 give the "govtrack@localhost" user access to it, with no password, or modify db.pl with the log in information load in the MySQL dumps into the database: mysql govtrack < database.tables.sql mysql govtrack < database.tables2.sql mysql govtrack < database.people.sql To access the GovTrack database remotely: Set the environment variable REMOTE_DB to 1 when running any script. (i.e. prefix any command on the command line with "REMOTE_DB=1 ".) Before Running The Scripts -------------------------- The scripts are all run from within the "gather" directory, but they expect that a "../data/us" directory exists. So you should create a "data" directory along side the "gather" directory. If you're also running a local instance of the GovTrack website (the "front-end"), you can put the gather directory inside the "www" directory so you reuse the website's data directory. Or symlink www/data to a "data" directory next to "gather". They also expect a "../mirror" directory (i.e. a mirror directory along side the gather directory) which some scripts will use to store files downloaded. (It stores it in files whose names are MD5 hashes of the URLs where the file came from.) The scripts won't use the mirrored (i.e. cached) files unless CACHED=1 is set in the environment. Running the Scripts ------------------- Most of the scripts are set up so you can call them from the command line, and also so they can be require'd by another script. When running from the command line, the first argument is generally the name of an action to take. The rest of the arguments depend on the command. Here are some examples: REMOTE_DB=1 perl parse_status.pl PARSE_STATUS 110 s 1 This fetches the status of bill S. 1 in the 110th Congress and some related information. The bill types are: h, s, hc, sc, hj, sj, hr, and sr. It writes out: ../data/us/110/bills/s1.xml The bill status file. ../data/us/110/bills.summary/s1.summary.xml The CRS summary (structured parse). ../data/us/110/rolls/s2007-19.xml (etc.) Votes related to the bill. ../data/us/110/bills.amdt/s3.xml (etc.) Amendments to this bill. This happens to be the Senate's 3rd 'amendment' in 2007. The fact that it amends S. 1 is encoded within the file. ../mirror/(various) Caches of remote pages that have been fetched, which can be used in place of future remote fetches by setting CACHED=1 in the environment (using bash, just put that at the very beginning like with REMOTE_DB=1). The formats of the XML files are somewhat documented here: http://wiki.govtrack.us/index.php/Data_Directory