Processing House Disbursements
This repository contains various scripts for manipulating the House's expenditure (disbursement) reports, published at disbursements.house.gov.
Process for parsing disbursements
When new disbursement reports are released, quarterly, download the latest "single volume" PDF for that quarter.
Chop that PDF off at either end so that it contains strictly pages reflecting actual disbursement records.
Extract the text from that PDF, and run it through the Python script in 1_pdf_to_csv/ to generate CSVs.
Take the generated CSVs and run them through the Ruby scripts in 2_add_bioguide_id/ to add a Bioguide ID column to those CSVs. If you get the
Unquoted fields do not allow \r or \nerror, check
bioguide_ids.csvand fix any additions.
Publish them to Sunlight's expenditure reports page. Sunlight staff with appropriate access can publish the CSVs to Amazon S3 with
s3cmd put -P -m text/csv [*.csv] s3://assets.sunlightfoundation.com/expenditures/house/, and update the page through the Django CMS.
Take the detail CSV only and run it through the 4 Ruby scripts in 3_extract_staffers/ to generate new CSVs of various staff records (
We also save various versions of the data along the way in an internal Dropbox account.
The project is in the public domain within the United States, and copyright and related rights in the work worldwide are waived through the CC0 1.0 Universal public domain dedication.
All contributions to this project will be released under the CC0 dedication. By submitting a pull request, you are agreeing to comply with this waiver of copyright interest.