This repository has been archived by the owner on Mar 6, 2019. It is now read-only.
/
params.json
1 lines (1 loc) · 15.8 KB
/
params.json
1
{"name":"Fech","tagline":"A Ruby parser for electronic candidate, PAC and party campaign filings from the Federal Election Commission.","body":"### Fech\r\n\r\nFech is a Ruby library for downloading and parsing electronic campaign finance filings from the [Federal Election Commission](http://www.fec.gov/). It supports filings from presidential and U.S. House candidates, F.E.C.-registered political action committees and most party committees (everything except U.S. Senate candidates and two senatorial party committees). Fech currently commits to supporting the [F.E.C.'s filing formats](http://www.fec.gov/elecfil/eFilingFormats.zip) back to version 3.0, where applicable.\r\n\r\nFech is tested under Ruby versions 1.8.7, 1.9.2, 1.9.3 and 2.0.0. The latest version is 1.6.0. Please consult the CHANGELOG for details on the latest release.\r\n\r\n***Note***: as of January 1, 2014, only Fech version 1.5.2 or higher will properly retrieve electronic filings. Please update your applications accordingly.\r\n\r\nBug reports and feature requests are welcomed by submitting an issue. Please be advised that development is focused on parsing filings, which may contain errors or be improperly filed, and not on providing wrappers or helper methods for working with filings in another context. To get started, fork the repo, make your additions or changes and send a pull request.\r\n\r\nFech is an open source project of [The New York Times](http://open.nytimes.com/). \r\n\r\n## Fech in Action<a name=\"action\"/>\r\n\r\n* [Political Consultants Rake It In, $466 Million And Counting In 2012 Cycle](http://www.huffingtonpost.com/2012/06/05/political-consultants-2012-campaign-big-money_n_1570157.html), The Huffington Post.\r\n\r\n## Pronunication guide\r\nIt's \"fetch\", with a soft \"ch\" sound. There are no other acceptable pronunciations.\r\n\r\n## Installation<a name=\"install\"/>\r\n\r\nInstall Fech as a gem:\r\n\r\n```ruby\r\ngem install fech\r\n``` \r\nFor use in a Rails 3 application, put the following in your Gemfile:\r\n\r\n```ruby\r\ngem 'fech'\r\n```\r\n\r\nthen issue the `bundle install` command.\r\n\r\n## Getting Started<a name=\"started\"/>\r\n\r\nStart by creating a Filing object that corresponds to an electronic filing from the FEC, using the unique numeric identifier that the F.E.C. assigns to each filing. You'll then have to download the file before parsing it:\r\n\r\n```ruby\r\nfiling = Fech::Filing.new(723604)\r\nfiling.download\r\n```\r\n\r\nOptional arguments: pass in a `:quote_char` argument, and Fech will use it as a quote character to parse the file (see details below). Specify the `:download_dir` on initialization to set where filings are stored. Otherwise, they'll go into a temp folder on your filesystem.\r\n\r\n### CSV Parsers<a name=\"csv_parsers\"/>\r\n\r\nNot all CSV Parsers are created equal. Some are fast and naive, others are slow but handle malformed data more reliably. By default, FasterCSV is used, but you can pass in your own choice of parser with the `:csv_parser` option:\r\n\r\n```ruby\r\n filing = Fech::Filing.new(723604, :csv_parser => Fech::CsvDoctor)\r\n```\r\n\r\nFech::CsvDoctor inherits from FasterCSV, and helps deal with certain types of malformed quotes in the data (see: Handling malformed filings). You could pass in any subclass with special handling.\r\n\r\n### Summary Data<a name=\"summary\"/>\r\n\r\nTo get summary data for the filing (various aggregate amounts, stats about the filing):\r\n\r\n```ruby\r\n filing.summary\r\n => {:coverage_from_date=>\"20110101\", :coverage_from_date=>\"20110301\", ... }\r\n```\r\n\r\nReturns a named hash of all attributes available for the filing. (Which fields are available can vary depending on the filing version number.)\r\n\r\n### Accessing specific line items<a name=\"line_items\"/>\r\n\r\nTo grab every row in the filing of a certain type (all Schedule A items, for example):\r\n\r\n```ruby\r\n filing.rows_like(/^sa/)\r\n => [{:transaction_id=>\"SA17.XXXX\", :contribution_date>\"20110101\" ... } ... ]\r\n```\r\n\r\nThis will return an array of hashes, one hash for each row found in the filing whose line number matches the regular expression you passed in. (SA17s and SA18s would both be returned in this example). You can also pass in strings for exact matches.\r\n\r\nWhen given a block, .rows_like will yield a single hash at a time:\r\n\r\n```ruby\r\n filing.rows_like(/^sa/) do |contribution|\r\n contribution.transaction_id\r\n => {:transaction_id=>\"SA17.XXXX\", :contribution_date>\"20110101\" ... }\r\n end\r\n```\r\n\r\nSince .rows_like loads all transactions matching the corresponding line number into memory, for very large filings (for example, for presidential campaigns) you may want to use .enum_for instead:\r\n\r\n```ruby\r\n filing.enum_for((:rows_like, /sa/)\r\n => [{:transaction_id=>\"SA17.XXXX\", :contribution_date>\"20110101\" ... } ... ]\r\n```\r\n\r\n## Usage<a name=\"usage\"/>\r\n\r\n### Accessing specific fields\r\n\r\nBy default, `.rows_like` will process every field in the matched rows (some rows have 200+ fields). You can speed up performance significantly by asking for just the subset of fields you need.\r\n\r\n```ruby\r\n filing.rows_like(/^sa/, :include => [:transaction_id]) do |contribution|\r\n contribution\r\n => {:transaction_id=>\"SA17.XXXX\"}\r\n end\r\n```\r\n\r\n### Miscellaneous Text<a name=\"misc_text\"/>\r\n\r\nForm 99 submissions contain no structured data but miscellaneous text communicated between committees and the F.E.C. The text is accessible via the summary:\r\n\r\n```ruby\r\n filing.summary[:text]\r\n => \"This is in response to Requests for Additional Information received on 9 and 10 February 2012....\"\r\n```\r\n\r\n### Raw data<a name=\"raw_data\"/>\r\n\r\nIf you want to access the raw arrays of row data, pass `:raw => true` to `.rows_like` or any of its shortcuts:\r\n\r\n```ruby\r\n filing.contributions(:raw => true)\r\n => [[\"SA17A\", \"C00XXXXXX\", \"SA17.XXXX\", nil, nil, \"IND\" ... ], ... ]\r\n \r\n filing.contributions(:raw => true) do |row|\r\n\trow => [\"SA17A\", \"C00XXXXX\", \"SA17.XXXX\", nil, nil, \"IND\" ... ]\r\n end\r\n```\r\n\r\nThe source maps for individual row types and versions may also be accessed directly:\r\n\r\n```ruby\r\n Fech::Filing.map_for(\"sa\")\r\n Fech::Filing.map_for(/sa/, :version => 6.1)\r\n => [:form_type, :filer_committee_id_number, :transaction_id ... ]\r\n```\r\n\r\nYou can then bypass some of the overhead of Fech if you're building something more targeted.\r\n\r\n### Converting / Preprocessing data\r\n\r\nFor performing bulk actions on specific types of fields, you can register \"translations\" which will manipulate specific data under certain conditions.\r\n\r\nAn example: dates within filings are formatted as YYYYMMDD. To automatically convert all Schedule B `:expenditure_date` values to native Ruby Dates:\r\n\r\n```ruby\r\n filing.translate do |t|\r\n t.convert(:row => /^sb/, :field => :expenditure_date) { |v| Date.parse(v) }\r\n end\r\n```\r\n\r\nThe block you give `.convert` will be given the original value of that field when it is run. After you run `.convert`, any time you parse a row beginning with \"SB\", the `:expenditure_date` value will be a Date object.\r\n\r\nThe `:field` parameter can also be a regular expression:\r\n\r\n```ruby\r\n filing.translate do |t|\r\n t.convert(:row => /^f3p/, :field => /^coverage_.*_date/) { |v| Date.parse(v) }\r\n end\r\n```\r\n\r\nNow, both `:coverage_from_date` and `:coverage_through_date` will be automatically cast to dates.\r\n\r\nYou can leave off any or all of the parameters (row, field, version) for more broad adoption of the translation.\r\n\r\n### Derived Fields\r\n\r\nYou may want to perform the same calculation on many rows of data (contributions minus refunds to create a net value, for example). This can be done without cluttering up your more app-specific parsing code by using a `.combine` translation. The translation blocks you pass .combine receive the entire row as their context, not just a single value. The `:field` parameter becomes what you want the new value to be named.\r\n\r\n```ruby\r\n filing.translate do |t|\r\n t.combine(:row => :f3pn, :field => :net_individual_contributions) do |row|\r\n contribs = row.col_a_individual_contribution_total.to_f\r\n refunds = row.col_a_total_contributions_refunds.to_f\r\n contribs - refunds\r\n end\r\n end\r\n```\r\n\r\nIn this example, every parsed Schedule A row would contain an attribute not found in the original filing data - `:net_individual_contributions` - which contains the result of the calculation above. The values used to construct combinations will have already been run through any .convert translations you've specified.\r\n\r\n### Built-in translations\r\n\r\nThere are two sets of translations that come with Fech for some of the more common needs:\r\n\r\n* Breaking apart names into their component parts and joining them back together, depending on which the filing provides\r\n* Converting date field strings to Ruby Date objects\r\n\r\nYou can mix these translations into your parser when you create it:\r\n\r\n```ruby\r\n filing = Fech::Filing.new(723604, :translate => [:names, :dates])\r\n```\r\n\r\nJust be aware that as you add more translations, the parsing will start to take slightly longer (although having translations that aren't used will not slow it down).\r\n\r\n### Aliasing fields\r\n\r\nYou can allow any field (converted, combined, or untranslated) to have an arbitrary number of aliases. For example, you could alias the F3P line's `:col_a_6_cash_on_hand_beginning_period` to the more manageable `:cash_beginning`\r\n\r\n```ruby\r\n filing.translate do |t|\r\n t.alias :cash_beginning, :col_a_cash_on_hand_beginning_period, :f3p\r\n end\r\n \r\n filing.summary.cash_beginning ## filing.summary.col_a_cash_on_hand_beginning_period.\r\n => true\r\n```\r\n\r\nWe found it useful to be able to access attributes using the name the fields in our database they correspond to.\r\n\r\n### Unofficial Senate Filings\r\n\r\nFech also supports [unofficial electronic filings submitted voluntarily by Senate campaign committees](http://query.nictusa.com/senate/). While Senate candidates are not required to file electronically, some do. To access a Senate filing, use `Fech::SenateFiling`:\r\n\r\n```filing = Fech::SenateFiling.new(853)```\r\n\r\nYou can then work with the filing as you would any other.\r\n\r\n### Comparing filings<a name=\"compare\"/>\r\n\r\nYou may want to compare two electronic filings to each other, particularly an original filing and its amendment, since amendments are a full replacement of the original filing. Fech has a Comparison class that allows you to compare the summary or a specific schedule of itemizations between two filings. The use of Comparison requires that you have created both Filing objects and downloaded them:\r\n\r\n```ruby\r\n filing_1 = Fech::Filing.new(767437)\r\n filing_1.download\r\n filing_2 = Fech::Filing.new(751798)\r\n filing_2.download\r\n comparison = Fech::Comparison.new(filing_1, filing_2)\r\n comparison.summary\r\n```\r\n\r\nComparison's summary method returns a hash of summary fields whose values have changed in filing_2. The schedule method accepts either a symbol or string representing the schedule. So, for Schedule A (itemized contributions): \r\n\r\n```ruby\r\n comparison.schedule(:sa)\r\n```\r\n\r\nThe result will be an array of Fech contribution objects that differ in any way. If an amendment updates the employer or occupation of a contributor, for example, `comparison.schedule(:sa)` would return that contribution. If the array is empty, no itemized records changed.\r\n\r\n### Handling malformed filings\r\n\r\nSome filers put double quotes around each field in their filings, which can cause problems if not done correctly. For example, filing No. 735497 has the following value in the field for the name of a payee:\r\n\r\n \"Stichin\" LLC\"\r\n\r\nBecause the field is not properly quoted FasterCSV and CSV will not be able to parse it, and will instead raise a MalformedCSVError.\r\n\r\nTo get around this problem, you can pass in Fech::CsvDoctor as the `:csv_parser` when initializing the filing (see: CSV Parsers). It will then take the following steps when parsing the filing:\r\n\r\n * Try to parse the line using the default CSV parser, using double quotes as the quote character (by default) or a different quote character if specified.\r\n * If a MalformedCSVError is raised, try to parse the line using a null value as the quote character. If this succeeds, each value in the line is then parsed, so that the quote characters around properly quoted fields are not included in Fech's output.\r\n\r\nIf you'd like to parse an entire filing using a specific CSV quote character, see: Quote character. \r\n\r\n### Quote character<a name=\"quote_char\"/>\r\n\r\nBy default, Fech, like FasterCSV and CSV, assumes that double quotes are used around values that contain reserved characters.\r\n\r\nHowever, the parsing of some filings might fail because of the existence of the default quote character in a field. To get around this, you may pass a `:quote_char` argument, and Fech will use it to parse the file.\r\n\r\nFor example, filing No. 756218 has the following value in the field for the contributor's last name:\r\n\r\n O\"\"Leary\r\n\r\nAttempting to parse this filing with Fech fails because of illegal quoting.\r\n\r\nSince the FEC doesn't use a quote character in its electronic filings, we don't need a quote character at all. One way to achieve this is to use \"\\0\" (a null ASCII character) as `:quote_char`:\r\n\r\n```ruby\r\n filing = Fech::Filing.new(756218, :quote_char => \"\\0\")\r\n```\r\n\r\n## Warnings<a name=\"warnings\"/>\r\n\r\nFilings can contain data that is incomplete or wrong: contributions might be in excess of legal limits, or data may have just been entered incorrectly. While some of these mistakes are corrected in later filing amendments, you should be aware that the data is not perfect. Fech will only return data as accurate as the source.\r\n\r\nWhen filings get very large, be careful not to perform operations that attempt to transform many rows in memory.\r\n\r\n## Supported row types and versions<a name=\"row_types\"/>\r\n\r\nThe following row types are currently supported from filing version 3 through 8.0:\r\n\r\n* F1 (Statement of Organization)\r\n* F2 (Statement of Candidacy)\r\n* F24 (24/48 Hour Notice of Independent Expenditure)\r\n* F3 (including termination reports)\r\n* F3L (Report of Contributions Bundled by Lobbyists/Registrants)\r\n* F3P (Summaries)\r\n* F3PS\r\n* F3P31 (Items to be Liquidated)\r\n* F3S\r\n* F3X (Report of Receipts and Disbursements for Other Than an Authorized Committee)\r\n* F4 (Report of Receipts and Disbursements for a Convention Committee)\r\n* F5 (Report of Independent Expenditures Made and Contributions Received)\r\n* F56 (Contributions for Independent Expenditures)\r\n* F57 (Independent Expenditures)\r\n* F6 (48 Hour Notice of Contributions/Loans Received)\r\n* F65 (Contributions for 48 Hour Notices)\r\n* F9 (Electioneering Communications, first appeared in filing version 5.0)\r\n* F91 (Lists of Persons Exercising Control)\r\n* F92 (Contributions for Electioneering Communications)\r\n* F93 (Expenditures for Electioneering Communications)\r\n* F94 (Federal Candidates List for Electioneering Communications)\r\n* F99 (Miscellaneous Text)\r\n* H1 (Method of Allocation for Federal/Non-Federal Activity)\r\n* H2 (Federal/Non-Federal Allocation Ratios)\r\n* H3 (Transfers from Non-Federal Accounts)\r\n* H4 (Disbursements for Allocated Federal/Non-Federal Activity)\r\n* H5 (Transfers from Levin Funds)\r\n* H6 (Disbursements from Levin Funds)\r\n* SA (Contributions)\r\n* SB (Expenditures)\r\n* SC (Loans)\r\n* SC1\r\n* SC2\r\n* SD (Debts & Obligations)\r\n* SE (Independent Expenditures)\r\n* SF (Coordinated Expenditures)\r\n* SL (Levin Fund Summary)","google":"","note":"Don't delete this file! It's used internally to help with page regeneration."}