From c61d3c938a269485fee7a55f43024c2ecec050ef Mon Sep 17 00:00:00 2001 From: Derek Willis Date: Wed, 2 May 2012 07:46:59 -0400 Subject: [PATCH 01/13] added overview file --- overview.md | 20 ++++++++++++++++++++ 1 file changed, 20 insertions(+) create mode 100644 overview.md diff --git a/overview.md b/overview.md new file mode 100644 index 0000000..b085da4 --- /dev/null +++ b/overview.md @@ -0,0 +1,20 @@ +Overview +========== + +The Federal Election Commission offers three general types of downloadable campaign finance data: individual electronic filings in CSV format covering most committees, fixed-width bulk data via FTP from all committees and summary files (both fixed-width and CSV) covering two-year election cycles. Electronic filings are available in real time as they are filed, while the other two types are updated on a regular basis. Let's take them in turn. + +Electronic Filings +======== + +Most committees registered with the FEC are required to file all of their reports electronically. There are two significant exceptions: committees of U.S. Senate candidates (and two senatorial party committees), which file with the Secretary of the Senate, and [committees that raise or spend less than $50,000 in a calendar year](http://fec.gov/ans/answers_filing.shtml#Do_I_need_to_file_electronically), or expect to do so. Electronic filing applies to U.S. House and presidential candidate committees, non-candidate political committees (typically knows as PACs), national, state and local political party committees registered with the FEC and individuals or organizations engaged in independent spending. Electronic filing was mandated in 2001. + +Electronic filings can be found via the [FEC's search form](http://www.fec.gov/finance/disclosure/efile_search.shtml), which has a number of options for filtering the results to a particular committee, a particular date or more. The search results include the option to View or Download individual filings, which present the data in HTML and delimited (see __delimiters__) formats, respectively. Individual filings contain all of the records for that filing, stacked on top of each other in varying delimited layouts ([A zip file containing the formats is on FEC.gov](http://www.fec.gov/elecfil/eFilingFormats.zip).) + +Even though committees file reports [on a regular schedule](http://www.fec.gov/info/report_dates.shtml), electronic filings occur nearly every day of the year. Some are amendments of previous filings, others are filed in advance (or after) a deadline, and others are filed as changes warrant. Filings that are __amendments__ are indicated in the data, and serve as complete replacements for the original filings. + +Bulk FTP Data +======== + +The FEC has offered bulk data for years, and [its offerings](http://www.fec.gov/finance/disclosure/ftp_download.shtml) include summary and detailed files covering committees, candidates and contributions. The bulk files are updated weekly, late on Sunday nights, so depending on your timing and needs the bulk files may not be suitable for every task. The advantage of the bulk files is that they are vetted by the FEC, with some of the records standardized (by adding FEC-issued committee ids, for example) and others removed to prevent duplicate records from appearing. + +The [summary files](http://www.fec.gov/finance/disclosure/ftpsum.shtml) include canonical data for candidates and committees - one record for each candidate or committee per two-year election cycle. During the current cycle, \ No newline at end of file From a5e4038f51f37f91fe03dfd176ba9decdc095f71 Mon Sep 17 00:00:00 2001 From: Derek Willis Date: Fri, 4 May 2012 21:37:59 -0400 Subject: [PATCH 02/13] added ftp data descriptions --- overview.md | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-) diff --git a/overview.md b/overview.md index b085da4..234cc8a 100644 --- a/overview.md +++ b/overview.md @@ -15,6 +15,12 @@ Even though committees file reports [on a regular schedule](http://www.fec.gov/i Bulk FTP Data ======== -The FEC has offered bulk data for years, and [its offerings](http://www.fec.gov/finance/disclosure/ftp_download.shtml) include summary and detailed files covering committees, candidates and contributions. The bulk files are updated weekly, late on Sunday nights, so depending on your timing and needs the bulk files may not be suitable for every task. The advantage of the bulk files is that they are vetted by the FEC, with some of the records standardized (by adding FEC-issued committee ids, for example) and others removed to prevent duplicate records from appearing. +The FEC has offered bulk data for years, and [its offerings](http://www.fec.gov/finance/disclosure/ftp_download.shtml) include summary and detailed files covering committees, candidates and contributions. The bulk files are updated weekly, late on Sunday nights, so depending on your timing and needs the bulk files may not be suitable for every task. The advantage of the bulk files is that they are vetted by the FEC, with some of the records standardized (by adding FEC-issued committee ids, for example) and others removed to prevent duplicate records from appearing. The summary files are in fixed-width format, while the detailed files are pipe-delimited. -The [summary files](http://www.fec.gov/finance/disclosure/ftpsum.shtml) include canonical data for candidates and committees - one record for each candidate or committee per two-year election cycle. During the current cycle, \ No newline at end of file +The [summary files](http://www.fec.gov/finance/disclosure/ftpsum.shtml) include canonical data for candidates and committees - one record for each candidate or committee per two-year election cycle, depending on the file selected. There are two candidate summary files -- one for campaigns that have elections in the current cycle, and one for all candidates no matter if they face election in the cycle or not -- and they can differ in amounts and timeliness. The [current campaigns file](ftp://ftp.fec.gov/FEC/webl12.zip) may be more timely, but also contains a single total for PAC contributions (compared to totals for different kinds of PACs in the other file) and some of its totals may contain double-counted transactions. + +The [all candidates file](ftp://ftp.fec.gov/FEC/weball12.zip) can be updated slightly less frequently than the current campaigns one, but it contains more detailed breakdowns of certain types of transactions as noted above. The possibility of double-counting some kinds of transactions - transfers to and from authorized committees of a candidate - also exists. + +The FEC previously used to generate candidate summary files at the end of the election cycle that included totals for different types of PACs, but discontinued these files after the 2005-2006 cycle. Files are available from 1979-80 through 2005-06. + +There's one more thing to look out for - the way that the FEC used to store its data (detailed and summary) relied on the use of [an "overpunch" character](http://www.fec.gov/finance/disclosure/ftpsum.shtml#overpunch) to represent negative amounts. This [recently changed for the detailed contribution files](http://www.fec.gov/blog/disclosure/entry/indiv_oth_and_pas2_file), but it's possible that older summary files still contain such characters, and that the amount fields should be imported as text and then converted to numeric columns, accounting for negative amounts. \ No newline at end of file From 0e25939e36bdd7ab49dda406c6845e56d93294bb Mon Sep 17 00:00:00 2001 From: Derek Willis Date: Fri, 4 May 2012 21:49:16 -0400 Subject: [PATCH 03/13] finished summary ftp files section --- overview.md | 11 +++++++++-- 1 file changed, 9 insertions(+), 2 deletions(-) diff --git a/overview.md b/overview.md index 234cc8a..218216a 100644 --- a/overview.md +++ b/overview.md @@ -17,10 +17,17 @@ Bulk FTP Data The FEC has offered bulk data for years, and [its offerings](http://www.fec.gov/finance/disclosure/ftp_download.shtml) include summary and detailed files covering committees, candidates and contributions. The bulk files are updated weekly, late on Sunday nights, so depending on your timing and needs the bulk files may not be suitable for every task. The advantage of the bulk files is that they are vetted by the FEC, with some of the records standardized (by adding FEC-issued committee ids, for example) and others removed to prevent duplicate records from appearing. The summary files are in fixed-width format, while the detailed files are pipe-delimited. +Summary files +--------- + The [summary files](http://www.fec.gov/finance/disclosure/ftpsum.shtml) include canonical data for candidates and committees - one record for each candidate or committee per two-year election cycle, depending on the file selected. There are two candidate summary files -- one for campaigns that have elections in the current cycle, and one for all candidates no matter if they face election in the cycle or not -- and they can differ in amounts and timeliness. The [current campaigns file](ftp://ftp.fec.gov/FEC/webl12.zip) may be more timely, but also contains a single total for PAC contributions (compared to totals for different kinds of PACs in the other file) and some of its totals may contain double-counted transactions. The [all candidates file](ftp://ftp.fec.gov/FEC/weball12.zip) can be updated slightly less frequently than the current campaigns one, but it contains more detailed breakdowns of certain types of transactions as noted above. The possibility of double-counting some kinds of transactions - transfers to and from authorized committees of a candidate - also exists. -The FEC previously used to generate candidate summary files at the end of the election cycle that included totals for different types of PACs, but discontinued these files after the 2005-2006 cycle. Files are available from 1979-80 through 2005-06. +The FEC previously used to generate summary files at the end of the election cycle for candidates and PACs that included totals for different types of PACs, but discontinued these files after the 2005-06 cycle. Files are available from 1979-80 through 2005-06. Party committee summary files exist from the 1991-92 cycle through the 2003-04 cycle. One of the most useful files, which contained a record for each combination of candidate recipient and PAC contributor/independent spender, covers the 1991-92 cycle through the 2001-02 cycle. A similar file exists for candidate-party activities during the same time period. + +There's one more thing to look out for - the way that the FEC used to store its data (detailed and summary) relied on the use of [an "overpunch" character](http://www.fec.gov/finance/disclosure/ftpsum.shtml#overpunch) to represent negative amounts. This [recently changed for the detailed contribution files](http://www.fec.gov/blog/disclosure/entry/indiv_oth_and_pas2_file), but it's possible that older summary files still contain such characters, and that the amount fields should be imported as text and then converted to numeric columns, accounting for negative amounts. There is a tutorial for [working with the FTP files using Microsoft Access](http://www.fec.gov/finance/disclosure/working_with_data_files.pdf). + +Detailed files +--------- -There's one more thing to look out for - the way that the FEC used to store its data (detailed and summary) relied on the use of [an "overpunch" character](http://www.fec.gov/finance/disclosure/ftpsum.shtml#overpunch) to represent negative amounts. This [recently changed for the detailed contribution files](http://www.fec.gov/blog/disclosure/entry/indiv_oth_and_pas2_file), but it's possible that older summary files still contain such characters, and that the amount fields should be imported as text and then converted to numeric columns, accounting for negative amounts. \ No newline at end of file From 0094aa3cf0e6c4c1c2c046ac34e22b04f5871241 Mon Sep 17 00:00:00 2001 From: Derek Willis Date: Sun, 29 Jul 2012 23:05:00 -0400 Subject: [PATCH 04/13] added data catalog to overview --- overview.md | 20 ++++++-------------- 1 file changed, 6 insertions(+), 14 deletions(-) diff --git a/overview.md b/overview.md index 218216a..bb15c8a 100644 --- a/overview.md +++ b/overview.md @@ -1,7 +1,7 @@ Overview ========== -The Federal Election Commission offers three general types of downloadable campaign finance data: individual electronic filings in CSV format covering most committees, fixed-width bulk data via FTP from all committees and summary files (both fixed-width and CSV) covering two-year election cycles. Electronic filings are available in real time as they are filed, while the other two types are updated on a regular basis. Let's take them in turn. +The Federal Election Commission offers three general types of downloadable campaign finance data: individual electronic filings in CSV format covering most committees, pipe-delimited bulk itemized data and summary files via FTP from all committees covering two-year election cycles and specialized CSV or XML files via its [Data Catalog](http://fec.gov/data/DataCatalog.do?format=html). Electronic filings are available in real time as they are filed, while the other two types are updated on a regular basis. Let's take them in turn. Electronic Filings ======== @@ -15,19 +15,11 @@ Even though committees file reports [on a regular schedule](http://www.fec.gov/i Bulk FTP Data ======== -The FEC has offered bulk data for years, and [its offerings](http://www.fec.gov/finance/disclosure/ftp_download.shtml) include summary and detailed files covering committees, candidates and contributions. The bulk files are updated weekly, late on Sunday nights, so depending on your timing and needs the bulk files may not be suitable for every task. The advantage of the bulk files is that they are vetted by the FEC, with some of the records standardized (by adding FEC-issued committee ids, for example) and others removed to prevent duplicate records from appearing. The summary files are in fixed-width format, while the detailed files are pipe-delimited. +The FEC has offered bulk data for years, and [its offerings](http://www.fec.gov/finance/disclosure/ftp_download.shtml) include summary and detailed files covering committees, candidates and contributions. The bulk files are updated weekly, late on Sunday nights, so depending on your timing and needs the bulk files may not be suitable for every task. The advantage of the bulk files is that they are vetted by the FEC, with some of the records standardized (by adding FEC-issued committee ids, for example) and others removed to prevent duplicate records from appearing. The __summary___ files are in fixed-width format, while the __detailed__ files are either fixed-width or pipe-delimited. The FEC is increasingly moving to pipe-delimited formats. -Summary files ---------- +Bulk data files are contained inside zip files stored on the FTP server, so retrieving them via a web application requires several steps. The FTP data is updated early Monday morning each week, and previous cycles are updated as well, since committees can amend filings from an earlier election cycle. -The [summary files](http://www.fec.gov/finance/disclosure/ftpsum.shtml) include canonical data for candidates and committees - one record for each candidate or committee per two-year election cycle, depending on the file selected. There are two candidate summary files -- one for campaigns that have elections in the current cycle, and one for all candidates no matter if they face election in the cycle or not -- and they can differ in amounts and timeliness. The [current campaigns file](ftp://ftp.fec.gov/FEC/webl12.zip) may be more timely, but also contains a single total for PAC contributions (compared to totals for different kinds of PACs in the other file) and some of its totals may contain double-counted transactions. - -The [all candidates file](ftp://ftp.fec.gov/FEC/weball12.zip) can be updated slightly less frequently than the current campaigns one, but it contains more detailed breakdowns of certain types of transactions as noted above. The possibility of double-counting some kinds of transactions - transfers to and from authorized committees of a candidate - also exists. - -The FEC previously used to generate summary files at the end of the election cycle for candidates and PACs that included totals for different types of PACs, but discontinued these files after the 2005-06 cycle. Files are available from 1979-80 through 2005-06. Party committee summary files exist from the 1991-92 cycle through the 2003-04 cycle. One of the most useful files, which contained a record for each combination of candidate recipient and PAC contributor/independent spender, covers the 1991-92 cycle through the 2001-02 cycle. A similar file exists for candidate-party activities during the same time period. - -There's one more thing to look out for - the way that the FEC used to store its data (detailed and summary) relied on the use of [an "overpunch" character](http://www.fec.gov/finance/disclosure/ftpsum.shtml#overpunch) to represent negative amounts. This [recently changed for the detailed contribution files](http://www.fec.gov/blog/disclosure/entry/indiv_oth_and_pas2_file), but it's possible that older summary files still contain such characters, and that the amount fields should be imported as text and then converted to numeric columns, accounting for negative amounts. There is a tutorial for [working with the FTP files using Microsoft Access](http://www.fec.gov/finance/disclosure/working_with_data_files.pdf). - -Detailed files ---------- +Data Catalog +======== +The data catalog is a collection of some of the summary files available via FTP as well as other files covering disbursements, independent expenditures and leadership PACs, among other subjects. The files are available in CSV or XML formats, and cover single cycles (mostly 2010 and 2012, although the summary files also include 2008 data). One advantage of the data catalog files is that they can be called directly from a web application without having to unzip them, but there are some drawbacks. [Independent Expenditures](http://fec.gov/data/IndependentExpenditure.do?format=html&election_yr=2012) include both original transactions and amendments, resulting in duplicate records in those cases. In another example, the [listing of leadership PACs](http://fec.gov/data/Leadership.do?format=html&election_yr=2012) contains an entry for the corporate PAC of Interactive Corp. Most of the data catalog files are updated daily, and they are the one place where it's possible to find [candidate disbursements](http://fec.gov/data/CandidateDisbursement.do?format=html&election_yr=2012) in statewide or district-level files. The files themselves are stored on the FEC's FTP server, so it's possible to grab them directly. The FEC also maintains [a blog about its data](http://fec.gov/blog/) that includes changes and additions to its data offerings. From e8d972574f377a9ee65ae7d9cab6479e38f57beb Mon Sep 17 00:00:00 2001 From: Derek Willis Date: Mon, 30 Jul 2012 21:59:12 -0400 Subject: [PATCH 05/13] added initial files for bulk data --- bulk_detailed.md | 8 ++++++++ bulk_summary.md | 12 ++++++++++++ 2 files changed, 20 insertions(+) create mode 100644 bulk_detailed.md create mode 100644 bulk_summary.md diff --git a/bulk_detailed.md b/bulk_detailed.md new file mode 100644 index 0000000..b6f0161 --- /dev/null +++ b/bulk_detailed.md @@ -0,0 +1,8 @@ +Detailed files +======== + +The [detailed files](http://www.fec.gov/finance/disclosure/ftpdet.shtml) include cycle-specific data for committees, candidates, individual contributions and committee transactions to candidates and between committees. Each cycle, beginning with 1979-80, has five files with data representing that cycle. The current cycle and its immediate 2-3 preceding cycles also have three smaller tables that represent additions, changes and deletions to the individual contributions file. The FEC changed the format of these files from fixed-width to pipe-delimited in 2012. + +Committees +--------- + diff --git a/bulk_summary.md b/bulk_summary.md new file mode 100644 index 0000000..9643147 --- /dev/null +++ b/bulk_summary.md @@ -0,0 +1,12 @@ +Summary files +======== + +The [summary files](http://www.fec.gov/finance/disclosure/ftpsum.shtml) include canonical election-cycle data for candidates and committees - one record for each candidate or committee per two-year election cycle, depending on the file selected. There are two candidate summary files -- one for campaigns that have elections in the current cycle, and one for all candidates no matter if they face election in the cycle or not -- and they can differ in amounts and timeliness. The [current campaigns file](ftp://ftp.fec.gov/FEC/webl12.zip) may be more timely, but also contains a single total for PAC contributions (compared to totals for different kinds of PACs in the other file) and some of its totals may contain double-counted transactions. + +The [all candidates file](ftp://ftp.fec.gov/FEC/weball12.zip) can be updated slightly less frequently than the current campaigns one, but it contains more detailed breakdowns of certain types of transactions as noted above. The possibility of double-counting some kinds of transactions - transfers to and from authorized committees of a candidate - also exists. + +The FEC previously used to generate summary files at the end of the election cycle for candidates and PACs that included totals for different types of PACs, but discontinued these files after the 2005-06 cycle. Files are available from 1979-80 through 2005-06. Party committee summary files exist from the 1991-92 cycle through the 2003-04 cycle. One of the most useful files, which contained a record for each combination of candidate recipient and PAC contributor/independent spender, covers the 1991-92 cycle through the 2001-02 cycle. A similar file exists for candidate-party activities during the same time period. + +These summary files had been stored in fixed-width format but were converted to pipe-delimited format in late July 2012. + +There's one more thing to be aware of: the way that the FEC used to store its data (detailed and summary) relied on the use of [an "overpunch" character](http://www.fec.gov/finance/disclosure/ftpsum.shtml#overpunch) to represent negative amounts. This [recently changed for the detailed contribution files](http://www.fec.gov/blog/disclosure/entry/indiv_oth_and_pas2_file) and for the candidate and committee files, but it's possible that older summary files still contain such characters, and that the amount fields should be imported as text and then converted to numeric columns, accounting for negative amounts. There is a tutorial for [working with the FTP files using Microsoft Access](http://www.fec.gov/finance/disclosure/working_with_data_files.pdf). \ No newline at end of file From a0238bb18bf2c5f28a954234a925e6e7f423abaf Mon Sep 17 00:00:00 2001 From: Derek Willis Date: Mon, 30 Jul 2012 22:03:31 -0400 Subject: [PATCH 06/13] added bulk data and data catalog sections to overview --- overview.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/overview.md b/overview.md index bb15c8a..8f37bf4 100644 --- a/overview.md +++ b/overview.md @@ -15,11 +15,11 @@ Even though committees file reports [on a regular schedule](http://www.fec.gov/i Bulk FTP Data ======== -The FEC has offered bulk data for years, and [its offerings](http://www.fec.gov/finance/disclosure/ftp_download.shtml) include summary and detailed files covering committees, candidates and contributions. The bulk files are updated weekly, late on Sunday nights, so depending on your timing and needs the bulk files may not be suitable for every task. The advantage of the bulk files is that they are vetted by the FEC, with some of the records standardized (by adding FEC-issued committee ids, for example) and others removed to prevent duplicate records from appearing. The __summary___ files are in fixed-width format, while the __detailed__ files are either fixed-width or pipe-delimited. The FEC is increasingly moving to pipe-delimited formats. +The FEC has offered bulk data for years, and [its offerings](http://www.fec.gov/finance/disclosure/ftp_download.shtml) include summary and detailed files covering committees, candidates and contributions. The bulk files are updated weekly, late on Sunday nights, so depending on your timing and needs the bulk files may not be suitable for every task. The advantage of the bulk files is that they are vetted by the FEC, with some of the records standardized (by adding FEC-issued committee ids, for example) and others removed to prevent duplicate records from appearing. The __bulk_summary___ files are in fixed-width format, while the __bulk_detailed__ files are either fixed-width or pipe-delimited. The FEC is increasingly moving to pipe-delimited formats. Bulk data files are contained inside zip files stored on the FTP server, so retrieving them via a web application requires several steps. The FTP data is updated early Monday morning each week, and previous cycles are updated as well, since committees can amend filings from an earlier election cycle. Data Catalog ======== -The data catalog is a collection of some of the summary files available via FTP as well as other files covering disbursements, independent expenditures and leadership PACs, among other subjects. The files are available in CSV or XML formats, and cover single cycles (mostly 2010 and 2012, although the summary files also include 2008 data). One advantage of the data catalog files is that they can be called directly from a web application without having to unzip them, but there are some drawbacks. [Independent Expenditures](http://fec.gov/data/IndependentExpenditure.do?format=html&election_yr=2012) include both original transactions and amendments, resulting in duplicate records in those cases. In another example, the [listing of leadership PACs](http://fec.gov/data/Leadership.do?format=html&election_yr=2012) contains an entry for the corporate PAC of Interactive Corp. Most of the data catalog files are updated daily, and they are the one place where it's possible to find [candidate disbursements](http://fec.gov/data/CandidateDisbursement.do?format=html&election_yr=2012) in statewide or district-level files. The files themselves are stored on the FEC's FTP server, so it's possible to grab them directly. The FEC also maintains [a blog about its data](http://fec.gov/blog/) that includes changes and additions to its data offerings. +The data catalog is a collection of some of the summary files available via FTP as well as other files covering disbursements, independent expenditures and leadership PACs, among other subjects. The files are available in CSV or XML formats, and cover single cycles (mostly 2010 and 2012, although the summary files also include 2008 data). One advantage of the data catalog files is that they can be called directly from a web application without having to unzip them, but there are some drawbacks. [Independent Expenditures](http://fec.gov/data/IndependentExpenditure.do?format=html&election_yr=2012) include both original transactions and amendments, resulting in duplicate records in those cases. In another example, the [listing of leadership PACs](http://fec.gov/data/Leadership.do?format=html&election_yr=2012) contains an entry for the corporate PAC of Interactive Corp. Most of the data catalog files are updated daily, and they are the one place where it's possible to find [candidate disbursements](http://fec.gov/data/CandidateDisbursement.do?format=html&election_yr=2012) in statewide or district-level files. The files themselves are stored on the FEC's FTP server, so it's possible to grab them directly. The FEC also maintains [a blog about its data](http://fec.gov/blog/) that includes changes and additions to its data offerings. \ No newline at end of file From 7e7ff385ca355e3d622d3beea0174f8cebc7632b Mon Sep 17 00:00:00 2001 From: Derek Willis Date: Tue, 31 Jul 2012 22:34:10 -0400 Subject: [PATCH 07/13] additions to bulk detailed --- bulk_detailed.md | 18 +++++++++++++++++- 1 file changed, 17 insertions(+), 1 deletion(-) diff --git a/bulk_detailed.md b/bulk_detailed.md index b6f0161..bf1c791 100644 --- a/bulk_detailed.md +++ b/bulk_detailed.md @@ -1,8 +1,24 @@ Detailed files ======== -The [detailed files](http://www.fec.gov/finance/disclosure/ftpdet.shtml) include cycle-specific data for committees, candidates, individual contributions and committee transactions to candidates and between committees. Each cycle, beginning with 1979-80, has five files with data representing that cycle. The current cycle and its immediate 2-3 preceding cycles also have three smaller tables that represent additions, changes and deletions to the individual contributions file. The FEC changed the format of these files from fixed-width to pipe-delimited in 2012. +The [detailed files](http://www.fec.gov/finance/disclosure/ftpdet.shtml) include cycle-specific data for committees, candidates, individual contributions and committee transactions to candidates and between committees. Each cycle, beginning with 1979-80, has five files with data representing that cycle. The current cycle and its immediate 2-3 preceding cycles also have three smaller tables that represent additions, changes and deletions to the individual contributions file. The FEC changed the format of these files from fixed-width to pipe-delimited in July 2012. These files are updated weekly on late Sunday evenings/early Monday mornings, so information submitted during the week before should be reflected in the next update. One advantage of using these files is that they represent "official" data that has been checked by the FEC. Nonetheless, there have been some errors Committees --------- +Known as the [committee master file](http://www.fec.gov/finance/disclosure/metadata/DataDictionaryCommitteeMaster.shtml), this has a record for each committee registered with the FEC during a cycle. The ID number assigned by the FEC to each committee is unique within the cycle, but committees often exist for years or even decades. If a committee is a campaign committee for a candidate for the House, Senate or President, it will include that candidate's ID number. A committee will be one of [several types](http://www.fec.gov/finance/disclosure/metadata/CommitteeTypeCodes.shtml). Other committees may have values for party affiliation, frequency of filings and connected organization, although these values are not universally present for each committee. + +Candidates +--------- + +Known as the [candidate master file](http://www.fec.gov/finance/disclosure/metadata/DataDictionaryCandidateMaster.shtml), this file contains one row for each candidate who either registered with the FEC or "appeared on a ballot list prepared by a state elections office." Key fields in this file include whether a candidate is an incumbent, challenger or seeking an open seat (although this field is not consistently populated), and the state and district the candidate is running in. + +Candidate Committee Linkage +--------- + +A new offering as of July 2012, [this file](http://www.fec.gov/finance/disclosure/metadata/DataDictionaryCandCmteLinkage.shtml) contains one row for each combination of candidate and committee during an election cycle. Candidates can have a primary committee, authorized committees, joint fundraising committees and other relationships. The linkage file helps keep track of all of those relationships. In particular, joint fundraising committees can benefit multiple candidates; previously the committee master file would only list one. + +Committee Contributions To Candidates +--------- + +[This file](http://www.fec.gov/finance/disclosure/metadata/DataDictionaryContributionstoCandidates.shtml) contains one row for each contribution from a committee to a candidate and for each independent expenditure made by a committee about a candidate. It is a subset of the file containing all transactions between one committee and another. From 42dafa88a89aa2e418c5751310d9796a59bfdd84 Mon Sep 17 00:00:00 2001 From: Derek Willis Date: Wed, 1 Aug 2012 21:33:10 -0400 Subject: [PATCH 08/13] added initial text --- README.md | 1 + 1 file changed, 1 insertion(+) diff --git a/README.md b/README.md index d5f6201..fcd766e 100644 --- a/README.md +++ b/README.md @@ -1,3 +1,4 @@ A developer's guide to FEC data ============= +This document is an attempt to describe the range of federal campaign finance data provided by the [Federal Election Commission](http://fec.gov/) and its usage by developers. It assumes no prior knowledge of campaign finance law or practice, but some familiarity with the U.S. federal electoral system, such as the difference between the House and Senate. \ No newline at end of file From e9f3b3a0785e66c270ef09b11ddfa8f14f9c2262 Mon Sep 17 00:00:00 2001 From: Derek Willis Date: Wed, 1 Aug 2012 21:46:44 -0400 Subject: [PATCH 09/13] updated links --- overview.md | 18 ++++++++++++++---- 1 file changed, 14 insertions(+), 4 deletions(-) diff --git a/overview.md b/overview.md index 8f37bf4..061809f 100644 --- a/overview.md +++ b/overview.md @@ -1,21 +1,31 @@ Overview ========== -The Federal Election Commission offers three general types of downloadable campaign finance data: individual electronic filings in CSV format covering most committees, pipe-delimited bulk itemized data and summary files via FTP from all committees covering two-year election cycles and specialized CSV or XML files via its [Data Catalog](http://fec.gov/data/DataCatalog.do?format=html). Electronic filings are available in real time as they are filed, while the other two types are updated on a regular basis. Let's take them in turn. +The Federal Election Commission offers three general types of downloadable campaign finance data: individual electronic filings in CSV format covering most committees, pipe-delimited bulk itemized data and summary files via FTP from all committees covering two-year election cycles and specialized CSV or XML files via its [Data Catalog](http://fec.gov/data/DataCatalog.do?format=html). Electronic filings are available in real time as they are filed, while the other two types are updated on a regular basis. + +The base element of federal campaign finance data is the [committee](/glossary/committee/). Committees are the recipient of money and the spenders of money, and there are a number of different kinds of committees. Although most FEC rules apply to all committees, some committees have different limits or rules to follow, and these matter. + +The second element is the [filing](/glossary/filing/), a report by a committee to the FEC. There are many different kinds of filings, but in general they cover 3 different types of information: + + 1. The formation of committees and candidacies, and their details. + 2. The raising of money for campaigns. + 3. The spending of money on campaigns. + +There is a consistent schedule for filings covering the last two types, and filings of the first type can be made at any time. In general, the FEC recognizes a roughly two-year election cycle that corresponds to U.S. House elections (where all 435 seats are up for election every 2 years). In the case of senators, who are elected to six year terms on a staggered basis, an election cycle may be considered to include the previous four years as well as the current two-year period. The election cycle is used not only as a logical boundary for calculating money raised and spent but also to calculate [contribution limits](/glossary/contribution-limits/) for candidates and committees. Electronic Filings ======== Most committees registered with the FEC are required to file all of their reports electronically. There are two significant exceptions: committees of U.S. Senate candidates (and two senatorial party committees), which file with the Secretary of the Senate, and [committees that raise or spend less than $50,000 in a calendar year](http://fec.gov/ans/answers_filing.shtml#Do_I_need_to_file_electronically), or expect to do so. Electronic filing applies to U.S. House and presidential candidate committees, non-candidate political committees (typically knows as PACs), national, state and local political party committees registered with the FEC and individuals or organizations engaged in independent spending. Electronic filing was mandated in 2001. -Electronic filings can be found via the [FEC's search form](http://www.fec.gov/finance/disclosure/efile_search.shtml), which has a number of options for filtering the results to a particular committee, a particular date or more. The search results include the option to View or Download individual filings, which present the data in HTML and delimited (see __delimiters__) formats, respectively. Individual filings contain all of the records for that filing, stacked on top of each other in varying delimited layouts ([A zip file containing the formats is on FEC.gov](http://www.fec.gov/elecfil/eFilingFormats.zip).) +Electronic filings can be found via the [FEC's search form](http://www.fec.gov/finance/disclosure/efile_search.shtml), which has a number of options for filtering the results to a particular committee, a particular date or more. The search results include the option to View or Download individual filings, which present the data in HTML and delimited (see [delimiters](/electronic-filings/delimiters/)) formats, respectively. Individual filings contain all of the records for that filing, stacked on top of each other in varying delimited layouts ([A zip file containing the formats is on FEC.gov](http://www.fec.gov/elecfil/eFilingFormats.zip).) -Even though committees file reports [on a regular schedule](http://www.fec.gov/info/report_dates.shtml), electronic filings occur nearly every day of the year. Some are amendments of previous filings, others are filed in advance (or after) a deadline, and others are filed as changes warrant. Filings that are __amendments__ are indicated in the data, and serve as complete replacements for the original filings. +Even though committees file reports [on a regular schedule](http://www.fec.gov/info/report_dates.shtml), electronic filings occur nearly every day of the year. Some are amendments of previous filings, others are filed in advance (or after) a deadline, and others are filed as changes warrant. Filings that are [amendments](/electronic-filings/amendments/) are indicated in the data, and serve as complete replacements for the original filings. Bulk FTP Data ======== -The FEC has offered bulk data for years, and [its offerings](http://www.fec.gov/finance/disclosure/ftp_download.shtml) include summary and detailed files covering committees, candidates and contributions. The bulk files are updated weekly, late on Sunday nights, so depending on your timing and needs the bulk files may not be suitable for every task. The advantage of the bulk files is that they are vetted by the FEC, with some of the records standardized (by adding FEC-issued committee ids, for example) and others removed to prevent duplicate records from appearing. The __bulk_summary___ files are in fixed-width format, while the __bulk_detailed__ files are either fixed-width or pipe-delimited. The FEC is increasingly moving to pipe-delimited formats. +The FEC has offered bulk data for years, and [its offerings](http://www.fec.gov/finance/disclosure/ftp_download.shtml) include summary and detailed files covering committees, candidates and contributions. The bulk files are updated weekly, late on Sunday nights, so depending on your timing and needs the bulk files may not be suitable for every task. The advantage of the bulk files is that they are vetted by the FEC, with some of the records standardized (by adding FEC-issued committee ids, for example) and others removed to prevent duplicate records from appearing. The [summary](/bulk/summary/) and [detailed](/bulk/detailed/) files are pipe-delimited, a relatively recent change for the FEC, and some older files may still be in fixed-width format. Bulk data files are contained inside zip files stored on the FTP server, so retrieving them via a web application requires several steps. The FTP data is updated early Monday morning each week, and previous cycles are updated as well, since committees can amend filings from an earlier election cycle. From 987034592f2e766bd7d90b4294a61b3fc4e9ccf1 Mon Sep 17 00:00:00 2001 From: Derek Willis Date: Wed, 1 Aug 2012 21:47:02 -0400 Subject: [PATCH 10/13] organize files in sub-folders --- bulk_detailed.md => bulk/detailed.md | 7 ++++++- bulk_summary.md => bulk/summary.md | 10 +++++++--- amendments.md => electronic-filings/amendments.md | 4 ++-- delimiters.md => electronic-filings/delimiters.md | 0 headers.md => electronic-filings/headers.md | 0 glossary/committee.md | 0 glossary/contribution-limits.md | 0 glossary/filing.md | 0 8 files changed, 15 insertions(+), 6 deletions(-) rename bulk_detailed.md => bulk/detailed.md (69%) rename bulk_summary.md => bulk/summary.md (59%) rename amendments.md => electronic-filings/amendments.md (58%) rename delimiters.md => electronic-filings/delimiters.md (100%) rename headers.md => electronic-filings/headers.md (100%) create mode 100644 glossary/committee.md create mode 100644 glossary/contribution-limits.md create mode 100644 glossary/filing.md diff --git a/bulk_detailed.md b/bulk/detailed.md similarity index 69% rename from bulk_detailed.md rename to bulk/detailed.md index bf1c791..89c4791 100644 --- a/bulk_detailed.md +++ b/bulk/detailed.md @@ -21,4 +21,9 @@ A new offering as of July 2012, [this file](http://www.fec.gov/finance/disclosur Committee Contributions To Candidates --------- -[This file](http://www.fec.gov/finance/disclosure/metadata/DataDictionaryContributionstoCandidates.shtml) contains one row for each contribution from a committee to a candidate and for each independent expenditure made by a committee about a candidate. It is a subset of the file containing all transactions between one committee and another. +[This file](http://www.fec.gov/finance/disclosure/metadata/DataDictionaryContributionstoCandidates.shtml) contains one row for each contribution from a committee to a candidate and for each independent expenditure made by a committee about a candidate. It is a subset of the file containing all transactions between one committee and another. This file can be used to calculate things such as which candidate got the most PAC money and which candidates have received contributions from a specific committee. + +Any Transaction from One Committee to Another +--------- + +[This file](http://www.fec.gov/finance/disclosure/metadata/DataDictionaryCommitteetoCommittee.shtml) contains a row for each transaction between two committees, regardless of whether either committee is a candidate committee or not. For example, this file includes contributions from a corporate PAC to a national party committee in addition to contributions to candidate committees. The list of [transaction types](http://www.fec.gov/finance/disclosure/metadata/DataDictionaryTransactionTypeCodes.shtml) describes the kind of transaction each row represents. The quirk here is that in many cases, both sides of the transaction get separate records: one for the committee making the contribution (usually beginning with 2X) and another for the committee receiving the contribution (usually beginning with 1X). Using this file for SQL queries means including transaction type in some manner in pretty much every query; which types you want depends on a couple of factors, such as timeliness (contributing committees can file before recipient committees) or completeness (a recipient committee usually is the definitive account of what it has gotten). \ No newline at end of file diff --git a/bulk_summary.md b/bulk/summary.md similarity index 59% rename from bulk_summary.md rename to bulk/summary.md index 9643147..1243b1d 100644 --- a/bulk_summary.md +++ b/bulk/summary.md @@ -1,11 +1,15 @@ Summary files ======== -The [summary files](http://www.fec.gov/finance/disclosure/ftpsum.shtml) include canonical election-cycle data for candidates and committees - one record for each candidate or committee per two-year election cycle, depending on the file selected. There are two candidate summary files -- one for campaigns that have elections in the current cycle, and one for all candidates no matter if they face election in the cycle or not -- and they can differ in amounts and timeliness. The [current campaigns file](ftp://ftp.fec.gov/FEC/webl12.zip) may be more timely, but also contains a single total for PAC contributions (compared to totals for different kinds of PACs in the other file) and some of its totals may contain double-counted transactions. +The [summary files](http://www.fec.gov/finance/disclosure/ftpsum.shtml) include canonical election-cycle data for candidates and committees - one record for each candidate or committee per two-year election cycle, depending on the file selected. There are two candidate summary files -- one for campaigns that have elections in the current cycle, and one for all candidates no matter if they face election in the cycle or not -- and they can differ in amounts and timeliness. -The [all candidates file](ftp://ftp.fec.gov/FEC/weball12.zip) can be updated slightly less frequently than the current campaigns one, but it contains more detailed breakdowns of certain types of transactions as noted above. The possibility of double-counting some kinds of transactions - transfers to and from authorized committees of a candidate - also exists. +The [current campaigns file](ftp://ftp.fec.gov/FEC/webl12.zip) may be more timely, but also contains a single total for PAC contributions (compared to totals for different kinds of PACs in the other file) and some of its totals may contain double-counted transactions. More at the [data dictionary](http://www.fec.gov/finance/disclosure/metadata/DataDictionaryWEBL.shtml). -The FEC previously used to generate summary files at the end of the election cycle for candidates and PACs that included totals for different types of PACs, but discontinued these files after the 2005-06 cycle. Files are available from 1979-80 through 2005-06. Party committee summary files exist from the 1991-92 cycle through the 2003-04 cycle. One of the most useful files, which contained a record for each combination of candidate recipient and PAC contributor/independent spender, covers the 1991-92 cycle through the 2001-02 cycle. A similar file exists for candidate-party activities during the same time period. +The [all candidates file](ftp://ftp.fec.gov/FEC/weball12.zip) can be updated slightly less frequently than the current campaigns one, but it contains more detailed breakdowns of certain types of transactions as noted above. The possibility of double-counting some kinds of transactions - transfers to and from authorized committees of a candidate - also exists. More at the [data dictionary](http://www.fec.gov/finance/disclosure/metadata/DataDictionaryWEBALL.shtml). + +The [PAC summary file](ftp://ftp.fec.gov/FEC/2012/webk12.zip) provides the latest summary information on political action and party committees, including independent expenditures. More at the [data dictionary](http://www.fec.gov/finance/disclosure/metadata/DataDictionaryWEBK.shtml). + +The FEC previously used to generate summary files at the end of the election cycle for candidates and PACs that included totals for different types of PACs, but discontinued these files after the 2005-06 cycle. Files are available from 1979-80 through 2005-06. Party committee-only summary files exist from the 1991-92 cycle through the 2003-04 cycle. One of the most useful files, which contained a record for each combination of candidate recipient and PAC contributor/independent spender, covers the 1991-92 cycle through the 2001-02 cycle. A similar file exists for candidate-party activities during the same time period. These summary files had been stored in fixed-width format but were converted to pipe-delimited format in late July 2012. diff --git a/amendments.md b/electronic-filings/amendments.md similarity index 58% rename from amendments.md rename to electronic-filings/amendments.md index 26f7ce1..bdb53df 100644 --- a/amendments.md +++ b/electronic-filings/amendments.md @@ -9,13 +9,13 @@ Once an amendment is filed, it replaces the original filing and any previous ame Is this an amendment? --------- -In the section on __header rows__, we learned that amendments will have a value in the Report Number field of the first line of the electronic filing. The value of the Report Number represents the number of times the original filing has been amended. The easiest way to determine whether any given filing is an amendment is to check whether the Report Number field is blank. If it is, we can move on, confident that this is an original filing. If the field contains a value, we know that the filing is an amendment. +In the section on [header rows](/electronic-filings/headers/), we learned that amendments will have a value in the Report Number field of the first line of the electronic filing. The value of the Report Number represents the number of times the original filing has been amended. The easiest way to determine whether any given filing is an amendment is to check whether the Report Number field is blank. If it is, we can move on, confident that this is an original filing. If the field contains a value, we know that the filing is an amendment. What's being amended? --------- -The section on __header rows__ also taught us that the Report ID field will contain a value on amended filings (and will be blank otherwise, just like the Report Number field): +The section on [header rows](/electronic-filings/headers/), also taught us that the Report ID field will contain a value on amended filings (and will be blank otherwise, just like the Report Number field): > If the filing is an amendment, the report ID will look like this: *FEC-763780* diff --git a/delimiters.md b/electronic-filings/delimiters.md similarity index 100% rename from delimiters.md rename to electronic-filings/delimiters.md diff --git a/headers.md b/electronic-filings/headers.md similarity index 100% rename from headers.md rename to electronic-filings/headers.md diff --git a/glossary/committee.md b/glossary/committee.md new file mode 100644 index 0000000..e69de29 diff --git a/glossary/contribution-limits.md b/glossary/contribution-limits.md new file mode 100644 index 0000000..e69de29 diff --git a/glossary/filing.md b/glossary/filing.md new file mode 100644 index 0000000..e69de29 From 7dd8a1d861d3f208a41628834affe934579a7b12 Mon Sep 17 00:00:00 2001 From: Derek Willis Date: Wed, 1 Aug 2012 21:49:33 -0400 Subject: [PATCH 11/13] updated local links --- overview.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/overview.md b/overview.md index 061809f..90ea45b 100644 --- a/overview.md +++ b/overview.md @@ -25,7 +25,7 @@ Even though committees file reports [on a regular schedule](http://www.fec.gov/i Bulk FTP Data ======== -The FEC has offered bulk data for years, and [its offerings](http://www.fec.gov/finance/disclosure/ftp_download.shtml) include summary and detailed files covering committees, candidates and contributions. The bulk files are updated weekly, late on Sunday nights, so depending on your timing and needs the bulk files may not be suitable for every task. The advantage of the bulk files is that they are vetted by the FEC, with some of the records standardized (by adding FEC-issued committee ids, for example) and others removed to prevent duplicate records from appearing. The [summary](/bulk/summary/) and [detailed](/bulk/detailed/) files are pipe-delimited, a relatively recent change for the FEC, and some older files may still be in fixed-width format. +The FEC has offered bulk data for years, and [its offerings](http://www.fec.gov/finance/disclosure/ftp_download.shtml) include summary and detailed files covering committees, candidates and contributions. The bulk files are updated weekly, late on Sunday nights, so depending on your timing and needs the bulk files may not be suitable for every task. The advantage of the bulk files is that they are vetted by the FEC, with some of the records standardized (by adding FEC-issued committee ids, for example) and others removed to prevent duplicate records from appearing. The [summary](bulk/summary/) and [detailed](bulk/detailed/) files are pipe-delimited, a relatively recent change for the FEC, and some older files may still be in fixed-width format. Bulk data files are contained inside zip files stored on the FTP server, so retrieving them via a web application requires several steps. The FTP data is updated early Monday morning each week, and previous cycles are updated as well, since committees can amend filings from an earlier election cycle. From 943ebc0f646e8299d89c14b0dac886d2f0b583e1 Mon Sep 17 00:00:00 2001 From: Derek Willis Date: Wed, 1 Aug 2012 21:52:01 -0400 Subject: [PATCH 12/13] updated local links --- electronic-filings/amendments.md | 4 ++-- overview.md | 12 ++++++------ 2 files changed, 8 insertions(+), 8 deletions(-) diff --git a/electronic-filings/amendments.md b/electronic-filings/amendments.md index bdb53df..33409c5 100644 --- a/electronic-filings/amendments.md +++ b/electronic-filings/amendments.md @@ -9,13 +9,13 @@ Once an amendment is filed, it replaces the original filing and any previous ame Is this an amendment? --------- -In the section on [header rows](/electronic-filings/headers/), we learned that amendments will have a value in the Report Number field of the first line of the electronic filing. The value of the Report Number represents the number of times the original filing has been amended. The easiest way to determine whether any given filing is an amendment is to check whether the Report Number field is blank. If it is, we can move on, confident that this is an original filing. If the field contains a value, we know that the filing is an amendment. +In the section on [header rows](electronic-filings/headers.md), we learned that amendments will have a value in the Report Number field of the first line of the electronic filing. The value of the Report Number represents the number of times the original filing has been amended. The easiest way to determine whether any given filing is an amendment is to check whether the Report Number field is blank. If it is, we can move on, confident that this is an original filing. If the field contains a value, we know that the filing is an amendment. What's being amended? --------- -The section on [header rows](/electronic-filings/headers/), also taught us that the Report ID field will contain a value on amended filings (and will be blank otherwise, just like the Report Number field): +The section on [header rows](electronic-filings/headers.md), also taught us that the Report ID field will contain a value on amended filings (and will be blank otherwise, just like the Report Number field): > If the filing is an amendment, the report ID will look like this: *FEC-763780* diff --git a/overview.md b/overview.md index 90ea45b..baceda9 100644 --- a/overview.md +++ b/overview.md @@ -3,29 +3,29 @@ Overview The Federal Election Commission offers three general types of downloadable campaign finance data: individual electronic filings in CSV format covering most committees, pipe-delimited bulk itemized data and summary files via FTP from all committees covering two-year election cycles and specialized CSV or XML files via its [Data Catalog](http://fec.gov/data/DataCatalog.do?format=html). Electronic filings are available in real time as they are filed, while the other two types are updated on a regular basis. -The base element of federal campaign finance data is the [committee](/glossary/committee/). Committees are the recipient of money and the spenders of money, and there are a number of different kinds of committees. Although most FEC rules apply to all committees, some committees have different limits or rules to follow, and these matter. +The base element of federal campaign finance data is the [committee](glossary/committee.md). Committees are the recipient of money and the spenders of money, and there are a number of different kinds of committees. Although most FEC rules apply to all committees, some committees have different limits or rules to follow, and these matter. -The second element is the [filing](/glossary/filing/), a report by a committee to the FEC. There are many different kinds of filings, but in general they cover 3 different types of information: +The second element is the [filing](glossary/filing.md), a report by a committee to the FEC. There are many different kinds of filings, but in general they cover 3 different types of information: 1. The formation of committees and candidacies, and their details. 2. The raising of money for campaigns. 3. The spending of money on campaigns. -There is a consistent schedule for filings covering the last two types, and filings of the first type can be made at any time. In general, the FEC recognizes a roughly two-year election cycle that corresponds to U.S. House elections (where all 435 seats are up for election every 2 years). In the case of senators, who are elected to six year terms on a staggered basis, an election cycle may be considered to include the previous four years as well as the current two-year period. The election cycle is used not only as a logical boundary for calculating money raised and spent but also to calculate [contribution limits](/glossary/contribution-limits/) for candidates and committees. +There is a consistent schedule for filings covering the last two types, and filings of the first type can be made at any time. In general, the FEC recognizes a roughly two-year election cycle that corresponds to U.S. House elections (where all 435 seats are up for election every 2 years). In the case of senators, who are elected to six year terms on a staggered basis, an election cycle may be considered to include the previous four years as well as the current two-year period. The election cycle is used not only as a logical boundary for calculating money raised and spent but also to calculate [contribution limits](glossary/contribution-limits.md) for candidates and committees. Electronic Filings ======== Most committees registered with the FEC are required to file all of their reports electronically. There are two significant exceptions: committees of U.S. Senate candidates (and two senatorial party committees), which file with the Secretary of the Senate, and [committees that raise or spend less than $50,000 in a calendar year](http://fec.gov/ans/answers_filing.shtml#Do_I_need_to_file_electronically), or expect to do so. Electronic filing applies to U.S. House and presidential candidate committees, non-candidate political committees (typically knows as PACs), national, state and local political party committees registered with the FEC and individuals or organizations engaged in independent spending. Electronic filing was mandated in 2001. -Electronic filings can be found via the [FEC's search form](http://www.fec.gov/finance/disclosure/efile_search.shtml), which has a number of options for filtering the results to a particular committee, a particular date or more. The search results include the option to View or Download individual filings, which present the data in HTML and delimited (see [delimiters](/electronic-filings/delimiters/)) formats, respectively. Individual filings contain all of the records for that filing, stacked on top of each other in varying delimited layouts ([A zip file containing the formats is on FEC.gov](http://www.fec.gov/elecfil/eFilingFormats.zip).) +Electronic filings can be found via the [FEC's search form](http://www.fec.gov/finance/disclosure/efile_search.shtml), which has a number of options for filtering the results to a particular committee, a particular date or more. The search results include the option to View or Download individual filings, which present the data in HTML and delimited (see [delimiters](electronic-filings/delimiters.md)) formats, respectively. Individual filings contain all of the records for that filing, stacked on top of each other in varying delimited layouts ([A zip file containing the formats is on FEC.gov](http://www.fec.gov/elecfil/eFilingFormats.zip).) -Even though committees file reports [on a regular schedule](http://www.fec.gov/info/report_dates.shtml), electronic filings occur nearly every day of the year. Some are amendments of previous filings, others are filed in advance (or after) a deadline, and others are filed as changes warrant. Filings that are [amendments](/electronic-filings/amendments/) are indicated in the data, and serve as complete replacements for the original filings. +Even though committees file reports [on a regular schedule](http://www.fec.gov/info/report_dates.shtml), electronic filings occur nearly every day of the year. Some are amendments of previous filings, others are filed in advance (or after) a deadline, and others are filed as changes warrant. Filings that are [amendments](electronic-filings/amendments.md) are indicated in the data, and serve as complete replacements for the original filings. Bulk FTP Data ======== -The FEC has offered bulk data for years, and [its offerings](http://www.fec.gov/finance/disclosure/ftp_download.shtml) include summary and detailed files covering committees, candidates and contributions. The bulk files are updated weekly, late on Sunday nights, so depending on your timing and needs the bulk files may not be suitable for every task. The advantage of the bulk files is that they are vetted by the FEC, with some of the records standardized (by adding FEC-issued committee ids, for example) and others removed to prevent duplicate records from appearing. The [summary](bulk/summary/) and [detailed](bulk/detailed/) files are pipe-delimited, a relatively recent change for the FEC, and some older files may still be in fixed-width format. +The FEC has offered bulk data for years, and [its offerings](http://www.fec.gov/finance/disclosure/ftp_download.shtml) include summary and detailed files covering committees, candidates and contributions. The bulk files are updated weekly, late on Sunday nights, so depending on your timing and needs the bulk files may not be suitable for every task. The advantage of the bulk files is that they are vetted by the FEC, with some of the records standardized (by adding FEC-issued committee ids, for example) and others removed to prevent duplicate records from appearing. The [summary](bulk/summary.md) and [detailed](bulk/detailed.md) files are pipe-delimited, a relatively recent change for the FEC, and some older files may still be in fixed-width format. Bulk data files are contained inside zip files stored on the FTP server, so retrieving them via a web application requires several steps. The FTP data is updated early Monday morning each week, and previous cycles are updated as well, since committees can amend filings from an earlier election cycle. From 611f015ffb89c15b0b874a3289517c8aac0b16bd Mon Sep 17 00:00:00 2001 From: Derek Willis Date: Wed, 1 Aug 2012 22:12:05 -0400 Subject: [PATCH 13/13] added note about leadership pacs to ccl description --- bulk/detailed.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/bulk/detailed.md b/bulk/detailed.md index 89c4791..9df732e 100644 --- a/bulk/detailed.md +++ b/bulk/detailed.md @@ -16,7 +16,7 @@ Known as the [candidate master file](http://www.fec.gov/finance/disclosure/metad Candidate Committee Linkage --------- -A new offering as of July 2012, [this file](http://www.fec.gov/finance/disclosure/metadata/DataDictionaryCandCmteLinkage.shtml) contains one row for each combination of candidate and committee during an election cycle. Candidates can have a primary committee, authorized committees, joint fundraising committees and other relationships. The linkage file helps keep track of all of those relationships. In particular, joint fundraising committees can benefit multiple candidates; previously the committee master file would only list one. +A new offering as of July 2012, [this file](http://www.fec.gov/finance/disclosure/metadata/DataDictionaryCandCmteLinkage.shtml) contains one row for each combination of candidate and committee during an election cycle. Candidates can have a primary committee, authorized committees, joint fundraising committees and other relationships. The linkage file helps keep track of all of those relationships. In particular, joint fundraising committees can benefit multiple candidates; previously the committee master file would only list one. Note: it does not include a link between candidates and their leadership committees. Committee Contributions To Candidates ---------