ferc1 respondents using previous years' form -> data filed in wrong row_number

[zaneselvans]

plant_in_service_ferc1 table headers contain data

In the transform step of the plant_in_service_ferc1 table, there are a handful of cases in which the header rows contain data in the original database. This should never happen. All instances of this phenomenon are in the years 2004-2005, which is immediately after the table underwent a revision (2003). In addition, the numbers reported in the header rows for 2004 and 2005 are almost identical to each other, so it seems like maybe someone was just re-submitting the same form in both years, ignoring the fact that the form itself had changed.

This means that all of these records are probably bad. However, there are only a few dozen of them (out of ~30,000 total plant-years). It might be possible to recover this data, but it's probably not super

raw_dfs = {"plant_in_service_ferc1": pd.read_sql("f1_plant_in_srvce", ferc1_engine)}
transformed_dfs = {}
transformed_dfs = pudl.transform.ferc1.new_plant_in_service(raw_dfs, transformed_dfs)
pis_head = (
    transformed_dfs["plant_in_service_ferc1"]
    .set_index(["utility_id_ferc1", "report_year", "amount_type"])
    .filter(regex=".*_head$")
)
dude = pd.DataFrame()
for field in pis_head:
    bad_df = pis_head[pis_head[field].notnull()]
    dude = pd.concat([dude, bad_df])
(
    dude
    .reset_index()
    .drop_duplicates()
    .set_index(["utility_id_ferc1", "report_year", "amount_type"]).index
)

The indices of the resulting problematic records:

MultiIndex([(  7, 2004, 'starting_balance'),
            (  7, 2005, 'starting_balance'),
            ( 94, 2004, 'starting_balance'),
            ( 94, 2005, 'starting_balance'),
            (121, 2004, 'starting_balance'),
            (121, 2005, 'starting_balance'),
            (145, 2004, 'starting_balance'),
            (145, 2005, 'starting_balance'),
            (159, 2004, 'starting_balance'),
            (166, 2004, 'starting_balance'),
            (187, 2004, 'starting_balance'),
            (187, 2005, 'starting_balance'),
            (  7, 2004,   'ending_balance'),
            (  7, 2005,   'ending_balance'),
            ( 94, 2004,   'ending_balance'),
            ( 94, 2005,   'ending_balance'),
            (121, 2004,   'ending_balance'),
            (121, 2005,   'ending_balance'),
            (145, 2004,   'ending_balance'),
            (145, 2005,   'ending_balance'),
            (159, 2004,   'ending_balance'),
            (166, 2004,   'ending_balance'),
            (187, 2004,   'ending_balance'),
            (187, 2005,   'ending_balance'),
            ( 25, 2004, 'starting_balance'),
            ( 25, 2005, 'starting_balance'),
            ( 95, 2004, 'starting_balance'),
            (101, 2004, 'starting_balance'),
            (101, 2005, 'starting_balance'),
            (132, 2004, 'starting_balance'),
            (132, 2005, 'starting_balance'),
            (167, 2004, 'starting_balance'),
            (167, 2005, 'starting_balance'),
            (247, 2004, 'starting_balance'),
            ( 25, 2004,   'ending_balance'),
            ( 25, 2005,   'ending_balance'),
            ( 95, 2004,   'ending_balance'),
            (101, 2004,   'ending_balance'),
            (101, 2005,   'ending_balance'),
            (132, 2004,   'ending_balance'),
            (132, 2005,   'ending_balance'),
            (167, 2004,   'ending_balance'),
            (167, 2005,   'ending_balance'),
            (247, 2004,   'ending_balance'),
            (153, 2004, 'starting_balance'),
            (153, 2005, 'starting_balance'),
            (153, 2004,   'ending_balance'),
            (153, 2005,   'ending_balance')],
           names=['utility_id_ferc1', 'report_year', 'amount_type'])
``

[cmgosnell]

A new instance of this problem shows up in the income_statement_ferc1 table in 2003. I had to add a little cleanup method in transform.ferc1.IncomeStatementFerc1TableTransformer.process_dbf. Docs from that method:

    In 2003, two rows were added to the ``f1_income_stmnt`` dbf table, which bumped
    the starting ``row_number`` of ``f1_incm_stmnt_2`` from 25 to 27. A small
    handful of respondents seem to have not gotten the memo about this this in
    2003 and have information on these row numbers that shouldn't exist at all for
    this table.

These bad rows consist of 33% (!) of the respondents for that year.