Transform function for AMI EIA861 #647
Conversation
Add manually compiled balancing authority id fixes
src/pudl/transform/eia861.py
Outdated
| .dropna(subset=["utility_id_eia"]) | ||
| .query("utility_id_eia not in (88888, 99999)") |
There was a problem hiding this comment.
Just checking, were these operations actually required for this table? Or is it just cut-and-paste from the sales_eia861 table?
There was a problem hiding this comment.
These were just cut and pasted from sales.
There was a problem hiding this comment.
I think we should probably remove any operations that aren't actually required.
src/pudl/transform/eia861.py
Outdated
| # Split the table into index, data, and "denormalized" columns for processing: | ||
| # Separate customer classes and reported data into a hierarchical index | ||
| logger.debug( | ||
| "Stacking EIA861 Advanced Metering Infrastructure data columns by customer class.") | ||
| data_cols = _filter_customer_cols(clean_ami, CUSTOMER_CLASSES) | ||
| data_cols.columns = ( | ||
| data_cols.columns.str.split("_", n=1, expand=True) | ||
| .set_names(["customer_class", None]) | ||
| ) | ||
|
|
||
| # Now stack the customer classes into their own categorical column, | ||
| data_cols = ( | ||
| data_cols.stack(level=0, dropna=False) | ||
| .reset_index() | ||
| ) | ||
| denorm_cols = ( | ||
| _filter_non_customer_cols(clean_ami, CUSTOMER_CLASSES) | ||
| .reset_index() | ||
| ) | ||
|
|
||
| # Merge the index, data, and denormalized columns back together | ||
| tidy_ami = pd.merge(denorm_cols, data_cols, on=idx_cols) | ||
|
|
||
| # Remove the now redundant "Total" records -- they can be reconstructed | ||
| # from the other customer classes. | ||
| tidy_ami = tidy_ami.query("customer_class!='total'") |
There was a problem hiding this comment.
If you notice that we're ending up doing exactly this set of operations for multiple tables, we should probably turn it into a local _helper_function() of some kind.
src/pudl/transform/eia861.py
Outdated
| ) | ||
| if dupes.any(): | ||
| raise AssertionError( | ||
| f"Found {dupes.sum()} duplicate rows in the Advanced Metring Infrastructure table, when zero were expected!" |
There was a problem hiding this comment.
I think dupes.sum() will add up the values in each of the columns, and return a Series here. Did you maybe mean len(dupes)?
There was a problem hiding this comment.
Hmm this is one of those weird things where I tried it with len(dupes) on my computer and it was throwing me errors so I changed it but now that I've gone back and tried it again len(dupes) seems to work. I'll change it back.
…unc for customer class tables.
| def _remove_total_col(df): | ||
| """Remove totals from customer class.""" | ||
| df = df.query("customer_class!='total'") | ||
| return df |
There was a problem hiding this comment.
Probably best to avoid functions that have only a single statement, unless it's a super opaque thing that becomes more readable by virtue of having a readable function name -- in this case just having a query() or drop() call inline is probably better.
Added Transform function for AMI and updated the totals comparison tool in the notebook.