-
-
Notifications
You must be signed in to change notification settings - Fork 110
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
standardize the calc checks for the total to subtotal calcs #2886
standardize the calc checks for the total to subtotal calcs #2886
Conversation
logger.info("Checking sub-total calcs.") | ||
subtotal_calcs = pudl.transform.ferc1.calculate_values_from_components( | ||
calculation_components=calculations_intertable[ | ||
calculations_intertable.is_total_to_subdimensions_calc | ||
], | ||
data=exploded, | ||
calc_idx=calc_idx, | ||
value_col=self.value_col, | ||
) | ||
subtotal_calcs = pudl.transform.ferc1.check_calculation_metrics( | ||
calculated_df=subtotal_calcs, | ||
value_col=self.value_col, | ||
calculation_tolerance=self.calculation_tolerance.intertable_calculation_errors, | ||
table_name=self.root_table, | ||
add_corrections=True, | ||
) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
there are no total to sub-dimension inter-table calcs rn but this felt like a good thing to run nonetheless
Codecov ReportAll modified lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## dev #2886 +/- ##
=======================================
- Coverage 88.6% 88.6% -0.1%
=======================================
Files 90 90
Lines 10809 10808 -1
=======================================
- Hits 9577 9576 -1
Misses 1232 1232
☔ View full report in Codecov by Sentry. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Definitely an improvement!
PR Overview
I don't love this overall... but it is a first attempt to standardize the way we are checking the total to subtotal calcs. I:
is_total_to_subdimensions_calc
bool column back into the calc components tablereconcile_table_calculations
) and in the explode step (inreconcile_intertable_calculations
) we now usecalculate_values_from_components
andcheck_calculation_metrics
twice: once for the "normal" calcs and once for the total-to-subdimension calcs.Notes:
is_total_to_subdimensions_calc
into the pks of the groupby and ran into a slight snag.. i still think its possible but i wanted to get a proof out before leaving.electric_plant_depreciation_change_ferc1
has... ALLL OF THE DIMENSIONS. and perhaps we are introducing some double counting into the calc components because of it? not sure at all just a hunch.PR Checklist
dev
).