Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add minor inter table calc fixes #2635

Merged
merged 8 commits into from
Jun 8, 2023

Conversation

cmgosnell
Copy link
Member

@cmgosnell cmgosnell commented Jun 6, 2023

PR Overview

small pr to address some suggestions from #2622

additional changes this provoked:

  • renaming source_table -> source_tables (bc it is a list of tables - usually a list of 1 but still)
  • migrating the de-duplication of the metadata based on the xbrl_factoid out of process_xbrl_metadata into it own method. this is by default just a check/a no-op, but now three table transformers have bespoke version. This was necessary because we added a calc to fix to a table that had duplicative metadata, which broke the calc fixer bc it expects to be able to extract a calculation from the metadata based on an xbrl_factoid name. So now the deduping of the metadata happens within process_xbrl_metadata but before apply_xbrl_calculation_fixes

PR Checklist

  • Merge the most recent version of the branch you are merging into (probably dev).
  • All CI checks are passing. Run tests locally to debug failures
  • Make sure you've included good docstrings.
  • For major data coverage & analysis changes, run data validation tests
  • Include unit tests for new functions and classes.
  • Defensive data quality/sanity checks in analyses & data processing functions.
  • Update the release notes and reference reference the PR and related issues.
  • Do your own explanatory review of the PR to help the reviewer understand what's going on and identify issues preemptively.

Comment on lines 1725 to 1729
# these next two fixes are related. we're replacing
# commercial_and_industrial_sales which is from a different table
# with two swappable components from this table. There is no way rn
# to say swap thing A for thing B & C for two so we swap A for B and
# then add C.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't understand what this comment is saying. What are "two swappable components?"

We're deleting one component and adding two components, right?

It seems like it might be simpler to have the more atomic "addition" and "deletion" operations rather than always defaulting to "replace" which is a composite of a deletion and an addition. We could define a replacement method in terms of deletion and addition methods.

What happens when you replace a component that has a list of source tables with a component that doesn't have a list of source tables?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think what's happening is

# Replace commercial_and_industrial_sales which is reported in
# electricity_sales_by_rate_schedule_ferc1 with two components
# (small_or_commercial_sales... and large_or_industrial_sales...)
# from this table which should sum to the same amount as the value
# being replaced.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

code wise i hard agree that having "additions" and "deletions" would make way more sense. As a human conceptually it makes way more sense to say "replace A for B" (and A or B can be a null element which results in "add B" or "delete A" respective).

i only added that comment bc rn the fixes are setup as "replace A for B" not "replace A for B & C" which is what needed to happen here. but that needed to be added as "replace A for B" & "add C".

i'd love a deeper structural change for these calcs but that is for #2605.

src/pudl/transform/ferc1.py Outdated Show resolved Hide resolved
@zaneselvans zaneselvans added ferc1 Anything having to do with FERC Form 1 data-repair Interpolating or extrapolating data that we don't actually have. metadata Anything having to do with the content, formatting, or storage of metadata. Mostly datapackages. xbrl Related to the FERC XBRL transition labels Jun 6, 2023
@codecov
Copy link

codecov bot commented Jun 7, 2023

Codecov Report

Patch coverage: 100.0% and no project coverage change.

Comparison is base (794e827) 87.1% compared to head (af7aef7) 87.1%.

Additional details and impacted files
@@                Coverage Diff                @@
##           xbrl_meta_reshape   #2635   +/-   ##
=================================================
  Coverage               87.1%   87.1%           
=================================================
  Files                     84      84           
  Lines                   9903    9912    +9     
=================================================
+ Hits                    8627    8636    +9     
  Misses                  1276    1276           
Impacted Files Coverage Δ
src/pudl/transform/ferc1.py 96.6% <100.0%> (+<0.1%) ⬆️

☔ View full report in Codecov by Sentry.
📢 Do you have feedback about the report comment? Let us know in this issue.

@cmgosnell cmgosnell merged commit a218eb2 into xbrl_meta_reshape Jun 8, 2023
10 checks passed
@cmgosnell cmgosnell deleted the inter-table-calc-component-cleanup branch June 8, 2023 12:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
data-repair Interpolating or extrapolating data that we don't actually have. ferc1 Anything having to do with FERC Form 1 metadata Anything having to do with the content, formatting, or storage of metadata. Mostly datapackages. xbrl Related to the FERC XBRL transition
Projects
Archived in project
Development

Successfully merging this pull request may close these issues.

None yet

2 participants