Rename MCOE and plant part list assets #2904

bendnorman · 2023-09-29T09:56:39Z

PR Overview

This PR:

Applies new naming convention to MCOE and PPL assets
Remove reference to mcoe in assets because they don't fully calculate mcoe yet. See discussion in Apply naming convention to output assets #2788
Adds Package.get_sorted_resourced() method so we can order the tables in the data dictionary and datasette.
Adds a dictionary of helper functions called JINJA_FILTERS to pudl.metadata.helpers that can be added to jinja environments. I added this because I was getting a malformed cross ref sphinx error on intermediate table names because of the preceding underscore.

PR Checklist

Merge the most recent version of the branch you are merging into (probably dev).
All CI checks are passing. Run tests locally to debug failures
Make sure you've included good docstrings.
For major data coverage & analysis changes, run data validation tests
Include unit tests for new functions and classes.
Defensive data quality/sanity checks in analyses & data processing functions.
Update the release notes and reference reference the PR and related issues.
Do your own explanatory review of the PR to help the reviewer understand what's going on and identify issues preemptively.

codecov · 2023-10-03T08:58:25Z

Codecov Report

Attention: 1 lines in your changes are missing coverage. Please review.

Comparison is base (7a7a441) 88.5% compared to head (9f578b3) 88.5%.

Additional details and impacted files

@@                Coverage Diff                 @@
##           rename-core-assets   #2904   +/-   ##
==================================================
  Coverage                88.5%   88.5%           
==================================================
  Files                      90      90           
  Lines                   10808   10819   +11     
==================================================
+ Hits                     9570    9580   +10     
- Misses                   1238    1239    +1

Files	Coverage Δ
src/pudl/analysis/allocate_gen_fuel.py	`91.3% <ø> (ø)`
src/pudl/analysis/mcoe.py	`97.4% <ø> (ø)`
src/pudl/analysis/plant_parts_eia.py	`96.5% <100.0%> (ø)`
src/pudl/metadata/helpers.py	`97.8% <100.0%> (+<0.1%)`	⬆️
src/pudl/metadata/resources/eia.py	`100.0% <ø> (ø)`
src/pudl/metadata/resources/eia860.py	`100.0% <ø> (ø)`
src/pudl/metadata/resources/mcoe.py	`100.0% <ø> (ø)`
src/pudl/output/eia.py	`59.0% <100.0%> (ø)`
src/pudl/output/pudltabl.py	`89.1% <ø> (ø)`
src/pudl/metadata/classes.py	`86.4% <90.0%> (-0.1%)`	⬇️

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

bendnorman · 2023-10-03T11:30:05Z

src/pudl/analysis/mcoe.py

        compute_kind="Python",
+        io_manager_key="pudl_sqlite_io_manager",
+        description=f"{agg_freqs[freq].title()} heat rate estimates by generation unit. Generation "


I added the descriptions to the assets because their metadata will be removed from pudl.metadata.resources when we deprecate PudlTabl.

Is this true for other assets created through asset factories, or is there something special about this set?

This is true for other intermediate assets that are currently written to the database because we provide access to them via PudlTabl. These assets will use the default IO manager when we deprecate PudlTabl.

There are a handful of other output intermediate assets that are being written to the database that don't have descriptions specified in the asset decorators. I can remove these descriptions for now and handle it when we deprecate PudlTabl.

bendnorman · 2023-10-03T11:35:38Z

src/pudl/metadata/classes.py

+        if exclude_intermediate_resources:
+            [
+                resource
+                for resource in self.resources
+                if not resource.name.startswith("_")
+            ]


Added this if we want to exclude intermediate assets from datasette. I think we should include them for now so the database, data dictionary and datasette are all consistent.

bendnorman · 2023-10-03T11:41:30Z

src/pudl/metadata/resources/mcoe.py

@@ -183,12 +183,10 @@
        for freq in AGG_FREQS
    }
    | {
-        f"mcoe_generators_{freq}": {
+        f"out_eia__{freq}_generators": {


We decided to add all generator attributes to this table so it's a one stop shop for users.

bendnorman · 2023-10-03T11:42:56Z

src/pudl/output/eia.py

@@ -114,7 +114,7 @@ def out_eia__yearly_plants(
    },
    compute_kind="Python",
 )
-def out_eia__yearly_generators(
+def _out_eia__yearly_generators(


I made this an intermediate table because the old mcoe_generators_{freq} table (now out_eia__{freq}_generators) has all of the same attributes plus the valuable derived attributes.

bendnorman · 2023-10-03T11:46:51Z

@ella the migrations changed on rename-core-assets when I merged dev in so you'll probably need to recreate you db.

e-belfer

One blocking question re the plant_parts asset name, otherwise just some comments and questions for clarification. Fast ETL worked perfectly for me out of the box.

docs/templates/package.rst.jinja

e-belfer · 2023-10-04T15:39:58Z

src/pudl/analysis/mcoe.py

        compute_kind="Python",
+        io_manager_key="pudl_sqlite_io_manager",
+        description=f"{agg_freqs[freq].title()} heat rate estimates by generation unit. Generation "


Is this true for other assets created through asset factories, or is there something special about this set?

src/pudl/analysis/mcoe.py

e-belfer · 2023-10-04T15:46:00Z

src/pudl/analysis/plant_parts_eia.py

    io_manager_key="pudl_sqlite_io_manager",
    compute_kind="Python",
 )
-def plant_parts_eia_asset(
-    mega_generators_eia: pd.DataFrame,
+def out_eia__plant_parts(


This asset name doesn't seem to fit with our existing model - doesn't it need some kind of asset type (probably yearly?).

asset type can be optional. I didn't think there was logical frequency or type for the table though I could be wrong. @cmgosnell and @katie-lamb what do you think?

I believe the plant parts are assigned annually, but correct me if I'm wrong @cmgosnell

it's an annual table! (it technically could be generated as a monthly table but it would be ginombo and rn this is mostly being used to link up to annual ferc data)

thanks! It has been renamed to out_eia__yearly_plant_parts.

src/pudl/etl/__init__.py

src/pudl/output/pudltabl.py

src/pudl/metadata/helpers.py

bendnorman · 2023-10-05T18:06:53Z

This is ready for another review @e-belfer!

e-belfer

Looks good and fast ETL runs as is.

bendnorman added 5 commits September 29, 2023 11:10

Rename mcoe and ppl assets

1aa5116

Merge branch 'rename-core-assets' into rename-mcoe-assets

01d3c73

Fix small ppl migration issue

f231452

Format and sort intermediate resource name cross refs in data dictionary

7b7dba1

Add upstream mcoe assets back to metadata

50cea89

bendnorman mentioned this pull request Oct 3, 2023

Remove references to mcoe #2910

Open

Update stragler PudlTabl method name

791a70b

bendnorman commented Oct 3, 2023

View reviewed changes

bendnorman requested a review from e-belfer October 3, 2023 11:46

bendnorman mentioned this pull request Oct 3, 2023

Apply naming convention to output assets #2788

Closed

e-belfer requested changes Oct 4, 2023

View reviewed changes

Add frequency to ppl asset name and some clean up

9f578b3

e-belfer self-requested a review October 5, 2023 18:10

e-belfer approved these changes Oct 6, 2023

View reviewed changes

bendnorman merged commit 1b5100e into rename-core-assets Oct 6, 2023
11 checks passed

bendnorman deleted the rename-mcoe-assets branch October 6, 2023 19:58

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Rename MCOE and plant part list assets #2904

Rename MCOE and plant part list assets #2904

bendnorman commented Sep 29, 2023 •

edited

codecov bot commented Oct 3, 2023 •

edited

bendnorman Oct 3, 2023

e-belfer Oct 4, 2023

bendnorman Oct 4, 2023

bendnorman Oct 3, 2023

bendnorman Oct 3, 2023

bendnorman Oct 3, 2023

bendnorman commented Oct 3, 2023

e-belfer left a comment

e-belfer Oct 4, 2023

e-belfer Oct 4, 2023

bendnorman Oct 4, 2023

e-belfer Oct 4, 2023

cmgosnell Oct 5, 2023

bendnorman Oct 5, 2023

bendnorman commented Oct 5, 2023

e-belfer left a comment

Rename MCOE and plant part list assets #2904

Rename MCOE and plant part list assets #2904

Conversation

bendnorman commented Sep 29, 2023 • edited

PR Overview

PR Checklist

codecov bot commented Oct 3, 2023 • edited

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bendnorman commented Oct 3, 2023

e-belfer left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bendnorman commented Oct 5, 2023

e-belfer left a comment

Choose a reason for hiding this comment

bendnorman commented Sep 29, 2023 •

edited

codecov bot commented Oct 3, 2023 •

edited