3243 PHMSA transmission part J

[jdangerx]

Overview

Closes #3243.

• add yearly_miles_of_pipe_by_decade to file_map, page_map, page_part_map, skiprows, skipfooter
• add raw_phmsagas__yearly_miles_of_pipe_by_decade asset to ETL
• add column map file

Testing

How did you make sure this worked? How can a reviewer verify this?

I ran the ETL, then ran:

from dagster import AssetKey
from pudl.etl import def
asset = defs.load_asset_value(AssetKey("raw_phmsagas__yearly_miles_of_pipe_by_decade"))

And looked at some of the data:

The spike in gathering miles in 2022 appears to be due to "Onshore Type C" pipes finally getting reported starting in 2022:

Not sure if there's any specific tests I should write about the raw data.

To-do list

Edit tasklist title

Beta Give feedback Tasklist To-do list, more options

• Rename
• Copy markdown
• Delete tasklist

Delete tasklist

Delete tasklist block?

Are you sure? All relationships in this tasklist will be removed.

Cancel Delete

1. Make sure full ETL runs & `make pytest-integration-full` passes locally

   Make sure full ETL runs & make pytest-integration-full passes locally
   Options

   • Convert to issue
   • Toggle completion
   • Rename
   • Remove
2. For major data coverage & analysis changes, [run data validation tests](https://catalystcoop-pudl.readthedocs.io/en/latest/dev/testing.html#data-validation)

   For major data coverage & analysis changes, run data validation tests
   Options

   • Convert to issue
   • Toggle completion
   • Rename
   • Remove
3. If updating analyses or data processing functions: make sure to update or write data validation tests

   If updating analyses or data processing functions: make sure to update or write data validation tests
   Options

   • Convert to issue
   • Toggle completion
   • Rename
   • Remove
4. Update the [release notes](../docs/release_notes.rst): reference the PR and related issues.

   Update the release notes: reference the PR and related issues.
   Options

   • Convert to issue
   • Toggle completion
   • Rename
   • Remove
5. Review the PR yourself and call out any questions or issues you have

   Review the PR yourself and call out any questions or issues you have
   Options

   • Convert to issue
   • Toggle completion
   • Rename
   • Remove

Add item to To-do list

[e-belfer]

I've been using onshore_type_a rather than onshore_a, so we should pick one to be consistent (there will definitely be one last clean up PR, if we just want to make note of it in an issue I'm ok with that too). Otherwise all looks good here.

[jdangerx]

I'll make it onshore_type_a, it matches the PDF more closely!

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Comparison is base (eb739ed) 92.7% compared to head (5df0503) 92.7%.
Report is 4 commits behind head on main.

Additional details and impacted files

@@           Coverage Diff           @@
##            main   #3266     +/-   ##
=======================================
- Coverage   92.7%   92.7%   -0.0%     
=======================================
  Files        144     144             
  Lines      13091   13091             
=======================================
- Hits       12135   12134      -1     
- Misses       956     957      +1     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.