Map PHMSA Natural Gas Transmission Part M columns

[cmgosnell]

Overview

Closes #3263.

What problem does this address?

What did you change?

Testing

How did you make sure this worked? How can a reviewer verify this?

To-do list

Edit tasklist title

Beta Give feedback Tasklist To-do list, more options

• Rename
• Copy markdown
• Delete tasklist

Delete tasklist

Delete tasklist block?

Are you sure? All relationships in this tasklist will be removed.

Cancel Delete

1. Make sure full ETL runs & `make pytest-integration-full` passes locally

   Make sure full ETL runs & make pytest-integration-full passes locally
   Options

   • Convert to issue
   • Toggle completion
   • Rename
   • Remove
2. Review the PR yourself and call out any questions or issues you have

   Review the PR yourself and call out any questions or issues you have
   Options

   • Convert to issue
   • Toggle completion
   • Rename
   • Remove
3. For major data coverage & analysis changes, [run data validation tests](https://catalystcoop-pudl.readthedocs.io/en/latest/dev/testing.html#data-validation)

   For major data coverage & analysis changes, run data validation tests
   Options

   • Convert to issue
   • Toggle completion
   • Rename
   • Remove
4. If updating analyses or data processing functions: make sure to update or write data validation tests

   If updating analyses or data processing functions: make sure to update or write data validation tests
   Options

   • Convert to issue
   • Toggle completion
   • Rename
   • Remove
5. Update the [release notes](../docs/release_notes.rst): reference the PR and related issues.

   Update the release notes: reference the PR and related issues.
   Options

   • Convert to issue
   • Toggle completion
   • Rename
   • Remove

Add item to To-do list

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Comparison is base (4528907) 92.7% compared to head (d3a6f45) 92.7%.
Report is 1 commits behind head on main.

Additional details and impacted files

@@          Coverage Diff          @@
##            main   #3270   +/-   ##
=====================================
  Coverage   92.7%   92.7%           
=====================================
  Files        144     144           
  Lines      13091   13091           
=====================================
  Hits       12134   12134           
  Misses       957     957           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[e-belfer]

• Do we want to call the main table yearly_transmission_failures_leaks_repairs?
• Columns in this table definitely go back all the way to 1990. See Part C/D/E in the 2005 form and the 1988 form as well.
• Just a note that I've been using either other or class_3_4_only to refer to nonhca_nonmca sections. See the advantages of both but we should pick one to use consistently. Otherwise column names look good.

[cmgosnell]

@e-belfer do you know that class 3 & 4 are always non-mca/non-hca? Or is it a square and rectangle thing? I tried to read about it and couldn't find a clear a == b although they are clearly related - in that they are about how important/regulated the land is where the infrastructure is located. But this part M has class 1 & 2/non-mca/non-hca.

[e-belfer]

@e-belfer do you know that class 3 & 4 are always non-mca/non-hca? Or is it a square and rectangle thing? I tried to read about it and couldn't find a clear a == b although they are clearly related - in that they are about how important/regulated the land is where the infrastructure is located. But this part M has class 1 & 2/non-mca/non-hca.

This has almost certainly changed overtime, but more recently it seems like operators can either 1) blanket define all Class 3&4 locations as HCAs, or use a formula provided by PHMSA (which would explain the existence of non-MCA/HCA Class 3&4 locations. Class 1 & 2 locations could also be HCAs or MCAs if they meet the criteria (which I think essentially boils down to "many human lives but slightly further away from the pipeline than would the criteria for a Class 3/4". (Source is p.8 here: https://www.phmsa.dot.gov/sites/phmsa.dot.gov/files/docs/technical-resources/pipeline/gas-transmission-integrity-management/62271/faqsgas-transmission-integrity-management201904.pdf)

[cmgosnell]

it seems like there is mostly overlap but I am starting on part Q now and it has all classes 1-4 associated with in HCA, in MCA or not in HCA or MCA. So I don't think it is a good idea to assume there is a perfect overlap.

[cmgosnell]

or @e-belfer is your suggestion that we use other in place of my vv long nonhca_nonmca? that feels good and fine to me. conflating the classes and high/mid consequence areas feels less clear to me.

[e-belfer]

or @e-belfer is your suggestion that we use other in place of my vv long nonhca_nonmca? that feels good and fine to me. conflating the classes and high/mid consequence areas feels less clear to me.

Yes using "other" instead of "nonhca_nonmca" is my suggestion, I would definitely not conflate the classes and MCA/HCA.

[e-belfer]

One column mapping I would like you to fix, a question re doe, and a response to your question. Otherwise thank you for handling this giant table, the asset generates in dagster as expected.

[e-belfer]

Great, thanks! One question I'll punt to #3277

[e-belfer]

We don't keep filing_date in the other tables other than A-D, I believe. But I also see an argument for keeping it. Let's merge this for now and take this up in #3277, as it'll affect all the tables we've integrated up until now.

[cmgosnell]

hm yea good point! but agreed let's punt it. I'll add a quick comment over there about all of the date fields.