Integrate EIA-923 Annual Environmental Information (Schedule 8C) spreadsheet maps #2447

zaneselvans · 2023-03-23T19:28:07Z

PR Overview

A draft of spreadhseet mapping metadata for the EIA-923 Schedule 8C, which is about emissions control equipment. Based on @grgmiller's PR #1950.

See notes on issue #2448

PR Checklist

Merge the most recent version of the branch you are merging into (probably dev).
All CI checks are passing. Run tests locally to debug failures

codecov · 2023-03-23T20:56:40Z

Codecov Report

Patch coverage: 100.0% and no project coverage change.

Comparison is base (76a986f) 86.7% compared to head (abab544) 86.7%.

Additional details and impacted files

@@          Coverage Diff          @@
##             dev   #2447   +/-   ##
=====================================
  Coverage   86.7%   86.7%           
=====================================
  Files         81      81           
  Lines       9438    9438           
=====================================
  Hits        8183    8183           
  Misses      1255    1255

Impacted Files	Coverage Δ
src/pudl/metadata/fields.py	`100.0% <ø> (ø)`
src/pudl/extract/eia923.py	`100.0% <100.0%> (ø)`

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report in Codecov by Sentry.
📢 Do you have feedback about the report comment? Let us know in this issue.

grgmiller · 2023-03-24T15:29:43Z

@zaneselvans as you work on integrating emission control data, one thing to be aware of is that the raw EIA-923 data contains incomplete _control_id mapping. For example, some data rows that contain non-missing so2 control efficiencies have a missing so2_control_id. In OGE, we address this by filling in the missing control id with the control id for another pollutant (see: https://github.com/singularity-energy/open-grid-emissions/blob/bb0c0329c54b25cfc0e0c550ebf242bb7930f733/src/emissions.py#L1058) but there may be a better way to address this on the pudl side.

e-belfer

I'm running into the same 2018 file error that you were seeing earlier when materializing the raw assets in dagster:

The above exception occurred during handling of the following exception:
KeyError: "No resources found for eia923: {'name': 'EIA923_Schedule_8_Annual_Environmental_Information_2018_Final.xlsx'}"

For me, oddly, the 2018 file dagster just grabbed from 10.5281-zenodo.7236677 is 3.2mb, similar in size to the other years. But I get an error when I try to open it locally, suggesting some kind of file error?

The rest is just suggestions about naming conventions.

e-belfer · 2023-03-27T13:28:09Z

src/pudl/metadata/fields.py

@@ -1762,6 +1766,10 @@
        "type": "boolean",
        "description": "Is the reporting entity an owner of power plants reported on Schedule 2 of the form?",
    },
+    "pm_control_id_eia": {


We talked about using 'particulate' instead of 'pm' in the 860 PRs to avoid prime mover confusion. I think we should be consistent and name this particulate_control_id_eia, which is what is currently used in the 860 extract tables (but not yet present in fields.py as we haven't transformed them yet).

e-belfer · 2023-03-27T13:28:49Z

src/pudl/package_data/eia923/column_maps/emissions_control.csv

+report_year,year,year,year,year,year,year,year,year,year,year
+plant_id_eia,plant_id,plant_id,plant_id,plant_id,plant_id,plant_id,plant_id,plant_id,plant_id,plant_id
+environmental_equipment_name,name_of_environmental_equipment_or_technology_type,name_of_environmental_equipment_or_technology_type,name_of_environmental_equipment_or_technology_type,name_of_environmental_equipment_or_technology_type,name_of_environmental_equipment_or_technology_type,name_of_environmental_equipment_or_technology_type,name_of_environmental_equipment_or_technology_type,name_of_environmental_equipment_or_technology_type,name_of_environmental_equipment_or_technology_type,name_of_environmental_equipment_or_technology_type
+pm_control_id_eia,pm_control_id,pm_control_id,pm_control_id,pm_control_id,pm_control_id,pm_control_id,pm_control_id,pm_control_id,pm_control_id,pm_control_id


See comment above re: naming.

e-belfer · 2023-03-27T13:29:37Z

src/pudl/package_data/eia923/column_maps/emissions_control.csv

+hours_in_service,hours_in_service,hours_in_service,hours_in_service,hours_in_service,hours_in_service,hours_in_service,hours_in_service,hours_in_service,hours_in_service,hours_in_service
+annual_nox_emission_rate_lb_per_mmbtu,nox_emission_rate_entire_year_lbs_mmbtu,nox_emission_rate_entire_year_lbs_mmbtu,nox_emission_rate_entire_year_lbs_mmbtu,nox_emission_rate_entire_year_lbs_mmbtu,nox_emission_rate_entire_year_lbs_mmbtu,nox_emission_rate_entire_year_lbs_mmbtu,nox_emission_rate_entire_year_lbs_mmbtu,nox_emission_rate_entire_year_lbs_mmbtu,nox_emission_rate_entire_year_lbs_mmbtu,nox_emission_rate_entire_year_lbs_mmbtu
+ozone_season_nox_emission_rate_lb_per_mmbtu,nox_emission_rate_may_through_september_lbs_mmbtu,nox_emission_rate_may_through_september_lbs_mmbtu,nox_emission_rate_may_through_september_lbs_mmbtu,nox_emission_rate_may_through_september_lbs_mmbtu,nox_emission_rate_may_through_september_lbs_mmbtu,nox_emission_rate_may_through_september_lbs_mmbtu,nox_emission_rate_may_through_september_lbs_mmbtu,nox_emission_rate_may_through_september_lbs_mmbtu,nox_emission_rate_may_through_september_lbs_mmbtu,nox_emission_rate_may_through_september_lbs_mmbtu
+pm_emission_rate_lb_per_mmbtu,pm_emissions_rate_lbs_mmbtu,pm_emissions_rate_lbs_mmbtu,pm_emissions_rate_lbs_mmbtu,pm_emissions_rate_lbs_mmbtu,pm_emissions_rate_lbs_mmbtu,pm_emissions_rate_lbs_mmbtu,pm_emissions_rate_lbs_mmbtu,pm_emissions_rate_lbs_mmbtu,pm_emissions_rate_lbs_mmbtu,pm_emissions_rate_lbs_mmbtu


Same goes here and for all the records below

e-belfer · 2023-03-27T13:39:20Z

src/pudl/package_data/eia923/column_maps/emissions_control.csv

+so2_test_date,so2_test_date,so2_test_date,so2_test_date,so2_test_date,so2_test_date,so2_test_date,so2_test_date,so2_test_date,so2_test_date,so2_test_date
+fgd_sorbent_consumption_1000_tons,fgd_sorbent_quantity_thousand_tons,fgd_sorbent_quantity_thousand_tons,fgd_sorbent_quantity_thousand_tons,fgd_sorbent_quantity_thousand_tons,fgd_sorbent_quantity_thousand_tons,fgd_sorbent_quantity_thousand_tons,fgd_sorbent_quantity_thousand_tons,fgd_sorbent_quantity_thousand_tons,fgd_sorbent_quantity_thousand_tons,fgd_sorbent_quantity_thousand_tons
+fgd_electricity_consumption_mwh,fgd_electricity_consumption_megawatthours,fgd_electricity_consumption_megawatthours,fgd_electricity_consumption_megawatthours,fgd_electricity_consumption_megawatthours,fgd_electricity_consumption_megawatthours,fgd_electricity_consumption_megawatthours,fgd_electricity_consumption_megawatthours,fgd_electricity_consumption_megawatthours,fgd_electricity_consumption_megawatthours,fgd_electricity_consumption_megawatthours
+hg_removal_efficiency,mercury_removal_efficiency,mercury_removal_efficiency,mercury_removal_efficiency,mercury_removal_efficiency,mercury_removal_efficiency,mercury_removal_efficiency,mercury_removal_efficiency,mercury_removal_efficiency,mercury_removal_efficiency,mercury_removal_efficiency


Would be nice to have consistency and use either mercury or hg in our codes. So far, we've been using mercury (e.g. compliance_year_mercury).

zaneselvans · 2023-03-27T14:48:44Z

Ah yeah you're right I should update all the names. I just took the ones that were here already.

Weirdly, my issue with the 2018 data magically disappeared! I thought it must have been something screwy with my local setup, but if you're getting it now too then I don't know what's going on!

e-belfer · 2023-03-27T15:04:04Z

One more weirdness while I'm here! I think the 923 archives aren't a part of the Catalyst organization repo on Zenodo. If I trace the doi for the record I find them here:
https://zenodo.org/record/7682417

But they don't show up in this list: https://zenodo.org/communities/catalyst-cooperative/search?page=1&size=20#

Is there a reason for this, or is this just an oversight? If so, I can make a separate issue.

zaneselvans · 2023-03-27T15:27:59Z

No, there's no reason it's not in the community except that adding it to the community has to be done manually, rather than happening automatically when we create a new archive. See catalyst-cooperative/pudl-archiver#76

I went ahead and added it!

Also updated the naming to match what we did in EIA-860 EnviroEquip.

e-belfer · 2023-03-27T15:53:16Z

Re the 2018 data: In the issue you mentioned that using the most recent archive would require more retooling. Is this because we'd need to either update the data source for PUDL across the board (and probably break some other stuff) or write in a one-off exception for the source for this archive? I'd assumed that PUDL grabs the most recent archive by default, but maybe that's not the case? I'm a bit hesitant to merge in an extraction function with the extraction function commented out.

Deleting the raw data and redownloading from Zenodo didn't fix the issue. Did you change anything else in your local environment to fix the issue?

Otherwise running this for 2012-2017 on my local worked great.

zaneselvans · 2023-03-27T18:25:06Z

Which archive is used is hard-coded in pudl.workspace.datastore. Basically every update for every data source breaks the ETL because FERC & EIA are not even attempting to be consistent, so we can only update to new archives by hand. For instance, when I tried to use the more recent EIA-923 achive, I discovered that they've dropped the FIPS ID column for coalmines in the Fuel Receipts and Costs table (just between the "final" release of 2021 data and the revisions to that "final" data that have been made since October). There is no way to include data from more than one archive.

One thing that I have been messing around with locally is using Python 3.11. So maybe this problem is magically fixed in 3.11?! Seems unlikely.

I think it would be okay (though definitely not ideal) to merge this in now with the emissions_control page commented out in the extraction just to get the metadata into the system and not have it be dangling in a limbo-PR. I suspect that the next time we update the eia923 DOI it'll work just fine with no weirdness.

e-belfer

With the rollback of the commenting out for now, this looks good to merge. I'd suggest leaving a comment in-line both in the assets and in the extraction so nobody's confused about why we have an empty asset, though!

zaneselvans · 2023-03-27T19:11:16Z

I've commented out the emissions_control asset, but I'm not sure why this would prevent it from attempting to extract the emissions_control page. But I also can't make the page fail locally any more so it's hard for me to test what's going on.

Where are the 2 places (assets + extraction) that you're thinking of here?

Maybe I need to add emissions_control to the blacklisted_pages in the EIA923 excel extractor?

e-belfer · 2023-03-27T22:15:24Z

Adding emissions_control to the blacklist did the trick. Should be good to go!

zaneselvans · 2023-03-27T23:06:41Z

Ah okay great! Thank you! I'll merge once the CI passes. Such a weird issue.

zaneselvans added 4 commits March 23, 2023 10:49

Integrate file/column/tab maps for EIA923 Schedule 8.

265ff2b

Merge branch 'eia923-schedule8' into dagster-eia861

9cbcd87

Draft extract metadata for EIA-923 Schedule 8C: Air Emissions Control

732f814

Remove early_release column from map

c9024e0

zaneselvans marked this pull request as draft March 23, 2023 19:28

zaneselvans changed the title ~~Eia923 schedule8~~ Integrate EIA-923 Schedule 8 spreadsheet maps Mar 24, 2023

zaneselvans added the eia923 Anything having to do with EIA Form 923 label Mar 24, 2023

zaneselvans mentioned this pull request Mar 24, 2023

Integrate EIA-923 Annual Environmental Information (Schedule 8) #2448

Open

zaneselvans linked an issue Mar 24, 2023 that may be closed by this pull request

Integrate EIA-923 Annual Environmental Information (Schedule 8) #2448

Open

zaneselvans changed the title ~~Integrate EIA-923 Schedule 8 spreadsheet maps~~ Integrate EIA-923 Annual Environmental Information (Schedule 8C) spreadsheet maps Mar 24, 2023

zaneselvans marked this pull request as ready for review March 24, 2023 00:39

zaneselvans self-assigned this Mar 24, 2023

zaneselvans requested a review from cmgosnell March 24, 2023 00:40

This was referenced Mar 24, 2023

Add EIA-923 Schedule 8C (Air Emissions Controls) to PUDL #1950

Closed

Integrate EIA-860 Schedule 6 and EIA-923 Schedule 8 (Emissions Controls) #889

Closed

Comment out emissions_control extraction for now.

ec1847f

zaneselvans added this to the 2023Q1 milestone Mar 24, 2023

zaneselvans requested review from e-belfer and removed request for cmgosnell March 24, 2023 16:18

zaneselvans added the new-data Requests for integration of new data. label Mar 24, 2023

zaneselvans added 3 commits March 24, 2023 16:01

Merge branch 'dagster-eia861' into eia923-schedule8

abd6fd4

Merge branch 'dagster-eia861' into eia923-schedule8

bf7dd8a

Re-enable emissions_controls extraction b/c now it's working?

9554465

e-belfer requested changes Mar 27, 2023

View reviewed changes

Use longer column names we adopated in EIA-860 EnviroEquip tables.

efdad7f

zaneselvans requested a review from e-belfer March 27, 2023 15:42

e-belfer approved these changes Mar 27, 2023

View reviewed changes

Comment out emissions_control page again.

adc5768

Base automatically changed from dagster-eia861 to dev March 27, 2023 19:14

zaneselvans and others added 3 commits March 27, 2023 13:35

Merge branch 'dev' into eia923-schedule8

9254b8c

Add emissions_control to blacklisted pages

a49a6e9

Add comment in extractor on blacklist

abab544

zaneselvans merged commit b14a8c1 into dev Mar 28, 2023

zaneselvans deleted the eia923-schedule8 branch March 28, 2023 00:36

bendnorman mentioned this pull request Mar 29, 2023

Solve 2023-03-29 nightly build failure issues #2469

Merged

8 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Integrate EIA-923 Annual Environmental Information (Schedule 8C) spreadsheet maps #2447

Integrate EIA-923 Annual Environmental Information (Schedule 8C) spreadsheet maps #2447

zaneselvans commented Mar 23, 2023 •

edited

codecov bot commented Mar 23, 2023 •

edited

grgmiller commented Mar 24, 2023

e-belfer left a comment

e-belfer Mar 27, 2023

e-belfer Mar 27, 2023

e-belfer Mar 27, 2023

e-belfer Mar 27, 2023

zaneselvans commented Mar 27, 2023

e-belfer commented Mar 27, 2023

zaneselvans commented Mar 27, 2023

e-belfer commented Mar 27, 2023 •

edited

zaneselvans commented Mar 27, 2023

e-belfer left a comment •

edited

zaneselvans commented Mar 27, 2023

e-belfer commented Mar 27, 2023

zaneselvans commented Mar 27, 2023

Integrate EIA-923 Annual Environmental Information (Schedule 8C) spreadsheet maps #2447

Integrate EIA-923 Annual Environmental Information (Schedule 8C) spreadsheet maps #2447

Conversation

zaneselvans commented Mar 23, 2023 • edited

PR Overview

PR Checklist

codecov bot commented Mar 23, 2023 • edited

Codecov Report

grgmiller commented Mar 24, 2023

e-belfer left a comment

Choose a reason for hiding this comment

e-belfer Mar 27, 2023

Choose a reason for hiding this comment

e-belfer Mar 27, 2023

Choose a reason for hiding this comment

e-belfer Mar 27, 2023

Choose a reason for hiding this comment

e-belfer Mar 27, 2023

Choose a reason for hiding this comment

zaneselvans commented Mar 27, 2023

e-belfer commented Mar 27, 2023

zaneselvans commented Mar 27, 2023

e-belfer commented Mar 27, 2023 • edited

zaneselvans commented Mar 27, 2023

e-belfer left a comment • edited

Choose a reason for hiding this comment

zaneselvans commented Mar 27, 2023

e-belfer commented Mar 27, 2023

zaneselvans commented Mar 27, 2023

zaneselvans commented Mar 23, 2023 •

edited

codecov bot commented Mar 23, 2023 •

edited

e-belfer commented Mar 27, 2023 •

edited

e-belfer left a comment •

edited