Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Integrate EIA-861 2021 Early Release data #1921

Merged
merged 7 commits into from Sep 10, 2022
Merged

Integrate EIA-861 2021 Early Release data #1921

merged 7 commits into from Sep 10, 2022

Conversation

zaneselvans
Copy link
Member

Made some additional changes based off of #1920 since I cant push to the rmi-electricity repo.

Still need to debug the actual ETL, as this fails with a strange error that doesn't seem related to the table/column mapping.

pudl_out = PudlTabl(pudl_engine=pudl_engine)
sales_eia861 = pudl_out.sales_eia861()
StopIteration                             Traceback (most recent call last)
File ~/code/catalyst/pudl/src/pudl/workspace/datastore.py:370, in Datastore.get_unique_resource(self, dataset, **filters)
    369 try:
--> 370     _, content = next(res)
    371 except StopIteration:

StopIteration: 

During handling of the above exception, another exception occurred:

KeyError                                  Traceback (most recent call last)
File ~/code/catalyst/pudl/src/pudl/extract/excel.py:322, in GenericExtractor.load_excel_file(self, page, **partition)
    314 try:
    315     # eia860m exports the resources as raw xlsx files that are not
    316     # embedded in zip archives. To support this, we will first try
   (...)
    320     # TODO(rousik): if we can make it so, it would be useful to normalize
    321     # the eia860m and zip the xlsx files. Then we could simplify this code.
--> 322     res = self.ds.get_unique_resource(
    323         self._dataset_name, name=xlsx_filename
    324     )
    325     excel_file = pd.ExcelFile(res)

File ~/code/catalyst/pudl/src/pudl/workspace/datastore.py:372, in Datastore.get_unique_resource(self, dataset, **filters)
    371 except StopIteration:
--> 372     raise KeyError(f"No resources found for {dataset}: {filters}")
    373 try:

KeyError: "No resources found for eia861: {'name': nan}"

@zaneselvans zaneselvans added new-data Requests for integration of new data. eia861 Anything having to do with EIA Form 861 labels Sep 9, 2022
@codecov
Copy link

codecov bot commented Sep 9, 2022

Codecov Report

Merging #1921 (48cd28c) into dev (34a1f7a) will increase coverage by 0.0%.
The diff coverage is 100.0%.

@@          Coverage Diff          @@
##             dev   #1921   +/-   ##
=====================================
  Coverage   83.2%   83.2%           
=====================================
  Files         65      65           
  Lines       7397    7398    +1     
=====================================
+ Hits        6157    6158    +1     
  Misses      1240    1240           
Impacted Files Coverage Δ
src/pudl/metadata/sources.py 100.0% <ø> (ø)
src/pudl/workspace/datastore.py 67.9% <ø> (ø)
src/pudl/extract/eia861.py 96.6% <100.0%> (+0.1%) ⬆️

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

@zaneselvans zaneselvans marked this pull request as ready for review September 9, 2022 16:24
@zaneselvans zaneselvans added the rmi label Sep 9, 2022
@zaneselvans zaneselvans self-assigned this Sep 9, 2022
@zaneselvans
Copy link
Member Author

@arengel this seems to be working now if you want to give it a look.

@zaneselvans zaneselvans merged commit c734d05 into dev Sep 10, 2022
@zaneselvans zaneselvans deleted the eia861_2021er branch September 10, 2022 22:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
eia861 Anything having to do with EIA Form 861 new-data Requests for integration of new data. rmi
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants