EIA860 2001-2003 #1122
EIA860 2001-2003 #1122
Conversation
Codecov Report
@@ Coverage Diff @@
## dev #1122 +/- ##
==========================================
+ Coverage 81.07% 81.10% +0.04%
==========================================
Files 48 48
Lines 6016 6054 +38
==========================================
+ Hits 4877 4910 +33
- Misses 1139 1144 +5
Continue to review full report at Codecov.
|
|
I was able to run the full ETL process on the 2001-2019 EIA years. I originally received a harvesting "inconsistent" error for @zaneselvans, what additional steps are needed to close out this issue and merge the PR? #1069 and running the data validation checks? |
…rds with missing nuc ids in test_nuclear_units_are_generators()
|
Check out this pull request on See visual diffs & provide feedback on Jupyter Notebooks. Powered by ReviewNB |
There was a problem hiding this comment.
I had a few minor questions but nothing blocking. This looks good to me. I can't remember, did you run the full integration tests? i.e. tox -e full_integration or tox -e full? That would exercise a lot of the output and analysis functions trying to make use of the earlier data.
src/pudl/extract/excel.py
Outdated
| @@ -236,11 +239,17 @@ def extract(self, **partitions): | |||
| if (len(self.METADATA._column_map[page].index) | |||
| + len(self.cols_added)) != len(df.columns): | |||
| # raise AssertionError( | |||
| # TODO (bendnorman): Enforce canonical fields for all raw fields? | |||
| # TODO (bendnorman): Doesn't the condition conflict with the warning message? Should the warning message include self.cols_added? | |||
There was a problem hiding this comment.
This looks like a bug and likely the source of some of the weird messages.
| @@ -452,6 +452,25 @@ def get_db_utils_eia(pudl_engine): | |||
| return db_utils_eia | |||
|
|
|||
|
|
|||
| def get_utility_most_recent_capacity(pudl_engine): | |||
| """Get a list of all utilities' most recent total net capacity of their generators.""" | |||
There was a problem hiding this comment.
It might not make sense to only look at the most recent capacities to rank the utilities, since we're sometimes mapping older entities. But I guess at this point usually it's a new year of data, so maybe it doesn't matter.
| energy_source_2_transport_3,,,,proposed_transportation_2c,proposed_transportation_2c,proposed_transportation_2c,,,,,,,,,,,,, | ||
| multiple_fuels,,,,,,,,,multiple_fuels,multiple_fuels,multiple_fuels,multiple_fuels,,,,multiple_fuels,multiple_fuels,multiple_fuels,multiple_fuels | ||
| ownership_code,owner,owner,owner,owner,owner,owner,ownership,ownership,ownership,ownership,ownership,ownership,ownership,ownership,ownership,ownership,ownership,ownership,ownership | ||
| turbines_num,propwind,propwind,propwind,proposed_turbines,proposed_turbines,proposed_turbines,turbines,turbines,turbines,turbines,turbines,turbines,,,,,,, |
There was a problem hiding this comment.
Are these early years referring to wind turbines? Is that true for the latter years as well?
|
Oh @bendnorman it looks like you need to merge |
* The new dbf extractor for the old eia860 years appears to depend on the xlsxreader package, which we didn't have a direct dependency on before. Not sure why this didn't break when @bendnorman ran the full ETL. I had recently removed this dependency since we weren't actually using it anywhere, so this seems like a stale test environment. But if the tests are run with tox they should recreate the environment as it is specified in setup.py. Anyway, I added xlsxreader back into the setup.py install_requires. * Fixed the datapackage merge errors that were happening because the EPA CEMS and EIA datapackages were using different years of EIA 860 data in the etl_full.yml settings. * The expected number of rows for some of the eia860 tables were wrong after the merge. I haven't fixed this because I'm going to almost immediately combine this branch with the direct SQL/Parquet branch, which also has a different number of rows. So I'll run the validation there instead. Closes #848
Previously we checked whether the **number** of columns extracted from Excel spreadsheets was what we expected, and if not, reported which extra columns were found in the data, but not in our maps of the data. But this is only one possible way that things can go wrong. I've updated the check to compare the actual columns of the extracted dataframe with those that we expect to see as sets, so if they're different but have the same number of elements, we'll still be notified. I've also updated it to report what partition of the data the error is happening in, and to report both when there are **extra** columns in the data that don't show up in our maps, and when there are columns **missing** from the data that we expect to see, based on our maps. And it's possible to get both kinds of warnings at the same time, since we're looking at set differences, rather than just the number of elements.
Added a DBF reader to GenericExtractor to handle the early years of EIA data.