You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
After simplifying our test suite setup (issue #942) I ran the data validation tests, to make sure they still worked with the new setup. There were a few tables with more rows than expected because (I think) of the integration of the eia860m data through November 2020. These included plants_eia860, utilities_eia860, pu_eia860, and generators_eia860 which all would be expected to change with the addition of new generators.
However, there were some other data validation failures that don't really make sense. Null distributed_generation column in the MCOE output, and too many records in the generation_fuel_eia923 table, which should be tracked down:
FAILED test/validate/eia_test.py::test_minmax_rows[eia_annual-gf_eia923-1551264-1250340-104195] - ValueError: Too many records (128817>109404.75) in dataframe gf_eia923
FAILED test/validate/mcoe_test.py::test_no_null_cols_mcoe[eia_annual-mcoe-all] - ValueError: Null column: distributed_generation found in dataframe mcoe
FAILED test/validate/eia_test.py::test_minmax_rows[eia_monthly-gf_eia923-1551264-1250340-104195] - ValueError: Too many records (1545804>1312857.0) in dataframe gf_eia923
FAILED test/validate/mcoe_test.py::test_no_null_cols_mcoe[eia_monthly-mcoe-all] - ValueError: Null column: distributed_generation found in dataframe mcoe
The text was updated successfully, but these errors were encountered:
I added a list of deprecated columns to the mcoe null columns check, since there some generator_eia860 columns which only have data prior to 2008, the earliest year for which we can calculate the MCOE / fuel costs based on our current methods. This fixes the distributed_generation error.
The generation_fuel row counts have also been investigated and updated.
Created a list of deprecated columns which are skipped in the mcoe null
column validation, since they only exist in 2006 and earlier data, and
we can only calculate the MCOE for 2008 and later.
Updated the null column validation to check all columns before raising
an exception, so you can get a list of all of them in the error message.
Updated release notes to remove this piece of errata.
Closes#943
After simplifying our test suite setup (issue #942) I ran the data validation tests, to make sure they still worked with the new setup. There were a few tables with more rows than expected because (I think) of the integration of the
eia860m
data through November 2020. These includedplants_eia860
,utilities_eia860
,pu_eia860
, andgenerators_eia860
which all would be expected to change with the addition of new generators.However, there were some other data validation failures that don't really make sense. Null
distributed_generation
column in the MCOE output, and too many records in thegeneration_fuel_eia923
table, which should be tracked down:The text was updated successfully, but these errors were encountered: