You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
EIA 176 data was originally getting pulled from the bundled data, after investigation of the bulk API data proved it was not suitable. In #2603 (comment) we discussed updating the data source to query NGQV directly. This has now been implemented in catalyst-cooperative/pudl-archiver#267, producing 3 new archives with 1 zipped CSV per year (176, 191, 757A archives). The advantage of this format is that we:
don't have to hand-map line numbers to variables
can easily extract a subset of years, as we do with other EIA data
However, the CsvExtractor written by @davidmudrauskas will have to be slightly updated to handle this new data format. The extractor should be able to take a subset of years from the ETL settings, pass it through dataset_settings to the extractor, and extract and concatenate the relevant zipped CSVs into one dagster asset. There should be three extractors with essentially identical implementations of the archiver - 176, 191 and 757A.
The text was updated successfully, but these errors were encountered:
EIA 176 data was originally getting pulled from the bundled data, after investigation of the bulk API data proved it was not suitable. In #2603 (comment) we discussed updating the data source to query NGQV directly. This has now been implemented in catalyst-cooperative/pudl-archiver#267, producing 3 new archives with 1 zipped CSV per year (176, 191, 757A archives). The advantage of this format is that we:
However, the
CsvExtractor
written by @davidmudrauskas will have to be slightly updated to handle this new data format. The extractor should be able to take a subset of years from the ETL settings, pass it throughdataset_settings
to the extractor, and extract and concatenate the relevant zipped CSVs into one dagster asset. There should be three extractors with essentially identical implementations of the archiver - 176, 191 and 757A.The text was updated successfully, but these errors were encountered: