Update archiver for EIA 176, write new archivers for Forms 191, 757A #267
Conversation
| @@ -27,7 +29,18 @@ def all_archivers(): | |||
| for module in module_names: | |||
| # AbstractDatasetArchiver won't know about the subclasses unless they are imported | |||
| __import__(module) | |||
| return AbstractDatasetArchiver.__subclasses__() | |||
|
|
|||
| def all_subclasses(cls): | |||
There was a problem hiding this comment.
@zschira This is the only part of the transition to subclassing the NGQV archiver that felt a bit questionable, but otherwise we wind up having a non-functional NGQV archiver class in the list and not including 191 or 757A. If you have suggestions on how to do this better I'm all ears.
There was a problem hiding this comment.
Hmm yeah it's a bit annoying, but I might just leave it. We could pull out the get_datasets and get_year_resource methods to be standalone functions that get called by the archivers, but I'm not sure if it's actually worth it
| return datasets | ||
|
|
||
| async def get_resources(self) -> ArchiveAwaitable: | ||
| """Download EIA 191 resources.""" |
There was a problem hiding this comment.
Probably want to make this docstring generic
| @@ -27,7 +29,18 @@ def all_archivers(): | |||
| for module in module_names: | |||
| # AbstractDatasetArchiver won't know about the subclasses unless they are imported | |||
| __import__(module) | |||
| return AbstractDatasetArchiver.__subclasses__() | |||
|
|
|||
| def all_subclasses(cls): | |||
There was a problem hiding this comment.
Hmm yeah it's a bit annoying, but I might just leave it. We could pull out the get_datasets and get_year_resource methods to be standalone functions that get called by the archivers, but I'm not sure if it's actually worth it
Closes #262. Switch to scraping NGQV data, rather than relying on the bundled zipfile which requires manual line mapping. This PR produces one zipped CSV per year per form. This PR also reorganizes all EIA archivers into an
eiasubfolder to make subclassing the natural gas archiver logic more straightforward and to mirror thefercformat.Testing
See sandbox archives for Form 176, Form 191 and Form 757A.