Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add multi-year partitions to PHMSA archiver #199

Closed
Tracked by #2848
e-belfer opened this issue Nov 22, 2023 · 0 comments · Fixed by #200
Closed
Tracked by #2848

Add multi-year partitions to PHMSA archiver #199

e-belfer opened this issue Nov 22, 2023 · 0 comments · Fixed by #200
Assignees
@e-belfer
Labels
phmsagas Pipelines and Hazardous Materials Safety Administration (PHMSA/USDOT) Annual Natural Gas Reports

Comments

@e-belfer
Copy link
Member

PHMSA archives contain zip files with multiple years of data, each as their own CSV. To make extraction possible, it's preferable to have one partition with start_year and a list of all included years of data, so that the extraction can be reconfigured to look for one zip file that contains multiple year partitions.
@e-belfer e-belfer added the phmsagas Pipelines and Hazardous Materials Safety Administration (PHMSA/USDOT) Annual Natural Gas Reports label Nov 22, 2023
@e-belfer e-belfer self-assigned this Nov 22, 2023
@e-belfer e-belfer linked a pull request Nov 23, 2023 that will close this issue
PHMSA archiver: add multi-year years partition, add new dataset #200
Merged
2 tasks
@e-belfer e-belfer mentioned this issue Nov 27, 2023
Closed
16 tasks
@e-belfer e-belfer mentioned this issue Jan 12, 2024
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@e-belfer e-belfer

Labels
phmsagas Pipelines and Hazardous Materials Safety Administration (PHMSA/USDOT) Annual Natural Gas Reports
Projects
Catalyst Megaproject
Archived in project
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

PHMSA archiver: add multi-year years partition, add new dataset catalyst-cooperative/pudl-archiver
1 participant
@e-belfer