Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Refresh Zenodo Archives Manually #79

Closed
17 tasks done
Tracked by #61
zaneselvans opened this issue Feb 27, 2023 · 0 comments Β· Fixed by #82
Closed
17 tasks done
Tracked by #61

Refresh Zenodo Archives Manually #79

zaneselvans opened this issue Feb 27, 2023 · 0 comments Β· Fixed by #82
Assignees

Comments

@zaneselvans
Copy link
Member

zaneselvans commented Feb 27, 2023

The new archivers are working, but we don't have the infrastructure set up to run them on an automated schedule. Instead up them manually, adding new archive lineages for the new datasets

  • πŸ†• New Dataset
  • βœ… Updated Existing Archive
  • β›” No Update
  • πŸ’€ Archiver is broken
@zaneselvans zaneselvans self-assigned this Feb 27, 2023
zaneselvans added a commit that referenced this issue Feb 28, 2023
All of the other archivers use a file naming pattern like:

dataset-part1-part2.ext

where:
- dataset is the name of the dataset (e.g. ferc1, eia923, mshamines)
- part1 is the value of the first partition dimension contained in
  the file (e.g. ca, ny, wy, for states, 2019 if it's a year)
- part2 is the value of the second partition dimension contained in
  the file, if applicable.
- ext is the file name extension indicating file type.

I've changed the naming for phmsagas to follow the `dataset-*` part of
the naming convention, but have left the filename in place rather than
using only the dataset and start year, since if we just use the start
year it looks like years are missing in the archive, as each file
contains several years of data.

The dataset and start_year partitions are still used in datapackage.json
to refer to the individual resources for programmatic purposes.

This came up when working on #79
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Archived in project
Development

Successfully merging a pull request may close this issue.

1 participant