Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Create combined datapackage with all resources #1474

Closed
7 tasks
akariv opened this issue Aug 7, 2019 · 1 comment
Closed
7 tasks

Create combined datapackage with all resources #1474

akariv opened this issue Aug 7, 2019 · 1 comment

Comments

@akariv
Copy link
Member

akariv commented Aug 7, 2019

We want to create a data-package which contains links to all different resources in the same OS Package (e.g. one resource per uploaded year data file)

This datapackage would be stored on AWS and its URL would be placed in the origin-datapackage property in the Package Registry record (i.e. in elasticsearch).

  • Add to the publication phase another step
  • In this step, scan all existing configuration
  • Locate those with the same taxonomy_id and dataset-name
  • Calculate the location of their AWS data files (see issue Push data into AWS #1473)
  • Create datapackage with url-resources pointing to those resources
  • Push datapackage.json to AWS
  • Store resulting URL in package registry
@akariv akariv created this issue from a note in GIFT Phase IV (Current) Aug 7, 2019
@jobarratt jobarratt moved this from Current to Backlog in GIFT Phase IV Sep 2, 2019
@sglavoie
Copy link
Contributor

WONTFIX/CLEANUP: Closing old issue without recent activity (~1 year) as part of #1478. Data will be migrated and merged into the DataHub.io platform. GIFT Phase IV was closed. Please feel free to re-open if this is still relevant.

@sglavoie sglavoie added this to Closed in Cleanup July 2020 Jul 17, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
No open projects
Development

No branches or pull requests

3 participants