Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

As a registry user, I want to ingest supplemental metadata from external data sources #49

Open
jordanpadams opened this issue Jun 28, 2021 · 1 comment
Assignees

Comments

@jordanpadams
Copy link
Member

jordanpadams commented Jun 28, 2021

Motivation

...so that I can ingest supplemental metadata into the registry that is both archival (Product_Metadata_Supplemental) and non-archival (e.g. external databases) to make the information searchable and accessible.

Additional Details

Note: This ticket is the parent epic for the overarching design consideration, there are several sub-tickets for the individual data source implementations.

Some use cases we need to think about:

  1. Use the PDS4 product Product_Metadata_Supplemental to revise data in products in the archive. NOTE: Need to check with RMS whether or not these should be considered archival.
  2. Non-archival metadata - this is "augmented" or "supplemental" metadata that is generated after archiving that can help for search and/or improves the existing archive metadata. e.g. image tags indicating a crater exists in the image, or more detailed target information like Mars - Olympus Mons, or tracking service information like "here is this products DOI that was never registered)"
    a. should we just treat these the same as Product_Metadata_Supplemental? (depends if those are considered archival or not)
  3. PDS3 databases - how do we integrate with databases that contain PDS3 data?
    a. should we just treat these the same as Product_Metadata_Supplemental? (depends if those are considered archival or not)

Acceptance Criteria

See sub-tickets for individual data source acceptance criteria

Engineering Details

Design thoughts

Should we just use Product_Metadata_Supplemental as the model for this and then augment for the non-archival data sources? - it seems like we should require some sort of archive-worthy definition of the data to be ingested. this could almost be a one-to-one translation of Product_Metadata_Supplemental, but just make it non-archival.

For the external databases, and initial workaround, we may also be just using a CSV for the time being: #51, and implement a MySQL plugin down the road since it would require some additional data modeling and configuration to connect to the database.

@jordanpadams
Copy link
Member Author

completed #121. moving the handling of the rest of these use cases to next build.

@jordanpadams jordanpadams added B12.1 and removed icebox labels Oct 13, 2021
@jordanpadams jordanpadams added icebox and removed B12.1 labels Jan 20, 2022
@jordanpadams jordanpadams transferred this issue from NASA-PDS/pds-registry-app May 11, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Status: Backlog
Development

No branches or pull requests

2 participants