Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Software appears to be excluding all the products associated with the collections #69

Closed
jordanpadams opened this issue Jun 19, 2020 · 2 comments · Fixed by #70
Closed
Assignees
Labels
bug Something isn't working

Comments

@jordanpadams
Copy link
Member

jordanpadams commented Jun 19, 2020

Describe the bug
When running deep archive on this EPOXI bundle (pds-gamma:/data/local/starbase/data/pds4/test-bundles/pds4-epoxi_mri-v1.0) all the data files under hartley2_photometry/data/ are missing.

To Reproduce
Steps to reproduce the behavior:

  1. Download the data from above
  2. Run pds-deep-archive -s PDS_SBN -b https://pdssbn.astro.umd.edu/holdings/ pds4-epoxi_mri-v1.0/bundle.xml

Expected behavior
the SIP and AIP should contain all the product references in pds4-epoxi_mri-v1.0/hartley2_photometry/inventory.csv

Note some of the weirdness with this bundle:

  • There is an overview.xml in the same directory as collection.xml so we need to make sure that file is included.
  • There are data and document sub-directories that both contain data from the inventory.csv (this is horrendous archiving, but it is what it is)
  • pds4-epoxi_mri-v1.0/hartley2_photometry/document/epoxi_photometry_v5.[xml|pdf] should actually not be included in the SIP/AIP because its LID does not match the LID in the inventory file

Refs #32

@jordanpadams jordanpadams added bug Something isn't working triage-needed and removed triage-needed labels Jun 19, 2020
@jordanpadams jordanpadams added this to the PDS.20 (ends 2020-07-01) milestone Jun 19, 2020
@jordanpadams jordanpadams changed the title Software does not include various data files from EPOXI bundle Software appears to be excluding all the products associated with the collections Jun 19, 2020
@jordanpadams
Copy link
Member Author

Update: It appears the software is only outputting bundles/collections and not any of the associated products. It appears all the files are getting written to the database, just not the AIP/SIP.

@nutjob4life
Copy link
Member

Note to self (i.e., @nutjob4life): we parse for "P" lines in any <file_name> so long as the root element is a <Product_Collection>. We don't have to key off of .tab files like we currently are doing.

nutjob4life added a commit that referenced this issue Jun 26, 2020
…ioned contain tables of additional primaries, but do assume that `<file_name>`s that are in `<Product_Collection>`s do
jordanpadams added a commit that referenced this issue Jul 1, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants