Add timestamp to SIP / AIP product LIDs and filenames #63
Comments
jordanpadams
added
enhancement
nutjob4life
added a commit
that referenced
this issue
Jun 14, 2020
…effort When SIP generation was originally done, the requirements weren't fully appreciated and they also spread fairly wide (examination of local filesystems versus querying a registry service, using pre-computed checksums (digests) in unspecified file formats, etc.) and AIP generation was purely based on directory contents and not label files. This commit fixes all that with a unified bundle comprehension method (using a temporary `sqlite3` database) that's shared between AIP and SIP and architected with multiprocessing capability for the future. It specifically addressess the following: - Resolves #24 by including support for the `--include-all-collections` argument for both AIP and SIP generation - Resolves #41 by building a structure for the PDS labels instead of blindly accepting what's in the bundle's directory structure - Resolves #65 by adding support for the `<directory_path_name>` element in XML labels - Resolves #64 by testing on the Insight Cameras dataset three times with reliable termination on the host `pds-dev-el7.jpl.nasa.gov` in approximately 1h 33m each time. - Resolves #63 by adding timestamps to logical identifiers and generated filenames. This commit also: - Updates the documentation with some additional notes and diagnostics produced by this version of the software - It also cleans up documentation with some cosmetic adjustments - It fixes the base URL described in the example usage (missing trailing slash) - Removes extraneous package dependencies - Factors bundle comprehension and database generation for shared use by AIP and SIP generation - Removed redundant database generation: if it's done for AIP just feed it into SIP - Adds functional tests for AIP generation - Factors SIP and AIP test cases for code reuse - Removes hundreds of lines of now redundant code
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Is your feature request related to a problem? Please describe.
Since people may be generating these products on bundles referencing collections by LID only, we need to add some other information to the product LIDs in order to ensure uniqueness. We could version the SIPs, but I think the best option is to add a timestamp (YYYYMMDD) to the
logical_identifierand filename for the SIP / AIP products so that they are unique. Otherwise, the software will produce the same files when run for each data release.e.g. :
with a LID of:
Open to other ideas?
Refs #24
The text was updated successfully, but these errors were encountered: