Create new pds-deep-archive program and improve performance #26
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
- Resolve #21 with a new driver program `aipsip` that generates both the AIP and uses it to make the SIP as well, leaving all in the current working directory (along with two—count 'em, *two*—PDS labels for the price of one!). - Updates the Python `setuptools` metadata to generate the new `aipsip` (helps with #21). - Refactors logging and command-line argument setup (also for #21). - Unifies logging between `aipgen` and `sipgen` with the new `aipsip` so that there are `--debug` and `--quiet` options; without either you get a nominal amount of "hand-holding" of output. - Resolve #13 so that instead of billions of redundant XML parsing and XPath lookups we use a local `sqlite3` database and LRU caching. - Factor out XML parsing from `aipgen` and `sipgen` so we can apply caching. - Clear up logging messages so we can know what's calling what. - Create a temp DB in `sipgen` and populate it with mappings from lidvids to XML files for rapid lookups - But see also #25 for other uses of that DB. - Add standardized `--version` arguments for all three programs. With these changes, running `sipgen` on my Mac¹ can process a 272GiB `insight_cameras` export in 1:03. On `pdsimg-int1`, it handles the 1.5TiB`insight_cameras` dataset in under 4 hours. Footnotes: - ¹2.4 GHz 8-core Intel Core i9, SSD - ²2.3 GHz 8-core Intel Xeon Gold 6140, unknown drive
|
Again, you can ignore the Packaging step of the check failing; it's because it ran earlier today, the file is date-stamped, and the test PyPI doesn't allow duplicate files to be uploaded. |
|
@nutjob4life could we change this to datetime? or too much of a pain? |
* After running validate, there were a few minor fixes that needed to be implemented. * Commented out / removed several CLI options for the time being until functionality is fully developed. * Updated file naming to take into the account bundle versioning separate from the AIP/SIP version * Updated docs per new pds-deep-archive script which combines aipgen and sipgen. Refs #21
jordanpadams
changed the title
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
aipsipthat generates both the AIP and uses it to make the SIP as well, leaving all in the current working directory (along with two—count 'em, two—PDS labels for the price of one!).setuptoolsmetadata to generate the newaipsip(helps with Develop one script to run both AIP and SIP generator #21).aipgenandsipgenwith the newaipsipso that there are--debugand--quietoptions; without either you get a nominal amount of "hand-holding" of output.sqlite3database and LRU caching.aipgenandsipgenso we can apply caching.sipgenand populate it with mappings from lidvids to XML files for rapid lookups--versionarguments for all three programs.With these changes, running
sipgenon my Mac¹ can process a 272GiBinsight_camerasexport in 1:03. Onpdsimg-int1, it handles the 1.5TiBinsight_camerasdataset in under 4 hours.Footnotes: