As a node operator, I want to harvest and ingest a subset of a bundle based on existing registered data.

[tloubrieu-jpl]

For more information on how to populate this new feature request, see the PDS Wiki on User Story Development:

https://github.com/NASA-PDS/nasa-pds.github.io/wiki/Issue-Tracking#user-story-development

Motivation

...so that I can ingest just the "new" data to the registry. This should also improve performance for the tools.

Additional Details

Currently we can load a full PDS4 archive or bundle, into the registry but in case of updated bundle or archive we would like to help the user to only load the updated part of the archive rather than 1. letting him select which files have been updated 2. reload the full archive.

Acceptance Criteria

Given a bundle that has previously been registered, but now has 1 or more new products to register
When I perform harvest / registry manager
Then I expect to only harvest and ingest those 1 or more new products

Engineering Details

This EPIC is likely to imply the creation of new repository enabling both harvest and registry-manager functions and their synchronization with the elasticsearch content.

[tloubrieu-jpl]

@tdddblog I created the task as an EPIC (it did not fit into an existing EPIC), you can add tasks under if you feel like it needs to be broken up.

[jordanpadams]

@tdddblog here is another one we should consider

[jordanpadams]

already implemented through using the the bundle section of the config file.

[rchenatjpl]

@jordanpadams @tdddblog @tloubrieu-jpl
I think this issue is getting resolved via the collection filter, which works well for NAIF, so I'm sorry the sample bundle I gave you was NAIF-based. Collections more typically grow by adding new products into the subdir, appending to collection.tab (which consists entirely of Primary LIDVIDs), and incrementing collection.xml's VID. So if I filter using <bundle>/<collection lidvid="something::8.0"/>, nothing gets excluded. (I'm assuming this. I didn't test this)

So I tried to filter via <harvest>/<registry>, but when I harvested a collection, added to it, incremented the collection VID, and re-harvested, the entire collection (not just the new products) showed up in /tmp/harvest/out/registry-docs.json. Since I really, really want to be done with testing, I'm calling this issue fixed because other, unchanged collections did not show up in the .json.

By the way, https://nasa-pds.github.io/pds-registry-app/operate/harvest.html# should indicate that <registry> only works with <bundles>. In email, I proposed changes to that doc, but apparently github is the way to get things done.

My suggestion: I believe this issue asks for a product-by-product filter. That may be too costly performance- or people-wise. If so, feel free to ignore this comment.

[jordanpadams]

@rchenatjpl FYI, i updated you comment above to wrap you XML with backticks `. otherwise, the Github markdown sees a < and thinks it is HTML

[tdddblog]

@rchenatjpl Fixed. See pull request: NASA-PDS/harvest#51

[tloubrieu-jpl]

The code has been included in release pds-registry-app v0.3.2