ref_lid_* fields are not added to the Registry schema prior to load

[jordanpadams]

Checked for duplicates

No - I haven't checked

🐛 Describe the bug

When I loaded data with a ref_lid_* not already in the registry, the fields are not searchable.

🕵️ Expected behavior

I expected the fields to be searchable

📜 To Reproduce

This query for ref_lid_target works because we include that in our initial schema creation:

https://pds.nasa.gov/api/search/1/products?q=ref_lid_target%20eq%20%22urn:nasa:pds:context:target:planet.mercury%22

This query for ref_lid_associate does not (but should):
https://pds.nasa.gov/api/search/1/products?q=ref_lid_associate%20eq%20%22urn:nasa:pds:context:node:node.imaging%22

Same for ref_lid_data (and it should work):
https://pds.nasa.gov/api/search/1/products?q=ref_lid_data%20eq%20%22urn:nasa:pds:messenger_mdis_4001:bdr_rdr:mdis_bdr_256ppd_h07nw2%22

🖥 Environment Info

• Chrome Browser

📚 Version of Software Used

API v1.1, Harvest v3.7.6

🩺 Test Data / Additional context

No response

🦄 Related requirements

No response

⚙️ Engineering Details

No response

[tloubrieu-jpl]

Harvest need to dymanically update the list of ref_lid_* fields which are searchable whenever they are found in the products.
That should make the end of this configuration file https://github.com/NASA-PDS/registry-mgr/blob/main/src/main/resources/elastic/registry.json#L59 obsolete

[al-niessner]

@jordanpadams @tloubrieu-jpl

Just to be clear, we now want to scan every product for ref_lid_* and add them to the index when found? When we say product do we mean all products or product not to include bundles or collections?

[tloubrieu-jpl]

Hi @al-niessner ,
We mean products as for any class of products including bundles and collections.
And yes, we want all the ref_lid_* found (created by harvest?) to be added to the schema before the product is loaded to opensearch.

[al-niessner]

@tloubrieu-jpl

Did a find/grep of all java files in harvest and there is no creation of ref_lid done there. Turns out it is done in registry-common but very, very far away from any connection to a DB. So, the simplest thing is to push the check back into harvest where it knows its connection and do it there. Doing all products makes it much simpler. Thanks.

[tloubrieu-jpl]

That sounds good, thanks @al-niessner