relate message to its associated discovery metadata

[tomkralidis]

It would be valuable to have an associated from a data notification to its related discovery metadata.

Options

Option 1

Add a link with the relevant URL to the GDC metadata record (implies infrastructure requirement from WIS2 Node):

{
    "rel": "publication",
    "href": "https://example.org/discovery-metadata/items/record123",
    "type": "application/json",
    "title": "Discovery metadata"
}

Option 2

Add a properties.metadata_id, consisting of only the identifier:

"properties": {
    "metadata_id": "record123"
}

Option 3

Potential to refactor properties.data_id and properties.metadata_id into properties.externalIds (lockstep with WCMP2):

"properties": {
    "externalIds": [{
        "scheme": "metadata",
        "value": "record123"
    }, {
        "scheme": "data",
        "value": "granule123"
    }]
}

Other options?

Additional information

Options 2 or 3 would facilitate a global broker adding a link object (to the GDC) as part of the message relay.

[golfvert]

Global Broker shouldn't alter the message, I don't think it is part of the "design". So, option 2 and 3, not sure.
Option 1, why not!

[tomkralidis]

For Option 1 then, the data notification would need to have logic to construct the correct GDC URL to the metadata record.

[tomkralidis]

cc @kaiwirt

Note: the GB would not alter the message, the collection ID would be the responsibility of the node. We could also consider the STAC approach of adding a collection property to the root of the message.

Providing the collection ID (i.e. the discovery metadata) would also help ensure/check that discovery metadata indeed exists for a given message/granule.

[kaiwirt]

I think Option 1 does not work, because a Node generating a message does not know the GDC URL for the metadata file. Additionally a GDC might want to update the URLs for some reason, so i would not consider these static.

For the same reason i think that the only entity that could potentially add the correct metadata link to the message is the GDC, which however is not in the message flow. A Broker should not modify the messages it forwards.

I prefer Option 2. Not sure about Option 3.

[antje-s]

Option 2 sounds good to me, if it is already added to origin-message.
A task is the assignment to the correct metadata id for different products at central pub function inside origin centre, but that's another topic...

[tomkralidis]

@antje-s @josusky associated PR in #31.

[josusky]

My concern is, that by adding this linkage to every data notification we are not lowering the barrier (to enter WIS) but heightening it. It is another element that needs to be appropriately maintained in the configuration of the notification publishing software and has little practical value. I think that the data_id (topicHierarchy) already provides sufficient linkage between the data and the metadata.

[tomkralidis]

@josusky note that adding properties.metadata_id is a recommendation (SHOULD), not a requirement (SHALL), and can act as a crosswalk/traceback to the actual dataset description as described in the global discovery catalogue. The value can also be used as a linkage to/for a Replay API of messages, back to the discovery metadata.

Another option is reducing to a permission (MAY), while still putting forth as a (future) KPI. In the cases where a WIS2 Node acts as an organization's central publishing facility, adding this value would be a useful option IMO.

[antje-s]

Since we didn't want to have a data flow (at least for core and globally distributed data) without metadata created, we need a way to check this. From my point of view, including the metadata_id in the notification allows such a check. However, I don't know if this check costs too much performance with increasing message volume. But our pilot phase is meant for such experiences...

[tomkralidis]

TT-WISMD 2023-03-08:

• TT agrees to SHOULD for the course of the pilot and reconsider/reviewed following the pilot phase
• we need to be careful that data does not go out without metadata (related: Make metadata a obligatory requirement wis2-guide#6), or else we will have the same issue we had in WIS1
• one approach could be a cache of GDC ids that would be used by GB
• can also be part of metrics/monitoring (offline, non-RT), or an entry point for WIS2 Node registration
• TT agrees to merge add properties.metadata_id #31