Use entry_id rather than shortname for retrieving collection from CMR

[ckhowardASDC]

Using v0.3.0a14, get_collection_concept_id() uses a shortname to query CMR to get the concept_id if concept_id is not defined in the dataset-config.

When you have multiple versions of a collection, using the shortname can result in the wrong collection version being pulled, which may have different UMM-S and UMM-Var, and can result in incorrect imagery or a failing workflow. To ensure the correct concept_id is pulled, entry_id should be used instead of shortname.

This request is low-priority for us since we discovered you can add concept_id explicitly in the dataset-config. I've requested that be documented for the official 0.3.0 release.

[jackiryan]

Hi chris, could you explain in a little more detail what you mean by entry_id? When I look in the collections.umm_json for a specific collection, I see that EntryId appears as an element in the MetadataAssociations. Here's an example for the PREFIRE COG collection in ops:

        "MetadataAssociations": [
          {
            "Type": "INPUT",
            "EntryId": "PREFIRE_SAT2_AUX-MET",
            "Description": "PREFIRE_SAT2_AUX-MET is a collection of analysis and satellite data that provides essential information used to process Level 1B PREFIRE data to Level 2 products.",
            "Version": "R00"
          },
          {
            "Type": "INPUT",
            "EntryId": "PREFIRE_SAT2_1B-RAD",
            "Description": "This product is derived from mid and far-infrared data contained in PREFIRE_SAT2_1B-RAD.",
            "Version": "R00"
          },
          {
            "Type": "INPUT",
            "EntryId": "PREFIRE_SAT1_2B-MSK",
            "Description": "PREFIRE_SAT1_2B-MSK is used to identify clear an cloudy scenes.",
            "Version": "R00"
          },
          {
            "Type": "INPUT",
            "EntryId": "PREFIRE_SAT2_2B-CLD",
            "Description": "PREFIRE_SAT2_2B-CLD provides cloud properties inputs for PREFIRE_SAT2_2B-FLX cloudy sky flux calculations.",
            "Version": "R00"
          }
        ],

None of these names refer to the collection that I have pulled up the metadata for.

[jackiryan]

concept_id was added to the documentation for the readme for 0.3.0

I think that while concept id is venue-specific, it is also the most precise method of referring to a collection especially when multiple collections in UAT receive the same name.

[frankinspace]

I think @ckhowardASDC is referring to granuleId within the CMA message at cma/event/payload/granules[*]/granuleId.

This granuleId can be used for a search on readable_granule_name from CMR in combination with provider.

For example:
https://github.com/podaac/bignbit/blob/develop/tests/sample_messages/cma.uat.workflow-input.20210115090000-JPL-L4_GHRSST-SSTfnd-MUR-GLOB-v02.0-fv04.1.json#L97

https://cmr.earthdata.nasa.gov/search/granules.umm_json?readable_granule_name=20210115090000-JPL-L4_GHRSST-SSTfnd-MUR-GLOB-v02.0-fv04.1&provider=POCLOUD

So the change necessary would be here:

bignbit/bignbit/get_collection_concept_id.py

Lines 62 to 76 in f96dc62

	cmr_search_collections_url = f'https://cmr.{"uat." if cmr_environment == "UAT" else ""}earthdata.nasa.gov/search/collections.umm_json'
	edl_user, edl_pass = utils.get_edl_creds()
	token = utils.get_cmr_user_token(edl_user, edl_pass, cmr_environment)
	umm_json_response = requests.get(cmr_search_collections_url,
	headers={'Authorization': f'Bearer {token}'},
	params={
	'provider': cmr_provider,
	'short_name': collection_shortname
	},
	timeout=10)
	umm_json_response.raise_for_status()
	umm_json = umm_json_response.json()
	return umm_json['items'][0]['meta']['concept-id']

Altering this query to use granule id instead of provider/shortname combo.

FYI - This wasn't an issue for PO.DAAC because podaac puts the version number in the collection shortname for (all?) collections.

Related thought: this might also affect the naming of the configuration files since that also uses shortname to lookup the correct config file.

[ckhowardASDC]

I'm referring to this line of code: https://github.com/podaac/bignbit/blob/develop/bignbit/get_collection_concept_id.py#L40

Our original test with TEMPO data did not have concept_id defined in the dataset-config. We ingested a V03 granule, and get_collection_concept_id() queried CMR and returned the V04 collection concept_id and not the correct V03 collection concept_id. Specifically, it returned C1274178436-LARC_CLOUD instead of C1262899916-LARC_CLOUD when we ingested a NO2_L2_V03 granule. These are two different collections (because they're two different versions), even though both are in UAT LARC_CLOUD. Our bignbit run failed because that former concept_id has no UMM-S and UMM-Var associated with it even though we're passing variables for that collection, it should have returned the latter concept_id because that is what does have the UMM-S and UMM-Vars associated with it.

entryId = shortName_version, as far as I know it's the only way to guarantee you're pulling the correct concept_id out of CMR.

[jackiryan]

This is actually the exact reason I implemented the concept-id workaround. At your DAAC, the shortname is not specific enough to find the correct collection. In ambiguous cases where multiple collections' metadata are returned by the query, the get_collection_concept_id function will choose the first one. On the other hand, the shortname is not specified directly but rather comes from the filename of the dataset config itself. This won't work for our DAAC as far as I'm aware and the conflict is why I decided it would be best to just specify collection-id directly as a fallback.

To Frank's point, I don't think specifying granule id works in that query but even if it did it wouldn't make sense imo to put a random granule id into the dataset config as a proxy for finding the collection id.

[frankinspace]

I understand the issue is that provider+shortname does not uniquely identify a collection, it would need to be provider+shortname+version.

@ckhowardASDC do your CMA messages contain a key called entryId? Do you know how that makes it into the message? PO.DAAC's CMA messages do not contain entryId so I might be missing some context there.

@jackiryan the granule id wouldn't go in the dataset config, the lambda would look it up using the content of the input CMA message. As you pointed out earlier, though, the concept-id in dataset config is not ideal either because then you need to maintain environment-specific configurations. So I agree concept-id in config is a good thing for testing and as a temp fix. However, I think updating the code to be able to correctly identify which collection the granule belongs to automatically is still valuable.

[ckhowardASDC]

Yeah that was good foresight on your part, Jackie.

And yeah this is a tricky problem because different DAACs may pass different messages to the bignbit workflow. We don't have an entryId key anywhere but there are some different places in our message where a name/dataType can be combined with version to create the entryId.

Because you're getting the UMM-G prior to that GetCollectionConceptId task, you could use the CollectionReference in the UMM-G maybe? e.g.:

"CollectionReference": {
        "ShortName": "PREFIRE_SAT2_2B-FLX_COG",
        "Version": "R01"
      }

I'd assume that's reliable since CollectionReference is required in the UMM-G metadata. I don't know if it adds any complexity because now you need to have that UMM-G task precede the GetCollectionConceptId task. I've seen this kind of assumption / requirement bite other Cumulus Core tasks.

Otherwise, looking at our CMA message, I think it might be possible to construct the entryId from two places:

1. event['cma']['event']['meta']['collection']['name'] (and ['version'])
2. event['cma']['event']['meta']['input_granules']['dataType'] (and ['version'])

#2 is probably a better option than #1 because technically I think the name/version defined in the collection config could possibly not match what's in CMR. For example, if a collection config has a name 'a' and version '01', and its regexes match files for an incoming CNM message that specifies name 'TEMPO_NO2_L2' and version 'V03', the granules could potentially still get ingested into the CMR TEMPO collection, it'd just be under that a_01 Cumulus collection maybe. Could be wrong.

I'm not sure where the dataType/version come from for the input_granules field.

[frankinspace]

Because you're getting the UMM-G prior to that GetCollectionConceptId task, you could use the CollectionReference in the UMM-G maybe?

I think that's a good call @ckhowardASDC
get_granule_umm_json already runs before get_collection_concept_id so the CollectionReference is available from event['cma']['event']['payload']['granule_umm_json']['CollectionReference'] by the time get_collection_concept_id runs.