publish fails with datetime of null

[matthewhanson]

The publish step will fail if datetime is null, even though that is allowable by the STAC spec (in which case start_datetime and end_datetime must be provided):

https://github.com/radiantearth/stac-spec/blob/master/item-spec/item-spec.md#datetime

[jkeifer]

@matthewhanson Do you know where this is failing and/or do you have an example log you could share?

[jkeifer]

Never mind, it looks like it must be here: https://github.com/cirrus-geo/cirrus-lib/blob/main/src/cirrus/lib/process_payload.py#L292.

[jkeifer]

@matthewhanson In this error case where datetime is not a property, would you want to

• use end_datetime in its place, retaining the attribute name datetime?
• use end_datetime and label the attribute end_datetime?
• either of the above using start_datetime?
• drop datetime but have both a start_datetime and end_datetime attribute?
• omit the datetime message attribute altogether?
• something else I haven't considered?

[matthewhanson]

Good question, I think go with

drop datetime but have both a start_datetime and end_datetime attribute

It's more complicated for the user, but nothing else would make any sense. If I'm subscribed to a topic and I have a particular range of dates in mind then I would want to match all messages where the start_datetime and end_datetime overlapped with my interval. A single datetime doesn't capture this, since it may lie outside the bounds, but the interval does overlap.

I think this is an open question with STAC API as well, which I'll bring up at stac-api-spec. If someone searches for a range of datetimes and the Items have a null datetime should it return matches whose interval overlaps? I think yes, and we'll have to figure out how best to implement it. My initial thought is that the API should always set a start_datetime and end_datetime, even if they are the same as datetime, so that internal db queries can just check for overlapping time intervals.

So maybe in a similar fashion the published SNS message should always set start and end so that subscriptions can only check for overlapping time intervals. I think AND/OR operators are possible with subscriptions but not sure of the syntax.

[jkeifer]

To summarize, it sounds like we should always set start_datetime and end_datetime, using datetime if one or both are missing, and we should also set datetime if it is included.

However, SNS to SQS supports at most 10 attributes, which we are bumping up against. After discussing, it might be worth considering a way for payloads to specify what attributes should be included, potentially also with support for defining custom attributes. We can then have this default set of attributes for backwards compatibility but also have a way out for situations where others are needed in the future.

[jkeifer]

Quick little nugget I just found: it appears when using raw mode for SNS -> SQS subscriptions, the attribute limit goes away.