Issues with temporal data, including (open) ranges / periods

[m-mohr]

I have several issues with how temporal data is handled by WFS at the moment, which I'd like to draw your attention to and hopefully facilitate some fruitful discussion:

First issue: In several parts of the spec, e.g. OpenAPI spec and REQ_fc-time-response.adoc, it states that:

temporal information is either a date-time or a period string that adheres to RFC 3339.

Examples given are:

"2018-02-12T00:00:00Z/2018-03-18T12:31:12Z" or "2018-02-12T00:00:00Z/P1M6DT12H31M12S"

RFC3339 does not specify any of that. Those formats are specified in ISO8601.

Second issue: Open date ranges, e.g. 2018-02-12T00:00:00Z until now, are not possible throughout the spec, may it be for filtering or describing data. We have use cases where this is required as data updates very frequently and it is not really a solution to update the timestamps in the accommodating collection metadata every time. Also it seems useful to search for data that is just newer or older than a timestamp.

Third issue: The temporal filter is specified to be named time. I don't really like that, because it's not specified to be just a time, but a date and a time. That's counter-intuitive. I'd vote to call it datetime instead. This would also allow to have an extension, which would allow to just filter by time, e.g. give me all features that are available at 12:00 or between 09:00 and 12:00. There was a similar discussion on that topic by the STAC guys: radiantearth/stac-spec#130

Fourth issue: In several JSON schema and OpenAPI definitions a the format is set to dateTime, but both JSON Schema and OpenAPI propose that it should be date-time.

[m-mohr]

Regarding the first issue:

The introduction of RFC3339 explicetly mentions:

Date and time expressions indicate an instant in time. Description of time periods, or intervals, is not covered here.

The periods mentioned in the WFS spec are just mentioned in the Annex, the title is "ISO 8601 Collected ABNF". Not sure why the periods are in there, but for me it sounds strange to refer to / rely on something that was explicetly stated to be not covered in the main text of the RFC and is only mentioned in an Annex that tries to create an ABNF for another spec (ISO 8601).

[m-mohr]

Regarding the *second issue:

I think it could be an option to just allow omitting parts of the temporal extent.
For strings: "2018-02-12T00:00:00Z/"
For arrays: ["2018-02-12T00:00:00Z", null]

[rcoup]

Hmm, ISO8601 doesn't specify open-ended ranges, or how to do inclusive vs exclusive bounds... and the next release of the standard is still WIP: https://stackoverflow.com/a/48872488/2662

[m-mohr]

Thanks for commenting!
No, ISO8601 doesn't specify that (yet). Dublin Core took a try, but only specified it for dates: http://www.ukoln.ac.uk/metadata/dcmi/date-dccd-odrf/2005-08-13/
Nevertheless, it is useful for us to allow open date ranges, so it would be nice to come up with a solution. As pointed out with the other issues there are things to fix anyway. Would appreciate discussions on all these issues.

[rcoup]

Postgres’ ranges are extensive & feature-complete: https://www.postgresql.org/docs/10/static/rangetypes.html But null/infinity/inclusive/exclusive might be OTT for what we’re after here. _Personally_ `A <= t < B` (ie. A is inclusive and B is exclusive) is what our team has coalesced around for range specifies. Where `null` for A/B extends to infinity. eg. “August” is `2018-08-01, 2018-09-01`, regardless of whether the particular timestamps are passed at date/hour/minute/second/anything precision.

[cportele]

First issue:

RFC3339 does not specify any of that. Those formats are specified in ISO8601.

Good catch. The idea was to restrict date/timestamps to the profiles specified in RFC3339. We need to be more precise here.

Second issue:

I agree we need open ranges. In our implementation we use null values in the temporal extent for this, too. I need to do some research about a good approach for the time parameter. Any additional ideas welcome.

[cportele]

Third issue: I do not have a strong opinion on this one. datetime would be ok for me, too. Any other views on this?

[cportele]

Fourth Issue: Oops, thanks for pointing this out!

[m-mohr]

@cportele Thanks for answering this issue.

Second issue: Somebody pointed out to me that ISO is preparing a new draft of ISO8601 and that they will probably add open date ranges by replacing the missing date with two dots (e.g. 2018-02-12T00:00:00Z/..). Looks odd to me, but not sure about their reasons. Google already implemented this in their Dataset Search, see the temporalCoverage property.. So maybe just follow whatever ISO specifies for strings in the (date)time parameter and stay with the null in arrays for the temporal extent?

[cportele]

OK, if this is what ISO8601 will do, then this would seem to be the right approach. I will get more information on this and see how firm their ideas are.

[cportele]

In the CD version of 8601-2 from 2016 they list two options:

An enhanced time interval extends the specification of time interval in Part 1 of this standard (start and end date separated by ‘/’) by indicating that the start or end date is:
...

• Open Start or date open. The start or end date may be left blank, either because there is none or for any other reason.
  ...
  Example: 1985-04-12/
  Example: /1985-04-12

This is a level 1 extension. The second option is the two-dots one and is a level 2 extension:

For level 2, the dot notation “..” (two dots) may be used before a date to denote “before or on the date” or after a date to denote “on this date or after:
Examples
..1984 before or during the year 1984
1984.. during the year 1984 or later
..1984-04-04 before or on 4 April 1984

As this is from a draft in 2016 I need to find out, if this has changed in the meantime.

[tomkralidis]

+1 for:

starting from: time=YYYY-MM-DD/
until time=/YYYY-MM-DD

[rcoup]

@tomkralidis are they >=/<= or >/<?

[cportele]

@rcoup - I guess it is >= / <=.

From the time interval definition in ISO 8601:

A time interval comprises all instants between the two limiting instants and, unless otherwise stated, the limiting instants themselves.

[cportele]

By the way, schema.org seems to be going for "..": https://webschemas.org/temporalCoverage

[rcoup]

So IIUC (would be nice to find the latest/current ISO 8601 draft though) to specify "events on 1 August between 3pm and 4pm UTC" it'd be 2018-08-01T15:00Z/2018-08-01T15:59Z? And server-side you'd need to parse that into a datetime + a precision (ie. minutes in this example), then turn that into a filter that looks like (PostgreSQL for example's sake):

WHERE event >= '2018-08-01T15:00Z'::timestamp AND date_trunc('minute', event) <= '2018-08-01T15:59Z'::timestamp
-- or alternatively:
WHERE event >= '2018-08-01T15:00Z` AND event < '2018-08-01T16:00Z'

Likewise "any data from 1 August" would be 2018-08-01/2018-08-01, which translates to:

WHERE event >= '2018-08-01'::timestamp AND date_trunc('day', event) <= '2018-08-01'::timestamp
-- or alternatively:
WHERE event >= '2018-08-01' AND event < '2018-08-02'

[cportele]

I now have access to the latest drafts of ISO 8601-1/2. Indeed, it has changed and it would be the double-dot notation in time intervals (blank is now "unknown"):

An enhanced time interval extends the specification of time interval in Part 1 of this standard (start and end date separated by ‘/’) by indicating that the start or end date is:

• Unknown: Start or end date unknown. The start or end date may be left blank to indicate “unknown”.
• Open Start or End: Double-dot (..) may be used for the start/end date when there is no start/end date.

This is from the DIS version, which should be pretty stable. It should be final before in a few months.

So, I propose we support the Open Start or End notation.

[tschaub]

I think it would be a mistake to ignore the reason for RFC 3339:

Date and time formats cause a lot of confusion and interoperability
problems on the Internet.

and

There are many ways in which date and time values might appear in
Internet protocols [like ISO 8601]: this document focuses on just one
common usage, viz. timestamps for Internet protocol events

It feels like the direction that the discussion here is going is to say that the time parameter is either a (well constrained and well specified) RFC 3339 date-time or an (overly broad, poorly specified or yet to be specified) ISO 8601(-1 or -2?) interval.

In my opinion, allowing for any date/time representation in ISO 8601 is a mistake. Since it puts an undue burden on servers to parse all of what ISO 8601 allows (and there is legitimate confusion over exactly what is allowed), most will handle an unspecified subset, and clients cannot know what to expect in terms of support.

One alternative would be to define a syntax for intervals based on RFC 3339 alone. For example, in ABNF:

wfs-interval-explicit    = date-time "/" date-time
wfs-interval-open-start  = "../" date-time
wfs-interval-open-end    = date-time "/.."
wfs-interval             = wfs-interval-explicit / wfs-interval-open-start / wfs-interval-open-end
wfs-time                 = date-time / wfs-interval

Where the syntax for date-time comes from https://tools.ietf.org/html/rfc3339#section-5.6. This would give clients a fair bit of control and wouldn't impose an excessive parsing burden on servers.

Another alternative would be to not try to cram all time handling into a single parameter and to instead wait until a more general filter syntax is figured out (that would allow logical expressions and operators that could be used to query by a time range).

[cportele]

@tschaub - My intent was that only RFC 3339 is supported for instants and that we allow open intervals where each instant conforms to RFC 3339. I see that this wasn't that clear - and it should be explicitly stated in the requirements. Thanks.

I still consider it important to have a simple capability for temporal filtering in the Core (just like we have that for spatial filtering).

[pvretano]

08-AUG-2019: Clemens will create a pull request as per Clemens' conclusion from 27OCT2018 comment.