ISO 8601-2:2019

[justingrant]

I just learned that a new iteration of the ISO 8601 standard that was released last year. https://www.iso.org/standard/70908.html

Are there any changes in the standard that are significant for Temporal?

[justingrant]

Spec abstract is below. I bolded the parts that looked most interesting.

This document specifies additional representations of dates of the Gregorian calendar and times based on the 24-hour clock that extend the basic rules and composite elements of those defined in ISO 8601-1. These representations are specified as character strings for use in information interchange. It is also applicable for representing times and time shifts based on Coordinated Universal Time (UTC).

These extensions include:
— uncertain or approximate dates, or dates with portions unspecified;
— extended time intervals;
— divisions of a year;
— sets and choices of calendar dates;
— grouped time scale units;
— repeat rules for recurring time intervals; and
— date and time arithmetic.

[justingrant]

BTW, downloading the standard costs a few hundred dollars. (!!!!)

[ptomato]

I hope @sffc doesn't mind me reposting this from the chat, where he listed some highlights from his reading of the standard:

• Negative durations are supported. They use the popular "-P2M1D" syntax. However, they also support mixed signs in the duration. They lay out an algorithm for how to resolve the mixed signs.
• Durations allow fractional values; for example, "P1M2.5D" is valid.
• New support for centuries ("16C" = 1600-1699), decades ("196J" = 1960-1969), and day of year ("350O" = 350th day of the year)
• New negative sign for relative days and weeks: "-3W" = third to last week of the year, such that you can do "1985Y-3W" to mean "third to last week of 1985"
• New syntax for BCE: "12YB" = 12 BCE = -11Y in ISO-8601
• Uncertainty: use "X" to stand in for a digit to signify that you are uncertain about it. '1390YXXM' expresses a two-digit calendar month in 1390. You can also express uncertainty using significant digits, tildes, percentages, and a number of other methods. A large portion of the spec is focused on this subject.
• Exponential notation (scientific notation) is allowed for representing years
• Month numbers 20 and up are defined to mean year subdivions; for example, "2001-21" represents "Spring of 2001". Subdivisions include seasons, quarters, quadrimesters, and semestrals.
• Grouped units: allows for more complex expressions such as "the second fortnight of February 2018"
• Syntax to support sets and ranges of dates: "{1667, 1668, 1670..1672}" is a set of years
• New syntax for dates and times: "1985Y4M12D" == "1985-04-12"
• Date intervals (start date plus duration) make a comeback
• New syntax for recurring date intervals

[ptomato]

And here are my opinions on whether the above changes are relevant...

Negative durations are supported. They use the popular "-P2M1D" syntax. However, they also support mixed signs in the duration. They lay out an algorithm for how to resolve the mixed signs.

As long as we have negative durations, then we should probably check whether we can conform to their algorithm to resolve mixed signs.

Durations allow fractional values; for example, "P1M2.5D" is valid.

It doesn't seem like we would be able to support this on years, months, weeks, or days, and still be consistent with what we have.

New syntax for dates and times: "1985Y4M12D" == "1985-04-12"

Does this seem like something that we want to support? Maybe it would be good to see if it gains traction as an interchange format first. It's something that could always be added as a follow-on proposal. (If we don't support this, then a lot of the other items become moot.)

New support for centuries ("16C" = 1600-1699), decades ("196J" = 1960-1969), and day of year ("350O" = 350th day of the year)

We don't have separate types for centuries or decades, so I don't think this is actionable. Maybe ordinal days would be relevant, although we already didn't support the existing 2019-350 for Temporal.Date (350th ordinal day of year 2019).

New negative sign for relative days and weeks: "-3W" = third to last week of the year, such that you can do "1985Y-3W" to mean "third to last week of 1985"

We don't have Temporal.YearWeek, so I don't think this is actionable. 2019Y-10O might be relevant if we support the new syntax, but then again we already didn't support constructing a Temporal.Date from the existing year/ordinal day notation.

New syntax for BCE: "12YB" = 12 BCE = -11Y in ISO-8601

Would only be relevant if we supported the new syntax. I'm indifferent about it otherwise.

Uncertainty: use "X" to stand in for a digit to signify that you are uncertain about it. '1390YXXM' expresses a two-digit calendar month in 1390. You can also express uncertainty using significant digits, tildes, percentages, and a number of other methods. A large portion of the spec is focused on this subject.

I thought the X had actually been in the previous version of the standard. Anyway, we don't have this concept in Temporal (although I suppose we might say that things like Temporal.YearMonth.from('2020-08-XX') are OK?)

Exponential notation (scientific notation) is allowed for representing years

Seems trivial to support, why not?

Month numbers 20 and up are defined to mean year subdivions; for example, "2001-21" represents "Spring of 2001". Subdivisions include seasons, quarters, quadrimesters, and semestrals.

I don't think this is relevant to Temporal since we don't have things like Temporal.YearSeason and Temporal.YearQuarter.

Grouped units: allows for more complex expressions such as "the second fortnight of February 2018"

Ditto.

Syntax to support sets and ranges of dates: "{1667, 1668, 1670..1672}" is a set of years

Ditto, we don't have this concept either.

Date intervals (start date plus duration) make a comeback
New syntax for recurring date intervals

Intervals were already in the previous edition and we do not support them. (Although we have had one request that it would be useful, there are some practical concerns: #205)

[justingrant]

As long as we have negative durations, then we should probably check whether we can conform to their algorithm to resolve mixed signs.

@sffc - could you post an excerpt of the algorithm here? I'd like to understand how the spec expects constrain-balancing to work if the starting point date and time zone isn't known.

[sffc]

Section 4.4.1.9 defines the term "negative duration" as follows:

A duration in the reverse direction, called a "negative duration" in this document, can be expressed
using the following representation based on the [duration] representation specified in ISO 8601-1:2019,
5.5.2. In this case, all time scale components within the duration representation shall be positive.

duration(m) = [!]["-"][positiveDuration]

where [positiveDuration] is the representation of a positive duration.

However, Section 11, entitled "Explicit duration and extensions", is an entire chapter dedicated to the subject of durations with mixed signs. It defines a new term for it, "composite durations". Here is an excerpt:

11.1 General

Duration can be expressed by a combination of time scale units in explicit form — years, months,
weeks, days, hours, minutes and seconds. The representations of duration in this clause are based on
CalConnect CC 18011, Clause 7, "Explicit duration".

These time scale components are specified in 4.3 and are used to specify positive duration.

11.2 Durational units

Time scale unit components are represented by a time scale component used to compose a duration
expression and are defined as "durational units".

...

11.3.2 Composite representation

The composite representation of a duration is a more flexible and relaxed specification for duration
than that of ISO 8601-1:2019, 5.5.2. It accepts all expressions of the duration representation given in
ISO 8601-1:2019, 5.5.2 and is given as follows.

[!]["-"]["P"][durationUnits(m)]

where [durationUnits(m)] contains time scale components for expressing (positive or negative)
duration (see 11.2).

That section goes on to define the syntax, which is the same as the syntax for "normal" (ISO 8601-1 style) durations, except that it explicitly allows mixed signs, such as:

EXAMPLE 5 'P-3M-3DT1H5M', duration of three months and three days in the reverse direction, with one hour
and five minutes in the original direction.
...
EXAMPLE 9 '-P8M-1D', duration in reverse, "eight months minus one day", is equivalent to 'P-8M1D', "eight
months ago with a day ahead"

The next section, "11.3.3 Precedence representation", further defines syntax with multiple durations in a sequence. It calls them "precedence durations". Example:

EXAMPLE 4 'P-3MP2DP-1Y' describes a duration of three months in the reverse direction, two days in the
original direction and one year in the reverse direction, to be evaluated in the order from left to right.

The algorithms for arithmetic are introduced in Section 14 and explained in detail in Annex D. Here is an example of duration-duration arithmetic from Section 14.2:

EXAMPLE 6 'P1Y10M3D - P2Y5MT10M' results in 'P3Y15M3DT-10M'

EXAMPLE 6 doesn't look right to me, but that's what it says.

Here is an example of datetime-duration arithmetic from Section 14.4, with a precedence durational unit, but not mixed signs:

EXAMPLE 3 The expression '2018Y9M10DT9H10M + P2MPT50S' is identical to the effect of
'(2018Y9M10DT9H10M + P2M) + PT50S', where the parentheses indicate precedence order.

Annex D is the part I think we're most interested in. It specifies the algorithm for datetime-duration arithmetic.

It defines two new operations on datetime types: "Carry-over of overflow in time scale components" (section D.3.1) and "Truncation at time scale component boundaries" (section D.3.2). Those two operations are then used in the algorithms, of which there are three: ones for simple durations (ISO 8601-1 style, but with only one time scale component; section D.4.1), composite durations (ISO 8601-2 style; section D.4.2), and precedence durations (section D.4.3).

Here is the algorithm for composite durations, from section D.4.2:

— For all values of time scale components in the composite duration, directly modify the values of the
corresponding time scale components in the date and time representation.
— In the resulting date and time representation, starting from the lowest order overflowed time scale
component, perform carry-over on all overflowed time scale components until there is no more
overflow in the representation.
— If any time scale components have become invalid, apply truncation to them.
— The resulting date and time representation is complete

"Carry-over" is what we call "balance", and "truncation" is what we call "constrain". They have special rules for leap months, leap seconds, etc.

I encourage you to obtain your own personal copy of ISO-8601-2 if you want more details. You can read the license agreement here:

https://www.iso.org/terms-conditions-licence-agreement.html

To my great disappointment, Annex D does not give any examples of how to resolve situations in which the time scale components become negative, even though my careful reading of the document strongly suggests that such a situation is possible. However, the spec references CalConnect CC 18011 multiple times when discussing these algorithms, so maybe that spec has some answers.

[justingrant]

Because I wasn't ready to pay $200 of my family's money for the ISO spec, I read CalConnect CC 18011 instead. ;-) Here's some notes:

• Overall Impressions - This standard includes a lot of formats. The logic required to parse and emit all these formats seems intimidating. A summary might be that many of the things we've been thinking of as only applying to code (e.g. centuries, 15-minute periods, order of operations, fractional units, etc.) are controllable in this standard via the persistence format. Comparing pre-2019 ISO 8601 to CalConnect CC 18011 feels like comparing circa-1998 CSS to CSS in 2020. Both are declarative but one of them feels much closer to a programming language than the other.
• The same formats are used for durations as are used for date/time values. So 2018Y1M1D instead of 2018-01-01
• 5. Grouped Units - this is how concepts like "the third quarter" or "the fifth 10-minute period" are serialized. I didn't notice if groups can be nested or not.
• 5.4.5.2. Truncation of partial units I believe this section discusses the case that @sffc was talking about in how to handle cases where a rounding increment isn't an exact divisor of the next-largest unit.
• Offsets can be any units, e.g. T23H20M50SZ-5H0M is "23:20:50 at local time, which is 5 hours behind UTC"
• 7.3.3. Precedence representation says that the order of operations is dependent on the order of units in the string. So P1M10D means "add 1 month and then add 10 days" while P10D1M means "add 10 days and then add 1 month".

the order of the [durC-i] components is used to determine the evaluation precedence order of the duration statement [durationP].

• I didn't see any mention of how subtraction should be handled, e.g. by reversing the order for subtraction like Temporal does.
• 8.4. Composite duration discusses the balancing algorithm. Still no discussing handling subtraction here either. ;-(
• "8.4. Composite duration" and "8.5. Precedence duration" discuss different algorithms but honestly I was confused. Why? Isn't a Composite duration just a special case of a precedence duration where the precedence matches ISO 8601?
• If units overflow without changing a unit, then truncate the overflowed unit. For example, adding P1M to 19981M31D would yield 19982M28D.
• 8.6. Involving fractional duration - Fractional units are supported too, e.g. P0.5M. An algorithm is provided for how to handle calculation with fractional units: first find the exact time in context, then apply the fraction.
• Absolutely nothing in the spec related to DST or time zones, other than how to append the offset. ;-( None of the DST-unsafe examples have any notes indicating that they're not safe for DST. e.g. https://standards.calconnect.org/csd/cc-18011.html#_77b61d3d-45bc-4a53-a557-c567c03ecf9a

Honestly I found this spec disappointing. It doesn't seem to offer resolution for any of the big open issues that we're trying to resolve with Temporal.

[sffc]

Okay, so it sounds like CalConnect CC 18011 has pretty much the same scope as ISO-8601-2, with the same types of answered and unanswered questions.

We have a liaison relationship with CalConnect in the works. Once that is established, then we can send our questions directly to the authors of that spec.

[ptomato]

I didn't realize until now that the 2019 edition split ISO 8601 into ISO 8601-1 and ISO 8601-2, where part 1 is "Basic rules" and part 2 is "Extensions". All the things above are from part 2, if I'm not mistaken, so it seems appropriate that we should be able to pick and choose which extensions we support.

[ronaldtse]

Hi guys, editor of ISO 8601-2 and CC 18011/18012 here, from CalConnect.

ISO 8601-2 is effectively a combination of three specs: Library of Congress' EDTF, CalConnect's CC 18011 and 18012. All of these are open specs so you're not missing out much on the ISO copy. (caveat below...)

Some replies:

I didn't see any mention of how subtraction should be handled, e.g. by reversing the order for subtraction like Temporal does.

Clause 14.2 of ISO 8601-2 (which is supposed to be CC 18011) deals specifically with temporal arithmetic including subtraction. I apologize here -- because 18011 was not updated to the latest text we contributed to 8601-2. We will update 18011 to reflect the missing changes. I can't just paste the images here due to copyright restrictions, but the text is from CalConnect.

A preview...

Absolutely nothing in the spec related to DST or time zones, other than how to append the offset. ;-( None of the DST-unsafe examples have any notes indicating that they're not safe for DST

Those examples were chosen to "hint" at the problem without naming it, because DST and timezones are out of scope of ISO 8601-2. In ISO 8601-1 and 8601-2 the only concept that exists is "time shift". This is intentional, because another project ISO 34200 (and the associated ISO 8601-3) is scoped to deal with that particular problem.

Last bit about some potential confusion -- ISO 8601-1 and ISO 8601-2 support the concept of "profiles" -- no one needs to support all the representations used in these documents. In ISO 8601-2, there is an Annex that provides the EDTF profile built upon ISO 8601-2 primitives (happy to share that via the CalConnect liaison).

[ronaldtse]

Answering @sffc

EXAMPLE 6 'P1Y10M3D - P2Y5MT10M' results in 'P3Y15M3DT-10M'
EXAMPLE 6 doesn't look right to me, but that's what it says.

I can't figure out why that was the case either. Probably was supposed to be P1Y10M3D + P2Y5MT-10M. Will mark this down for a future amendment.

To my great disappointment, Annex D does not give any examples of how to resolve situations in which the time scale components become negative, even though my careful reading of the document strongly suggests that such a situation is possible.

I just re-read 8601-2. Indeed, Annex D does not describe situations where time scale components become negative, but Clause 14 does provide examples of subtraction where they can be performed on the same units.

Could you briefly describe the issues encountered with subtraction? In the document we were very concerned about the comparisons between nominal durations vs realized durations (e.g. with positive / negative leap second), and would love to know how you deal with them.

[sffc]

Hello, @ronaldtse!

Could you briefly describe the issues encountered with subtraction? In the document we were very concerned about the comparisons between nominal durations vs realized durations (e.g. with positive / negative leap second), and would love to know how you deal with them.

The algorithm essentially says to take your composite duration, apply it field-by-field to your time scale unit, and then resolve problems in the resulting intermediate time scale unit by performing carry-over and truncation as necessary.

So, what happens with 2018Y01M10D minus P2M15D? You get an intermediate 2018Y-01M-5D. The algorithms for roll-over and truncation say what to do when the values are too big, but, as far as I can tell, not when the values are too small, unless I'm missing something.

Also see #857 for some more cases and related questions.

[sffc]

I just re-read 8601-2. Indeed, Annex D does not describe situations where time scale components become negative, but Clause 14 does provide examples of subtraction where they can be performed on the same units.

Section 14 shows examples of subtraction between two composite durations, but not between a time scale and a duration. Section 14.4, "Date time modified by duration", is the one I'm most interested in, and it only gives a few examples (none of which involve subtraction) and defers you to Annex D for more details.

[ronaldtse]

Since ISO 8601-2 Annex D omitted the algorithm and examples for dealing with underflow units, here's a draft supplement. As editor I apologize that this wasn't provided in the standard.

This will be contributed back to CC 18011, and the ISO group may perhaps consider as an amendment. (CalConnect/csd-datetime-explicit#12)

Please feel free to let us know what you think!

---

(insert) ISO 8601-2:2019, 14.4

In date-time arithmetic, some time scale units can be converted into one another through deterministic relationships, which are transitive. These freely convertible relationships include:

• century and decade (a calendar century contains 10 calendar decades)
• decade and year (a calendar decade contains 10 calendar years)
• year and week (a calendar year contains 52 calendar weeks)
• year and month (a calendar year contains 12 calendar months)
• week and day (a calendar week contains 7 calendar days)
• day and hour (a calendar day contains 24 clock hours)
• hour and minute (a clock hour contains 60 clock minutes)

The following time scale unit conversions are not deterministic in abstract, the actual durations of these time scale units depend on contextual information:

• year and day (a calendar year may contain 365 or 366 calendar days);
• month and day (a calendar month may contain 28, 29, 30, 31 calendar days);
• minute and second (a clock minute may contain 59, 60, 61 clock seconds, in UTC).

These rules are considered as the "time scale unit conversion boundaries".

The result of a date time arithmetic expression does not change with time scale unit conversion as long as the conversion boundaries are not violated.

When the full context of a date time expression is provided, these boundaries can be relaxed accordingly.

EXAMPLE 1 The date time expression '2023Y59O' can be freely converted into '2023Y2M28D', because the 59th ordinal day of a Gregorian year is always February 28th.

EXAMPLE 2 The date time expression '2020Y60O' can be freely converted into '2020Y2M29D', because the 60th ordinal day of the Gregorian year 2020 is February 29th.

EXAMPLE 3 The date time expression '2016Y12M31DT61S' can be freely converted into '2017Y1M1DT-1S', because a leap second was inserted into the last clock minute on 2016-12-31.

(add) ISO 8601-2, D.3.3 Borrow to resolve underflow in time scale components

The composite duration form was meant to delay the resolution of date time duration until application because some time scale units require contextual information.

In some cases, it is possible to convert a duration expression that contains a negative time scale component into an equivalent duration expression where the previously negative time scale component becomes positive.

This process of conversion involves two operations:

• the "borrow" operation, the counterpart to the "carry-over" operation;
• time scale unit conversion.

NOTE: If exact duration is not necessary or is considered too complex to calculate, one could simplify to omit some boundaries to adopt "nominal duration", such as ignoring the leap second (i.e. ignore rule 3) and treat 1 minute = 60 seconds.

EXAMPLE 4 The expression 'P1Y10M3D − P2Y5MT10M' results in 'P3Y15M3DT-10M'. The negative sign at '-10M' can be converted into a positive expression by "borrowing" from a time scale unit of a higher order. Since the "hour" time scale unit is 0, the next time scale unit to borrow from is the "day" unit. One "day" unit expands to 24H, resulting in 'P3Y15M2DT24H-10M'. One "hour" unit can expand into 60M, resulting in 'P3Y15M2DT23H(60-10)M', which ends up being 'P3Y15M2DT23H50M'.

EXAMPLE 5 'P1Y-10M3D + P2Y-5M' results in 'P-1Y-15M3D'. The negative sign at '-1Y' cannot be resolved because there is no higher order unit to borrow from. The negative sign at '-15M' can be resolved by borrowing from the "year" unit. In order to pad the "month" unit to a positive number, the minimum number of years to borrow from "year" is 2, resulting in 'P-3Y(24-15)M3D', which equals 'P-3Y9M3D'.

EXAMPLE 6 'PT1H60S - PT122M' results in 'PT1H-122M60S'. The negative sign at '-122M' can be resolved, because the conversion between "hour" and "minute" does not cross any conversion boundaries. In order to resolve the negative sign at '-122M', a total of 3 hours need to be borrowed, resulting in 'PT(1-3)H(180-122)M60S', which concludes as 'PT-2H58M60S'. This expression can also be represented as 'PT-62M60S', since "hour" and "minute" can be freely converted (-2H = -120M; -120M + 58M = -62M).

EXAMPLE 7 'PT5H120S - PT1M' results in 'PT5H-1M120S'. The negative sign at '-1M' can be resolved by borrowing from the "hour" unit without crossing conversion boundaries. The borrowing of 1 hour to minutes results in 'PT4H(60-1)M120S', which concludes as 'PT4H59M120S'. The '59M120S' portion cannot be simplified as the "minute to seconds" conversion requires contextual information.

(new) ISO 8601-2, D.4.6 Resolving negative time scale units in an expression ("underflows")

The steps are as follows:

1. Starting at the rightmost side of the composite duration expression, process every negative durational unit one by one as follows:
   a) From the negative time scale component (the "underflowed component"), find in the expression a higher order time scale component that is freely convertible (i.e. does not violate conversion boundaries) and has a positive value. There could be multiple higher order units that can be freely convertible, such as minute <> hour and minute <> day.
   b) Borrow units from the higher order value until the current time scale is positive. Ensure the time scale components being modified are properly updated.
   c) Once the underflowed component becomes positive, or if there are no more freely convertible components (even if the underflowed component is still negative), continue with step 1 on the resulting duration expression.
2. Once all durational units have been processed, the resulting date and time representation is complete.

EXAMPLE 1 Calculation of '2022Y2M2D - P1Y10M3D':

• Calculate in accordance to Annex D, compute the intermediate expression '(2022-1)Y(2-10)M(2-3)D' which results in '2021Y-8M-1D'.
• Begin resolving negative time scale units.
  • Process the first underflow component from the right: '-1D'.
  • The "day" unit does not have a freely convertible higher-order component in this expression.
  • Process the second durational unit from the right: '-8M'
  • The "month" unit can be freely convertible with the higher order component "year" in this expression.
  • The "month" unit borrows 1 year, which is equal to 12 months. The partial expression under consideration
    of '2021Y-8M' becomes '2020Y(12-8)M', which resolves to '2020Y4M'.
  • The intermediate expression becomes '2020Y4M-1D'.
  • There are no more underflow components.
• The resulting date and time expression after underflow resolution is '2020Y4M-1D'.

EXAMPLE 2 Calculation of '2025Y59O - P20DT1H30M':

• Notice that the former and latter expressions use 'O' and 'D' to express the same time scale unit of calendar day respectively. This means that their values can be calculated within the same time scale unit. To preserve the meaning of the "ordinal day" time scale component, we select 'O' as the component to perform calculations on.

• Calculate in accordance to Annex D, compute the intermediate expression '2025Y(59-20)OT(-1)H(-30)M' which results in '2025Y39OT-1H-30M'.

• Begin resolving negative time scale units.

  • Process the first underflow component from the right: '-30M'.
  • The "minute" unit can be freely convertible with the higher order component "hour" in this expression.
  • The "minute" unit borrows 1 hour, which is equal to 60 minutes. The partial expression under consideration
    of 'T-1H-30M' becomes 'T(-1-1)H(60-30)M', which resolves to 'T-2H30M'.
  • Process the second durational unit from the right: '-2H'
  • The "hour" unit can be freely convertible with the higher order component "day" in this expression.
  • The "hour" unit borrows 1 day, which is equal to 24 hours. The partial expression under consideration
    of '39OT-2H' becomes '(39-1)OT(24-2)H', which resolves to '38OT22H'.
  • The intermediate expression becomes '2025Y38OT22H'.
  • There are no more underflow components.

• The resulting date and time expression after underflow resolution is '2025Y38OT22H'.

EXAMPLE 3 Calculation of 'T10H10M10S - PT5H30M20S':

• Calculate in accordance to Annex D, compute the intermediate expression 'T(10-5)H(10-30)M(10-20)S' which results in 'T5H-20M-10S'.
• Begin resolving negative time scale units.
  • Process the first underflow component from the right: '-10S'.
  • The "second" unit does not have a freely convertible higher-order component in this expression.
  • Process the second durational unit from the right: '-20M'
  • The "minute" unit can be freely convertible with the higher order component "hour" in this expression.
  • The "minute" unit borrows 1 hour, which is equal to 60 minutes. The partial expression under consideration
    of 'T5H-20M' becomes 'T(5-1)H(60-20)M', which resolves to 'T4H40M'.
  • The intermediate expression becomes 'T4H40M-10S'.
  • There are no more underflow components.
• The resulting date and time expression after underflow resolution is 'T4H40M-10S'.

[sffc]

Thanks for writing this up! I commented on it in CalConnect/csd-datetime-explicit#12.

[ptomato]

Is there anything remaining to do on this issue? @justingrant Does the Duration arithmetic that we have, correspond with CalConnect/csd-datetime-explicit#12 ?

[ptomato]

There seems to be nothing remaining to do on this issue.