are the timestamps in the leap second table correct?

[BurntSushi]

I was reviewing this code today:

https://github.com/nyx-space/hifitime/blob/c28151dd94a28951342352802dd04e783e53db35/src/leap_seconds.rs

I noticed that the timestamp at which the leap second occurred is interpreted as TAI, but the timestamps in leap-seconds.list are recorded as NTP timestamps. NTP uses the same epoch as hifitime, but my understanding is that all NTP timestamps are equivalent to Unix timestamps minus 2208988800 (the number of seconds between 1900-01-01T00:00:00Z and 1970-01-01T00:00:00Z). This seems to imply to me that NTP timestamps are therefore not equivalent to TAI timestamps. Maybe I'm missing something?

I wonder if this is related to the behavior in this code snippet?

let leap = "2016-12-31T23:59:60 UTC".parse::<hifitime::Epoch>().unwrap();
let nonleap = "2016-12-31T23:59:59 UTC".parse::<hifitime::Epoch>().unwrap();
assert_ne!(leap, nonleap);

The assertion fails with the following output:

assertion `left != right` failed
  left: 2016-12-31T23:59:57 UTC
 right: 2016-12-31T23:59:57 UTC
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

Namely, I expected at least two things different in the output:

1. I expected the two different datetime strings to parse to two distinct
   instants in time.
2. I expected the first to parse to 23:59:60 and the second to parse to
   23:59:59. Instead, they both parse to 23:59:57, which I cannot explain.

(Please interpret this as a low confidence query on my part. I've done a fair bit of reading on this topic and I've found it very difficult to plant my feet firmly on solid ground.)

[BurntSushi]

Thinking about this more, aren't two timestamps needed for each leap second to do conversions from UTC to TAI and from TAI to UTC? (Or at least, adjust the conversion algorithm to be aware of this fact.)

Namely, when going from UTC to TAI, the leap second table is queried given UTC time (not TAI):

hifitime/src/epoch.rs

Lines 316 to 327 in c28151d

	#[must_use]
	/// Initialize an Epoch from the provided UTC seconds since 1900 January 01 at midnight
	pub fn from_utc_duration(duration: Duration) -> Self {
	let mut e = Self::from_tai_duration(duration);
	// Compute the TAI to UTC offset at this time.
	// We have the time in TAI. But we were given UTC.
	// Hence, we need to _add_ the leap seconds to get the actual TAI time.
	// TAI = UTC + leap_seconds <=> UTC = TAI - leap_seconds
	e.duration_since_j1900_tai += e.leap_seconds(true).unwrap_or(0.0) * Unit::Second;
	e.time_scale = TimeScale::UTC;
	e
	}

But when going from TAI to UTC, the same exact leap second table is queried given TAI time:

hifitime/src/epoch.rs

Lines 1928 to 1933 in c28151d

	#[must_use]
	/// Returns this time in a Duration past J1900 counted in UTC
	pub fn to_utc_duration(&self) -> Duration {
	// TAI = UTC + leap_seconds <=> UTC = TAI - leap_seconds
	self.duration_since_j1900_tai - self.leap_seconds(true).unwrap_or(0.0) * Unit::Second
	}

If the leap second table is indeed just NTP timestamps, then I think the UTC to TAI conversion is correct, but the TAI to UTC conversion is not.

[ChristopherRabotin]

If the leap second table is indeed just NTP timestamps, then I think the UTC to TAI conversion is correct, but the TAI to UTC conversion is not.
That's correct: this bug was reported a few months ago, #255 , and I'll address it as part of the next release. Most of my work requires the conversion from UTC to TAI and other time systems, and almost never the other way around, which is probably why I missed it. The calculations to/from UTC currently (should!) use the algorithm shown in NASA's naif00012.txt file here: https://github.com/nyx-space/hifitime/blob/master/naif0012.txt . Then the algorithm is reversed, but that may be wrong indeed. I'm actively working #237, which simplifies these conversions: instead of converting all of the epochs to TAI at initialization, the epochs will be kept in their own time scale until a new time scale is requested.

Concerning the first example of "2016-12-31T23:59:60 UTC" == "2016-12-31T23:59:59 UTC", my understanding is that the interpretation of equality on a leap second is ambiguous. This allows Hifitime to set the equality of two epochs to be whether or not they represent the same amount of time since 01 January 1900 at zero hours. In fact, when checking the duration since that time, via the to_tai_parts() function, we can see that both epochs have the same duration: {centuries: 1, nanoseconds: 536457634000000000}.

The fact that the date is printed incorrectly is an alarming bug: it should absolutely be printed correctly. From running a checks, the two second offset seems to be specific to printing a date that is within a second of the leap second, so I'm guessing it's related to #255 .

P.S.: Thanks for your patience and interest in this library, and please keep the bug reports coming! It's motivating to see people be interested in this pretty niche work, and I'm sorry there are still some bugs.

[BurntSushi]

(unless somehow we keep track of whether it was initialized at the 60th second)

I think that's the key. If 23:59:59 and 23:59:60 map to two distinct TAI Epoch values, and if your leap second table contains both NTP timestamps and TAI timestamps, then you can unambiguously map both TAI Epoch values back to the correct UTC rendering.

Here is some of the background reading I've done that leads me to this conclusion:

• http://www.madore.org/~david/computers/unix-leap-seconds.html
• https://geometrian.com/programming/reference/timestds/index.php
• https://unix.stackexchange.com/questions/283164/unix-seconds-tai-si-seconds-leap-seconds-and-real-world-code (This links to the above two articles.)

And in particular, in the first article, notice the first example:

TAI=2009-01-01T00:00:32.0; UTC=2008-12-31T23:59:59.0; UT1≅2008-12-31T23:59:58.407
TAI=2009-01-01T00:00:32.5; UTC=2008-12-31T23:59:59.5; UT1≅2008-12-31T23:59:58.907
TAI=2009-01-01T00:00:33.0; UTC=2008-12-31T23:59:60.0; UT1≅2008-12-31T23:59:59.407
TAI=2009-01-01T00:00:33.5; UTC=2008-12-31T23:59:60.5; UT1≅2008-12-31T23:59:59.907
TAI=2009-01-01T00:00:34.0; UTC=2009-01-01T00:00:00.0; UT1≅2009-01-01T00:00:00.407
TAI=2009-01-01T00:00:34.5; UTC=2009-01-01T00:00:00.5; UT1≅2009-01-01T00:00:00.907

That seems to me to clearly imply that 23:59:59 and 23:59:60 should correspond to two distinct points in time, and that the advantage of using TAI is that it can unambiguously represent either of them. The "only" cost is that you need a table of leap seconds to convert between them.

From your blog:

Moreover, the time between two corrections is not equally spaced. In other words, UTC is not a monotonic clock and attempting to use UTC over a leap second adjustment will cause problems.

This is another area where I've gotten hung up with the writing on this topic. The articles I linked above go into more detail, but the basic idea here is that it's not UTC that isn't monotonic, but rather Unix (or "POSIX") time. Indeed, I've found this from the first article linked above to be an extremely profound insight:

Unlike TAI and UT1, the UTC time scale should not be considered as a pure real number (or seconds count): instead, it should be viewed as a broken-down time (year-month-day-hour-minute-second-fraction) in which the number of seconds ranges from 0 to 60 inclusive (there can be 61 or 59 seconds in a minute); during a positive leap second the number of seconds takes the value 60 (while a negative leap second would skip the value 59, but this has never occurred).

As the article goes on to say, the problem is indeed when we try to condense UTC down into a single timestamp:

If we attempt to condense UTC to a single number (say, the number of seconds since 1970-01-01T00:00:00 or since 1900-01-01T00:00:00, or the number of 86400s-days since 1858-11-17T00:00:00, or something of the sort), we encounter the problem that the same value can refer to two different instants since the clock has been set back one second (negative leap seconds, of course, would cause no such difficulty). One of the themes of what follows is what Unix functions such as gettimeofday() or clock_gettime() can, should, do or might return at these various points in time.

And that's effectively where Unix time is. As the second article points out, "Unix is UTC but without leap seconds" is somewhat of an absurd notion, given that leap seconds are the foundational concept upon which UTC is built.

That time doesn't exist in the real world, we only say it does so that the UTC time scales slows down the counting of time to try to keep the sun at its zenith at noon.

I don't think I grok this. In my mind, a leap second exists exactly as much as Feb 29 exists. The latter is more predictable, of course.

[ChristopherRabotin]

Hmm. You raise excellent points. What has got me thinking most of the day is the idea that UTC isn't a time scale, but instead, it's a representation. This notion matches also what SPICE says (SPICE tends to be my main source because many if not most of the tests in hifitime are compared with SPICE, and the main goal of hifitime is to provide the time management capability of SPICE, but in Rust and Python).

UTC is not a time system, rather a representation of a time. But it is often considered a time system.

In version 4.x, which is development thanks to excellent work by @gwbres , we have the opportunity to fix this, even if that means introducing breaking changes. I'll need to think about the architectural changes needed. The concept of a time system in hifitime is that it's any duration since a reference epoch ... but this seems to be an incorrect interpretation of UTC ... and UTC is easily the most used representation of datetimes.

At first thought, it seems like an Epoch should be defined by a triplet of time system, representation, and duration in that time system. Representation would include UTC, Julian Days, Modified Julian Days, etc. I guess the other question is the following: how to handle other time scales that are leap second aware? I'm open to ideas.

[BurntSushi]

What has got me thinking most of the day is the idea that UTC isn't a time scale, but instead, it's a representation.

Yeah I agree. I think C++'s chrono library also has a somewhat similar confusion here. It treats UTC as a time scale too via its "clock" abstraction. It has at least the following clocks: system (which is just the Unix time scale), UTC and TAI. The confusing part is that it does the usual leap second adjustment (similar to hifitime) not when UTC is converted to TAI, but when Unix is converted to UTC. And then converting from UTC to TAI is just an offset to adjust to the 1958-01-01T00:00:00Z TAI epoch. What's actually going on here is that its UTC clock is using TAI, and its TAI clock is equivalent, but just with a different epoch. It's all very confusing.

As perhaps with hifitime, I think the issue with C++'s chrono is that it has a UTC clock in the first place. It should just have a TAI clock, and the conversion from Unix to TAI is what adjusts for leap seconds.

This notion matches also what SPICE says

Oh nice! Thanks for this link. I hadn't seen it yet. After a quick glance it does look like my understanding matches what's written there. Phew.

I'll need to think about the architectural changes needed. The concept of a time system in hifitime is that it's any duration since a reference epoch ... but this seems to be an incorrect interpretation of UTC ... and UTC is easily the most used representation of datetimes.

At first thought, it seems like an Epoch should be defined by a triplet of time system, representation, and duration in that time system. Representation would include UTC, Julian Days, Modified Julian Days, etc. I guess the other question is the following: how to handle other time scales that are leap second aware? I'm open to ideas.

I haven't thought too deeply about this, but you could maybe keep the UTC time scale, but do the "right" thing when formatting/parsing. The internal representation is still in TAI, so you have an unambiguous representation of the time.

[ChristopherRabotin]

As of #289, the Epoch structure is lazy: it only stores the duration since the start of the time scale it has been initialized with. This defines a new "prime epoch", separate from J1900, and set to 1900-01-01 midnight. Under the hood, any computation of time scale is handled by a single function: to_time_scale. (Note that so far, TAI reference epoch is still set to 1900-01-01 midnight although it should be 1958).
This major change provides two main advantages:

1. If there is a glitch in the time scale conversions, but a user does not actually change from one time scale to another, then the glitch will not impact the user (since time scales are only changed when requested instead of systematically converted to TAI).
2. Any operation on a given epoch is guaranteed to happen on that time scale alone without fancy and confusing checking (e.g. adding 10 seconds to a TDB epoch over a leap second will show a 10 second change in TDB, and not be affected at all by the leap second shenanigans).
3. It makes it quite simple to add other time scales in the future (notably looking at Lunar coordinated time  #291 since Philip Linden and I recently wrote a brief on the topic).

I want to ensure that the issue you've uncovered is fixed, and for this, I want to add regression testing (and the relevant code for it). What would be a good test here? Would it be that initializing an epoch from its Gregorian representation at a time where there is a 60th second (i.e. over a leap second) and ensuring that this time is printed again with that 60th second?

Thanks for your help

[BurntSushi]

Would it be that initializing an epoch from its Gregorian representation at a time where there is a 60th second (i.e. over a leap second) and ensuring that this time is printed again with that 60th second?

I think that's exactly right, yes.