Update time-related data in src/lib/libast/tm

[McDutchie]

On FreeBSD I get the following regression test failure (among others):

test builtins begins at 2020-06-12+22:04:08
        builtins.sh[315]: printf "%T" now

This test compares the output of printf "%T" now with that of the external command date and checks if they are identical. On my cloud server and in my (Dutch) locale and time zone, they produce:

$ date
vr 12 jun. 2020 22:12:37 CEST
$ LC_ALL=C date
Fri Jun 12 22:13:20 CEST 2020
$ arch/freebsd12.amd64-64/bin/ksh -c 'printf %T\\n now'
vr 12 jun. 22:13:44 2020
$ LC_ALL=C arch/freebsd12.amd64-64/bin/ksh -c 'printf %T\\n now'
Fri Jun 12 22:13:54 NST 2020

The last one is funny: it puts me in Newfoundland Standard Time.

Clearly, the time zone data compiled into ksh is broken. I also suspect it's roughly 27 years out of date.

Shells really have no business dealing with time zones. Providing and maintaining time zone data is the job of the operating system. Maybe it was justifiable in 1993 when not all OSs supported those facilities, but that time is long behind us.

So, a proper fix for this bug, with backward compatibility in mind, will be to:

1. Make printf %T use the operating system's standard date utility. It should search for it in the default utility path as in command -p so that it works regardless of $PATH somehow use the time zone data provided by the OS.
2. Figure out what else might depend on the broken time zone data, and how to get that to function in an alternative way.
3. Remove the internal time zone data.

[jghub]

time { i=0; while ((i <=10000)); do ((++i));  printf '%T' now  >/dev/null; done ;}
# real	0m0.34s

time { i=0; while ((i <=10000)); do ((++i));date  >/dev/null; done ;}
# real	0m14.78s

I'd say as with a couple builtins (printf itself, e.g....) it is not necessarily desirable to remove functionality from ksh itself just because the system does provide the same functionality due to the incurred performance hit. quite naturally one can envisage ksh scripts making heavy use of time/date info. even only 1e4 queries lead to massive waiting time (15s) in interactive use while the 1/3s of %T would be perfectly fine.

regarding the strange error you see with freebsd (I don't observe it in OSX), that sure looks like a bug. but I feel the solution should not be to make %T an "alias" of date. I also feel it would run against your stated conservative approach to avoid replicating the ksh2020 story. just my 2c

[McDutchie]

quite naturally one can envisage ksh scripts making heavy use of time/date info.

You are right. That would be a performance hit and we don't want that.

At the same time I simply can't see anyone volunteering to maintain timezone data that is internal to ksh. Even if someone did, the ksh and OS time zone data would inevitably be out of sync at some point, causing incorrect results. Modern OSs get updates to their time zone packages all the time.

And while performance is very important to ksh, correctness is actually more important. Something fast that doesn't work right is no use.

Surely there must be a way for C to use the operating system's time zone data directly, but I'm afraid this area is outside of my expertise and I don't have time to research it any time soon.

So I hope someone expert on this will pick up this bug and fix it.

[McDutchie]

FYI, the outdated time zone data is in src/lib/libast/tm/tmdata.c. At first glance, it seems to me that that whole libast/tm directory duplicates time-related functionality that every remotely current OS already provides.

The Tm_leap_t array in tmdata.c doesn't contain leap seconds past 2008. I'm sure plenty of other stuff is hopelessly out of date as well.

[JohnoKing]

The bug with printf %T on FreeBSD was fixed in ksh2020 (see att#591). I have made a pull request to backport this bugfix, although I will note that it still doesn't update the outdated set of leap seconds in the Tm_leap_t array.

[posguy99]

There are only three inserted leap seconds between that data and now, according to https://en.wikipedia.org/wiki/Leap_second , so it isn’t going to be all THAT far off. 🙂 Is the large number seconds since the epoch? I honestly have no idea how it works, but it’s interesting.

…

-- Marc Wilson posguy99@gmail.com

[posguy99]

Actually, I looked around for more understanding, and found this:

https://kb.meinbergglobal.com/kb/time_sync/ntp/configuration/ntp_leap_second_file#tai_offset_table

Which refers to multiple places you can get a leap-seconds.list file.

One is ftp://ftp.boulder.nist.gov/pub/time/leap-seconds.list

But the data in it doesn't match what ksh is using? I mean, for dates KSH does have?

[McDutchie]

The missing leap seconds have been inserted in 51f97cf thanks to @JohnoKing .

[McDutchie]

One is ftp://ftp.boulder.nist.gov/pub/time/leap-seconds.list

But the data in it doesn't match what ksh is using? I mean, for dates KSH does have?

I think that must be a different thing. The comments in that file say:

#	The following table shows the corrections that must
#	be applied to compute International Atomic Time (TAI)
#	from the Coordinated Universal Time (UTC) values that
#	are transmitted by almost all time services.

…whereas the ksh leap second values correspond to those in IETF's RFC 8536 The Time Zone Information Format, appendix B.1.

[McDutchie]

So, with the outright bug fixed, it seems right to change this issue to a TODO item.

[McDutchie]

As far as I can tell, there's nothing left to do here, as printf %T will obtain the time zone from the OS where available. There's still a problem with printf %T output in non-English locales, but that is bug #52.