localtime version of sec2gmt

[sitaramc]

I need a way to convert epoch seconds to localtime instead of GMT.

Currently there seems to be no way to do that. (I've tried all kinds of formats with strftime too).

[johnkerl]

Thanks for asking.

In my professional life I've seen many systems be bitten hard by timezone issues so I was reluctant to step into that unless someone really wanted the feature.

That said, strftime and strptime, in the C library under the hood, do support local timezones so it's a matter of exposing that.

I'm still iterating and I don't have all unit tests written out, but, at present in head, I have:

• sec2gmt and sec2localtime
• sec2gmtdate and sec2localdate
• gmt2sec and localtime2sec

Those are key-stroke savers for frequently-used Y-m-d H:M:S format strings. Then more generally:

• strftime and strftime_local
• strptime and strptime_local

These simply require you to set the TZ environment variable, e.g. export TZ=America/Manaus or export TZ=UTC+05:30.

The C library has a few quirks for timezone handling which I'm passing along:

• There's no clear way I can tell to programatically set arbitrary timezone, e.g. for there to be a Miller function with arbitrary timezone as an argument so you could produce multiple timezones in a single run of the executable. (A hack would be for me to implement this by setting TZ within the Miller C program.)
• When a timezone is misspelled, even by a single character, UTC is silently used.

[sitaramc]

Thanks! I tested it, and while strftime_local seems to work fine, strptime_local does not seem to have "taken" -- I get an error saying: mlr: Function name "strptime_local" not found. (Just in case, I've attached my test sequence, paltry as it is). I realise that setting an arbitrary timezone may be needed by some people (eventually you'll certainly get asked that), but for my purposes, setting TZ outside and using that to get local stuff is suffiall I need. As for your second caveat about TZ misspellings, well that's true for the humble 'date' command also so I wouldn't worry about it! thanks again sitaram

…

On Sat, Apr 28, 2018 at 06:41:23PM -0700, John Kerl wrote: Thanks for asking. In my professional life I've seen many systems be bitten hard by timezone issues so I was reluctant to step into that unless someone really wanted the feature. That said, `strftime` and `strptime`, in the C library under the hood, do support local timezones so it's a matter of exposing that. I'm still iterating and I don't have all unit tests written out, but, at present in head, I have: * `sec2gmt` and `sec2localtime` * `sec2gmtdate` and `sec2localdate` * `gmt2sec` and `localtime2sec` Those are key-stroke savers for frequently-used `Y-m-d H:M:S` format strings. Then more generally: * `strftime` and `strftime_local` * `strptime` and `strptime_local` These simply require you to set the `TZ` environment variable, e.g. `export TZ=America/Manaus` or `export TZ=UTC+05:30`. The C library has a few quirks for timezone handling which I'm passing along: * There's no clear way I can tell to programatically set arbitrary timezone, e.g. for there to be a Miller function with arbitrary timezone as an argument so you could produce multiple timezones in a single run of the executable. (A hack would be for me to implement this by setting `TZ` within the Miller C program.) * When a timezone is misspelled, even by a single character, UTC is silently used. -- You are receiving this because you authored the thread. Reply to this email directly or view it on GitHub: #170 (comment)

[sitaramc]

oh I forgot the attachment; sorry. Here it is
test.txt

[johnkerl]

yes, it's still under development -- I'm not done & have not closed the task out.

[johnkerl]

P.S. re "In my professional life I've seen many systems be bitten hard by timezone issues so I was reluctant to step into that unless someone really wanted the feature." -- what I meant to say was "In my professional life I've seen many systems be bitten hard by daylight-savings-time issues".

Here:

• Is DST handled properly for various timezones?
• There's a day a year when 1:30 a.m. (or something) won't exist at all on that date -- how to strptime that?
• There's a day a year when 1:30 a.m. (etc.) exists twice -- how to strptime that?

So it will come with caveats.

[johnkerl]

OK everything is pushed.

59ca76b shows the test cases, focusing on DST changes.

Something's not right though:

$ cat x
export TZ=America/Sao_Paulo
echo TZ=$TZ
mlr --opprint put '$b=localtime2sec($a); $c=sec2localtime($b); $d=sec2localdate($b)' <<_EOF
a=2017-02-16 00:00:00
a=2017-02-17 00:00:00
a=2017-02-18 00:00:00
a=2017-02-19 00:00:00
a=2017-02-20 00:00:00
a=2017-10-12 00:00:00
a=2017-10-13 00:00:00
a=2017-10-14 00:00:00
a=2017-10-15 00:00:00
a=2017-10-16 00:00:00
a=2017-10-17 00:00:00
_EOF

$ sh x
TZ=America/Sao_Paulo
a                   b                 c                   d
2017-02-16 00:00:00 1487214000.000000 2017-02-16 01:00:00 2017-02-16
2017-02-17 00:00:00 1487300400.000000 2017-02-17 01:00:00 2017-02-17
2017-02-18 00:00:00 1487386800.000000 2017-02-18 01:00:00 2017-02-18
2017-02-19 00:00:00 1487473200.000000 2017-02-19 00:00:00 2017-02-19
2017-02-20 00:00:00 1487559600.000000 2017-02-20 00:00:00 2017-02-20
2017-10-12 00:00:00 1507777200.000000 2017-10-12 00:00:00 2017-10-12
2017-10-13 00:00:00 1507863600.000000 2017-10-13 00:00:00 2017-10-13
2017-10-14 00:00:00 1507950000.000000 2017-10-14 00:00:00 2017-10-14
2017-10-15 00:00:00 1508036400.000000 2017-10-15 01:00:00 2017-10-15
2017-10-16 00:00:00 1508122800.000000 2017-10-16 01:00:00 2017-10-16
2017-10-17 00:00:00 1508209200.000000 2017-10-17 01:00:00 2017-10-17

Namely strptime and strftime aren't inverses during DST. I expect weirdness during the single transition hours themselves; but the non-invertibility is outside the single transition hours.

[johnkerl]

This is illuminating:

export TZ=America/New_York
mlr --oxtab put '
  $y=strptime_local($x,"%Y-%m-%d %H:%M:%S %Z");
  $z=strftime_local($y,"%Y-%m-%d %H:%M:%S %Z")
' <<_EOF
x=2000-07-01 00:00:00 EST
x=2000-07-01 00:00:00 EDT
_EOF

x 2000-07-01 00:00:00 EST
y 962427600.000000
z 2000-07-01 01:00:00 EDT

x 2000-07-01 00:00:00 EDT
y 962424000.000000
z 2000-07-01 00:00:00 EDT

Meanwhile

export TZ=America/New_York
while read a; do
  b=$(date -j -f  "%Y-%m-%d %H:%M:%S %Z" "$a" +%s)
  c=$(date -r $b "+%Y-%m-%d %H:%M:%S %Z")
  echo $a
  echo $b
  echo $c
  echo
done <<_EOF
2000-07-01 00:00:00 EST
2000-07-01 00:00:00 EDT
_EOF

2000-07-01 00:00:00 EST
962427600
2000-07-01 01:00:00 EDT

2000-07-01 00:00:00 EDT
962424000
2000-07-01 00:00:00 EDT

and

export TZ=America/New_York
while read a; do
  b=$(date -j -f  "%Y-%m-%d %H:%M:%S" "$a" +%s)
  c=$(date -r $b "+%Y-%m-%d %H:%M:%S %Z")
  echo $a
  echo $b
  echo $c
  echo
done <<_EOF
2000-07-01 00:00:00
_EOF

2000-07-01 00:00:00
962424000
2000-07-01 00:00:00 EDT

This means:

• Using the Miller strptime_local as coded: you have to say the timezone is DST. If you say it out then it's right but the default is to treat all input as non-DST.
• Using the system date command, then if you say the timezone is DST or not, it recognizes those; but the default is to correctly infer the presence of DST.

Miller isn't being clever in its use of strptime. Just straight through. And the struct tm coming back from strptime_local with timezone unspecific does indeed have tm_isdst=0.

So apparently the system date command is being a bit more clever.

[sitaramc]

Interesting. As you said, lots of issues. Speaking for myself, I'd be fine if you just document the caveats with a couple of examples (or just refer to this thread/issue), and leave it at that. But that may be because India doesn't *have* DST so this is unlikely to affect me; it is possible others will need more than that. Thank you once again for your efforts.

[johnkerl]

OK @sitaramc ... what's in head at the moment will definitely work for your no-DST case. And whatever mods I make from here on out, won't break non-DST. :)

[torbiak]

In order to use libc's capacity to automatically set tm_isdst, tm_isdst must be set to a negative value before calling mktime.

According to the OpenBSD strptime manpage:

There is no way to specify whether Daylight Saving Time is in effect when calling strptime. To use the resulting tm structure with functions that check the tm_isdst field, either set it to a negative value, which will cause mktime(3) to attempt to divine whether Daylight Saving Time would be in effect for the given time, or compute the value manually.

The GNU strptime manpage isn't as clear as this, but according to both the GNU and OpenBSD mktime(3) manpages, tm_isdst needs to be set to a negative value to get mktime to use the timezone db, and in my testing with glibc strptime doesn't touch tm_isdst. When I add ptm->tm_isdst = -1 to mlr_arch_timegmlocal(), right before calling mktime, I get the behaviour I expect and the epoch seconds returned agree with GNU date except in "fold" intervals during DST transitions, which date errors out on, saying the given time is invalid (fair enough). As you've remarked somewhere in the issues, GNU date does some fancy stuff in order to detect ambiguous timestamp-strings, but a lot of tools seem to just pass through whatever libc gives, and that seems fine to me.

@johnkerl, would you be interested in a PR that modifies mlr_arch_timegmlocal() and updates the expected regression test output? Are there any other tests that would need to be added/changed? Here's a gist comparing mlr's strptime_local to date, which I used to check my change.

[johnkerl]

@johnkerl, would you be interested in a PR

@torbiak goodness yes!!! :)

The test framework is in the c subdirectory invoked by ./reg_test/run. If you can get mods in there, great; if not, totally fine and I'll update it from your gist.