RFC: Mitigating DST troubles

[wobh]

For some reason, I've lately been thinking a lot about the troubles we had earlier this year with the gigasecond exercise, and I've come up with some ideas about detecting and preventing future issues with that or perhaps with future exercises (todo: make list of other datetime exercises).

Now that we have exercise/example testing setup, the first is simply to schedule a TravisCI build for the DST switch dates in the US. If the tests fail for some reason, the scheduled build should let us know almost as soon as can be known. I poked around the TravisCI settings and didn't see anything like this, so I thought about setting up an IFTTT solution (I'm pretty sure I have an account, but I don't remember the password, as I don't think I've ever used it). Any other suggestions welcome.

The second is that it's seems possible that perhaps only a few CL implementations could be affected by the DST switch due to a bug in time handling. It would be nice to figure out how to conditionally allow failures on a per-test-case basis, so that even if there's a problem with the DST affected exercise tests on that implementation, we can still get feedback on the other exercises until the issue is resolved (or until the switch back).

If more serious problems turn up, we could also look for a date-time library for CL that's well maintained and, if that works out, recommend it to users.

Lastly, I think we should also consider a ---well maybe "policy" is too strong a word, but something between a recommendation and a policy, where we prefer time-dependent test implementations to work for standard time, if for some reason we can't make them work for both DST and ST.

I don't suggest that any of these be permanent or immediately acted upon, I mainly wanted to get my thoughts on the topic out there and collect some ideas going forward.

http://www.nist.gov/pml/div688/dst.cfm

Keep the faith, someday, sometime, the tyranny will end.

[verdammelt]

On Mon, 22 Jun 2015 22:59:23 -0700, William Clifford notifications@github.com said:

wobh> [...] to schedule a TravisCI build for the DST switch dates in
wobh> the US. If the tests fail for some reason, the scheduled build
wobh> should let us know almost as soon as can be known.

I like this idea of setting it up so that the Travis job fires at the
DST change times. It appears that you are not the first to ask how to do
this
http://stackoverflow.com/questions/20395624/travis-ci-builds-on-schedule.

wobh> The second is that it's seems possible that perhaps only a few
wobh> CL implementations could be affected by the DST switch due to
wobh> a bug in time handling.

I had not thought about this possibility. I don't remember asking the
people submitting the reports of broken tests (is there a name for a
person who does Exercism exercises?) what implementations they were
using - but maybe one of us did and we have some data?

wobh> It would be nice to figure out how to conditionally allow
wobh> failures on a per-test-case basis, so that even if there's a
wobh> problem with the DST affected exercise tests on that
wobh> implementation, we can still get feedback on the other
wobh> exercises until the issue is resolved (or until the switch
wobh> back).

It seems to me (without deep review / thought) that the existing test
harness runs all tests and then reports pass/fail. So if that is correct
then even if a time related test failed due to DST/TZ issues we'd still
get feedback from the other tests. Or are you asking about allowing such
a test to fail - but allow the entire run for that implementation to
pass?

This sort of feature sounds like something we could build into the test
harness. But it also feels like the sort of thing that we should not add
unless we are sure we need it. Allowing some tests to sometimes fail and
not fail the test run sounds like a good way to not notice a problem.

wobh> Lastly, I think we should also consider a ---well maybe
wobh> "policy" is too strong a word, but something between a
wobh> recommendation and a policy, where we prefer time-dependent
wobh> test implementations to work for standard time, if for some
wobh> reason we can't make them work for both DST and ST.

I think a better policy is to not have time related exercises in
Exercism.IO. While time issues always seem easy, they never are. It
seems to me that complicated topics such as time or threading should be
kept out of these exercises.

wobh> Keep the faith, someday, sometime, the tyranny will end.

The tyranny of time? Down with clocks!

Suddenly I just thought of Metric Time - maybe that is what you mean by
'standard time'?

[wobh]

Good points, especially the last. I think we're agreed that we should at least set up some automatic test runs starting on 2015-11-01T02:00:00 (maybe a little later) so from here on out it's just a question of means and procedures.

TronCI seems like a good find, but I find myself a bit uncomfortable turning over my github credentials for this. I suppose it's a baseless worry though.

I think we should setup the test runs to use Eastern time per http://stackoverflow.com/a/23374438 and I think it might be a good idea to setup up the run time for a couple minutes after the change over. I wonder if we could set the datetime on the Travis VM and run some tests in advance?

[verdammelt]

Perhaps another way to approach this problem is to change the gigasecond interface to include a timezone parameter?

[wobh]

Hey, guess what came and went without any fuss?

[kytrinyx]

Happy winter, everyone.