Leap second tests fail if system clock is in the future (2021-06-04) #10228

bmwiedemann · 2020-04-29T08:43:02Z

Description

As part of my work on reproducible builds for openSUSE, I check that software still gives identical build results in the future.
The usual offset is +15 years, because that is how long I expect some software will be used in some places.
This showed up failing tests in our python-astropy package build.
See https://reproducible-builds.org/ for why this matters.

Expected behavior

Software should be able to build anytime.

Actual behavior

tests fail with

astropy/utils/iers/iers.py:1021: in auto_open
    warn('leap-second file is expired.', IERSStaleWarning)
E   astropy.utils.iers.iers.IERSStaleWarning: leap-second file is expired.

That check already fails before 2021-06-04

and when dropping that expiry check, there is another problem with Time.now()

astropy/time/core.py:583: in now
    return cls(val=dtnow, format='datetime', scale='utc')
astropy/time/core.py:430: in __init__ 
    self._init_from_vals(val, val2, format, scale, copy,
astropy/time/core.py:485: in _init_from_vals
    self._time = self._get_time_fmt(val, val2, format, scale,
astropy/time/core.py:543: in _get_time_fmt
    return cls(val, val2, scale, precision, in_subfmt, out_subfmt)
astropy/time/formats.py:148: in __init__ 
    self.set_jds(val1, val2)
astropy/time/formats.py:831: in set_jds
    jd1, jd2 = erfa.dtf2d(self.scale.upper().encode('ascii'),
astropy/_erfa/core.py:15703: in dtf2d 
    check_errwarn(c_retval, 'dtf2d')

Steps to Reproduce

set system clock to 2035
run tests: py.test-3.8

System Details

Linux-5.6.6-1-default-x86_64-with-glibc2.2.5
Python 3.8.2 (default, Mar 05 2020, 18:58:42) [GCC]
Numpy 1.18.3
astropy 4.0.1.post1
Scipy 1.4.1

The text was updated successfully, but these errors were encountered:

bmwiedemann · 2020-04-29T09:04:07Z

https://github.com/astropy/astropy/blob/master/astropy/utils/iers/tests/data/leap-seconds.list#L208

#       File expires on:  28 June 2020

Maybe the test should not use the current time.

pllim · 2020-04-29T16:57:45Z

Well, we cannot predict what the leap second file would look like in 15 years, can we? A simple solution would be xfail the test if system clock is too far off the future but I am not sure if it is the correct solution. cc @mhvk and @eteq

mhvk · 2020-04-29T17:25:01Z

@bmwiedemann - thanks for your message! As @pllim notices, it is not possible to predict how the earth's rotation evolves in the future, and hence the failures you are seeing are real: by that time, we cannot guarantee some parts of astropy will produce correct results (though we can hopefully guarantee they would produce the same results as they would have now). Of course, for all those astropy relies on external data sites, so presumably the failing tests are marked as remote_data - would it make sense to test excluding these for the reproducible builds? Presumably, we cannot count on any external sites to be available in 2035 anyway. Or, if I'm presuming wrongly and the tests are not marked remote_data, we should probably do that!

bmwiedemann · 2020-04-29T18:50:14Z

In openSUSE, we build without network-access to ensure that all required data comes from the build system.

I think, the best route is to always use a 2019 date instead of Time.now() for such a failing test, because future leap seconds cannot be predicted, but 2019 dates should still be working in 2035.

mhvk · 2020-04-29T19:42:20Z

@bmwiedemann - usually there is a specific reason to use .now() but maybe we can adjust the test in some way (or mark it remote_data or so) - but could you point me to the actual tests that fail? (Link to a log is fine too.)

dhomeier · 2020-04-29T19:48:40Z

Presumably, we cannot count on any external sites to be available in 2035 anyway. Or, if I'm presuming wrongly and the tests are not marked remote_data, we should probably do that!

When running the tests with remote_data=True and time set to next year, I am also getting a number of additional failures on expired SSL certificates...

At least some data sources are indeed defined locally in iers.py, e.g.
IERS_LEAP_SECOND_FILE = get_pkg_data_filename('data/Leap_Second.dat')
but test_read_leap_second_dat is actually using a fixed date 2020-06-28 to check its expiration date. I think it would be appropriate to add the IERSStaleWarning you saw to the filterwarnings list in setup.cfg.

bsipocz · 2020-04-29T20:03:41Z

I think it would be appropriate to add the IERSStaleWarning you saw to the filterwarnings list in setup.cfg.

But we would need to make sure we notice if those files becomes stale during the dev process. I agree that it makes sense to ignore them during third party packaging tests.

dhomeier · 2020-04-29T20:08:07Z

But we would need to make sure we notice if those files becomes stale during the dev process. I agree that it makes sense to ignore them during third party packaging tests.

test_builtin_not_expired will even raise an error on that, but this is marked remote_data – should probably do for the development.

mhvk · 2020-04-29T20:25:17Z

I'm confused, is TestReading.test_read_leap_second_dat() actually causing problems? Please do post traces, as I went to some trouble to ensure that the checks on expiration time do never involve leap seconds (but obviously may well have failed!)

mhvk · 2020-04-29T20:29:14Z

More generally: it is probably good to assume any non-remote_data tests that fail at a future time reflect bugs, either in the code proper or in the tests! The LeapSecond work has been among the most complicated I have done...

bmwiedemann · 2020-04-29T20:40:09Z

Here is a log of the failing 2021-06-04 build test

 + PYTHONPATH=:/home/abuild/rpmbuild/BUILDROOT/python-astropy-4.0.1.post1-0.0.x86_64/usr/lib64/python3.8/site-packages
 + py.test-3.8 --ignore=_build.python2 --ignore=_build.python3 --ignore=_build.pypy3 -v -W 'ignore:the imp module is deprecated:DeprecationWarning' -W 'ignore:Unknown pytest.mark.openfiles_ignore:pytest.PytestUnknownMarkWarning' --ignore docs/whatsnew
 Internet access disabled
 ============================= test session starts ==============================
 platform linux -- Python 3.8.2, pytest-5.3.5, py-1.8.1, pluggy-0.13.1 -- /usr/bin/python3
 cachedir: .pytest_cache
 Matplotlib: 3.2.1
 Freetype: 2.10.1
 hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/home/abuild/rpmbuild/BUILD/astropy-4.0.1.post1/.hypothesis/examples')
 
 Running tests with Astropy version 4.0.1.post1.
 Running tests in astropy docs.
 
 Date: 2021-06-04T21:39:57
 
 Platform: Linux-5.6.6-1-default-x86_64-with-glibc2.2.5
 
 Executable: /usr/bin/python3
 
 Full Python Version: 
 3.8.2 (default, Mar 05 2020, 18:58:42) [GCC]
 
 encodings: sys: utf-8, locale: UTF-8, filesystem: utf-8
 byteorder: little
 float info: dig: 15, mant_dig: 15
 
 Package versions: 
 Numpy: 1.18.3
 Scipy: 1.4.1
 Matplotlib: 3.2.1
 h5py: 2.10.0
 Pandas: 1.0.3
 astropy-helpers: 4.0.1
 
 Using Astropy options: remote_data: none.
 
 rootdir: /home/abuild/rpmbuild/BUILD/astropy-4.0.1.post1, inifile: setup.cfg, testpaths: astropy, docs
 plugins: mpl-0.11, arraydiff-0.3, hypothesis-5.8.0, remotedata-0.3.2, doctestplus-0.5.0, astropy-header-0.1.2, cov-2.8.1, filter-subpackage-0.0.0, asdf-2.5.2, openfiles-0.5.0
 collecting ... collected 0 items / 1 error
 
 ==================================== ERRORS ====================================
 ________________________ ERROR collecting test session _________________________
 /usr/lib/python3.8/site-packages/_pytest/config/__init__.py:459: in _importconftest
     return self._conftestpath2mod[key]
 E   KeyError: PosixPath('/home/abuild/rpmbuild/BUILD/astropy-4.0.1.post1/astropy/coordinates/tests/conftest.py')
 
 During handling of the above exception, another exception occurred:
 astropy/time/core.py:2588: in update_leap_seconds
     table = iers.LeapSeconds.auto_open(files)
 astropy/utils/iers/iers.py:1021: in auto_open
     warn('leap-second file is expired.', IERSStaleWarning)
 E   astropy.utils.iers.iers.IERSStaleWarning: leap-second file is expired.
 
 During handling of the above exception, another exception occurred:
 /usr/lib/python3.8/site-packages/_pytest/config/__init__.py:465: in _importconftest
     mod = conftestpath.pyimport()
 /usr/lib/python3.8/site-packages/py/_path/local.py:701: in pyimport
     __import__(modname)
 astropy/coordinates/__init__.py:16: in <module>
     from .builtin_frames import *
 astropy/coordinates/builtin_frames/__init__.py:29: in <module>
     from .fk5 import FK5
 astropy/coordinates/builtin_frames/fk5.py:8: in <module>
     from astropy.coordinates import earth_orientation as earth
 astropy/coordinates/earth_orientation.py:19: in <module>
     jd1950 = Time('B1950').jd
 astropy/time/core.py:390: in __new__
     update_leap_seconds()
 astropy/time/core.py:2592: in update_leap_seconds
     warn("leap-second auto-update failed due to the following "
 E   astropy.utils.exceptions.AstropyWarning: leap-second auto-update failed due to the following exception: IERSStaleWarning('leap-second file is expired.')
 
 During handling of the above exception, another exception occurred:
 /usr/lib/python3.8/site-packages/py/_path/common.py:383: in visit
     for x in Visitor(fil, rec, ignore, bf, sort).gen(self):
 /usr/lib/python3.8/site-packages/py/_path/common.py:435: in gen
     for p in self.gen(subdir):
 /usr/lib/python3.8/site-packages/py/_path/common.py:424: in gen
     dirs = self.optsort([p for p in entries
 /usr/lib/python3.8/site-packages/py/_path/common.py:425: in <listcomp>
     if p.check(dir=1) and (rec is None or rec(p))])
 /usr/lib/python3.8/site-packages/_pytest/main.py:626: in _recurse
     ihook = self.gethookproxy(dirpath)
 /usr/lib/python3.8/site-packages/_pytest/main.py:445: in gethookproxy
     my_conftestmodules = pm._getconftestmodules(fspath)
 /usr/lib/python3.8/site-packages/_pytest/config/__init__.py:437: in _getconftestmodules
     mod = self._importconftest(conftestpath)
 /usr/lib/python3.8/site-packages/_pytest/config/__init__.py:473: in _importconftest
     raise ConftestImportFailure(conftestpath, sys.exc_info())
 E   _pytest.config.ConftestImportFailure: (local('/home/abuild/rpmbuild/BUILD/astropy-4.0.1.post1/astropy/coordinates/tests/conftest.py'), (<class 'astropy.utils.exceptions.AstropyWarning'>, AstropyWarning("leap-second auto-update failed due to the following exception: IERSStaleWarning('leap-second file is expired.')"), <traceback object at 0x7f21cd993a40>))
 !!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!
 =============================== 1 error in 3.38s ===============================

dhomeier · 2020-04-29T21:09:19Z

I'm confused, is TestReading.test_read_leap_second_dat() actually causing problems? Please do post traces, as I went to some trouble to ensure that the checks on expiration time do never involve leap seconds (but obviously may well have failed!)

I may have attributed that to the wrong test; with Date: 2021-07-29T20:26:02 I got what I guess means the IERSStaleWarning from 4 separate tests:

tests/test_iers.py::TestBasic::test_simple[IERS_B]
tests/test_leap_second.py::TestDefaultAutoOpen::test_erfa_found
tests/test_leap_second.py::TestDefaultAutoOpen::test_builtin_found
tests/test_leap_second.py::TestDefaultAutoOpen::test_fake_expired_file
  /sw/lib/python3.8/site-packages/astropy/utils/iers/iers.py:1021: IERSStaleWarning: leap-second file is expired.
    warn('leap-second file is expired.', IERSStaleWarning)

bmwiedemann · 2020-04-29T21:15:21Z

test_fake_expired_file should probably remain?

mhvk · 2020-04-29T21:24:58Z

@bmwiedemann - thanks so much for sharing! This not even reaching testing we definitely need to fix!

It reminds me that there is a real problem with pytest that if one let's it error on warnings, one cannot override this for the initialization phase, before any tests are actually run. At least for days now, I've had problems with

   pytest.PytestDeprecationWarning: direct construction of DocTestModulePlus has been deprecated, please use DocTestModulePlus.from_parent

which I only seem able to solve by removing error in setup.cfg (presumably I have the wrong module, but really things should not crash just for that!). cc @astrofrog, @bsipocz who may know more...

pllim · 2020-04-29T21:31:01Z

Re: pytest.PytestDeprecationWarning -- That is caused by astropy/pytest-doctestplus#94 and is fixed by astropy/pytest-doctestplus#103

bsipocz · 2020-04-29T21:33:25Z

yes, please downgrade pytest to <5.4 for now. I'll do a pytest-doctestplus release very soon, that will have this resolved.

dhomeier · 2020-04-29T21:34:02Z

@bmwiedemann can you get at least the test collection to succeed by adding
ignore:leap-second file is expired.
to the filterwarnings section in setup.cfg (perhaps even better, make 'IERSStaleWarning' known as a warning category to pytest...).

bmwiedemann · 2020-04-30T03:02:36Z

This patch makes it pass in 2021, but not 2035 (see 2nd partial backtrace in original message)

--- astropy-4.0.1.post1.orig/setup.cfg
+++ astropy-4.0.1.post1/setup.cfg
@@ -107,6 +107,7 @@ xfail_strict = true
 qt_no_exception_capture = 1
 filterwarnings =
     error
+    ignore:leap-second file is expired.
     ignore:unclosed file:ResourceWarning
     ignore:unclosed <socket:ResourceWarning
     ignore:unclosed <ssl.SSLSocket:ResourceWarning

My python-foo is not sufficient for pytest adjustment,

bnavigator · 2020-04-30T08:39:30Z

@bmwiedemann that partial backtrace looks incomplete. The actual Exception is missing. Could you post the full buildlog or provide a link to the OBS repo?

bmwiedemann · 2020-07-05T04:15:57Z

@bnavigator https://build.opensuse.org/package/show/openSUSE:Factory/python-astropy is the OBS package

And to build in the future, you can use

osc checkout openSUSE:Factory/python-astropy && cd $_
osc build --noservice --clean --vm-type=kvm \
 --build-opt=--vm-custom-opt="-rtc base=2034-09-04T00:00:00"

bnavigator · 2020-07-05T10:52:30Z

@bmwiedemann the openSUSE package switched to "multibuild" a month ago. So the python-astropy rpm package gets built independently of the unit test runs. Is it still crucial for reproducible builds to also pass the python-astropy:test flavor with a future date?

My branch, where I started to fix the currently failing tests:
https://build.opensuse.org/package/show/home:bnavigator:branches:devel:languages:python:numeric/python-astropy

Edit: Above branched package python-astropy:tests now builds for present and future dates.

bnavigator · 2020-07-05T12:20:37Z

The fail in 2034 is because of some functions reporting it as "dubious year":

astropy.utils.exceptions.ErfaWarning: ERFA function "dtf2d" yielded 1 of "dubious year (Note 6)"

And I thought 2020 is already a dubious year 👀

bmwiedemann · 2020-07-05T19:53:21Z

Ah, I just retested it and it now passes fine with the tests split off via _multibuild.
If you like, you can close this issue.

bnavigator · 2020-07-05T20:08:22Z

So the python-astropy:test package does not need to be reproducible?

(Edit: Doesn't matter. The fix by ignoring the relevant warning messages is on its way into openSUSE:Factory)

pllim · 2020-07-06T15:16:00Z

I'll defer to @mhvk on this one. Re-reading his response in #10228 (comment) , I feel like we can close this but I am not 100% sure.

mhvk · 2020-07-06T16:43:01Z

Indeed, it would seem that the failures were "real" in that they would correctly warn that the code was not reliable for the "present time". So, let's close this. But please do not hesitate to open another issue; I very much appreciate the reproducible build efforts!!

bmwiedemann · 2023-01-07T12:27:32Z

python-gwpy seems to be indirectly suffering from this.

test_host_resolution_order_warning
   /usr/lib64/python3.8/site-packages/astropy/utils/iers/iers.py:1074: IERSStaleWarning: leap-second file is expired.
     warn('leap-second file is expired.', IERSStaleWarning)

bnavigator · 2023-01-07T12:44:25Z

@bmwiedemann does this warning prevent any package to build reproducibly? python-gwpy in openSUSE is at 2.1.5, there is an upstream update available. @badshah400 is the maintainer.

One could always add another ignore to the filterwarnings.

pllim · 2023-01-23T15:56:09Z

Hello, is this still a problem or has this been fixed up/down-stream?

bmwiedemann changed the title ~~tests fail in 2035~~ tests fail in 2021 Apr 29, 2020

pllim changed the title ~~tests fail in 2021~~ openSUSE: tests fail in 2021 Apr 29, 2020

pllim added testing time utils.iers labels Apr 29, 2020

pllim changed the title ~~openSUSE: tests fail in 2021~~ Leap second tests fail if system clock is in the future (2021-06-04) Apr 29, 2020

mhvk closed this as completed Jul 6, 2020

bnavigator mentioned this issue Jul 28, 2020

1.24 fails to build in 2021 skyfielders/python-skyfield#415

Closed

stvoutsin mentioned this issue Feb 4, 2022

Re-enabling gaiaxpy wfau/gaia-dmp#626

Merged

bmwiedemann mentioned this issue Apr 5, 2022

sunpy tests fail in future sunpy/sunpy#6030

Closed

Leap second tests fail if system clock is in the future (2021-06-04) #10228

Leap second tests fail if system clock is in the future (2021-06-04) #10228

bmwiedemann commented Apr 29, 2020 •

edited

bmwiedemann commented Apr 29, 2020 •

edited

pllim commented Apr 29, 2020

mhvk commented Apr 29, 2020

bmwiedemann commented Apr 29, 2020

mhvk commented Apr 29, 2020

dhomeier commented Apr 29, 2020

bsipocz commented Apr 29, 2020

dhomeier commented Apr 29, 2020

mhvk commented Apr 29, 2020

mhvk commented Apr 29, 2020

bmwiedemann commented Apr 29, 2020

dhomeier commented Apr 29, 2020

bmwiedemann commented Apr 29, 2020

mhvk commented Apr 29, 2020

pllim commented Apr 29, 2020

bsipocz commented Apr 29, 2020

dhomeier commented Apr 29, 2020 •

edited

bmwiedemann commented Apr 30, 2020

bnavigator commented Apr 30, 2020

bmwiedemann commented Jul 5, 2020

bnavigator commented Jul 5, 2020 •

edited

bnavigator commented Jul 5, 2020

bmwiedemann commented Jul 5, 2020

bnavigator commented Jul 5, 2020 •

edited

pllim commented Jul 6, 2020

mhvk commented Jul 6, 2020

bmwiedemann commented Jan 7, 2023

bnavigator commented Jan 7, 2023

pllim commented Jan 23, 2023

Leap second tests fail if system clock is in the future (2021-06-04) #10228

Leap second tests fail if system clock is in the future (2021-06-04) #10228

Comments

bmwiedemann commented Apr 29, 2020 • edited

Description

Expected behavior

Actual behavior

Steps to Reproduce

System Details

bmwiedemann commented Apr 29, 2020 • edited

pllim commented Apr 29, 2020

mhvk commented Apr 29, 2020

bmwiedemann commented Apr 29, 2020

mhvk commented Apr 29, 2020

dhomeier commented Apr 29, 2020

bsipocz commented Apr 29, 2020

dhomeier commented Apr 29, 2020

mhvk commented Apr 29, 2020

mhvk commented Apr 29, 2020

bmwiedemann commented Apr 29, 2020

dhomeier commented Apr 29, 2020

bmwiedemann commented Apr 29, 2020

mhvk commented Apr 29, 2020

pllim commented Apr 29, 2020

bsipocz commented Apr 29, 2020

dhomeier commented Apr 29, 2020 • edited

bmwiedemann commented Apr 30, 2020

bnavigator commented Apr 30, 2020

bmwiedemann commented Jul 5, 2020

bnavigator commented Jul 5, 2020 • edited

bnavigator commented Jul 5, 2020

bmwiedemann commented Jul 5, 2020

bnavigator commented Jul 5, 2020 • edited

pllim commented Jul 6, 2020

mhvk commented Jul 6, 2020

bmwiedemann commented Jan 7, 2023

bnavigator commented Jan 7, 2023

pllim commented Jan 23, 2023

bmwiedemann commented Apr 29, 2020 •

edited

bmwiedemann commented Apr 29, 2020 •

edited

dhomeier commented Apr 29, 2020 •

edited

bnavigator commented Jul 5, 2020 •

edited

bnavigator commented Jul 5, 2020 •

edited