newexp-mock fails to download IERS file

[weaverba137]

Minitest notebook failed on the wrap-newexp stage. One of the newexp-mock jobs failed. Why is this file not already cached at NERSC? Maybe the job couldn't find the cached copy? Maybe desisim/specsim should cache this file so that downloads don't happen in the pipeline?

Here's the log output:

INFO:newexp_mock.py:73:main: Simulating night 20200315 expid 10 tile 28408
DEBUG:simexp.py:408:simulate_spectra: loading specsim desi config desi
DEBUG:simexp.py:412:simulate_spectra: creating specsim desi simulator
Downloading http://maia.usno.navy.mil/ser7/finals2000A.all [Failed]
WARNING: failed to download http://maia.usno.navy.mil/ser7/finals2000A.all: <urlopen error timed out>.
A coordinate or time-related calculation might be compromised or fail because the dates are not covered by the available IERS file.  See the "IERS data access" section of the astropy documentation for additional information on working offline. [astropy.utils.iers.iers]
WARNING: Tried to get polar motions for times after IERS data is valid. Defaulting to polar motion from the 50-yr mean for those. This may affect precision at the 10s of arcsec level [astropy.coordinates.builtin_frames.utils]
Traceback (most recent call last):
  File "/global/common/software/desi/cori/desiconda/20180709-1.2.6-spec/code/desisim/master/bin/newexp-mock", line 9, in <module>
    sys.exit(main())
  File "/global/common/software/desi/cori/desiconda/20180709-1.2.6-spec/code/desisim/master/py/desisim/scripts/newexp_mock.py", line 82, in main
    obsconditions=obs, psfconvolve=False)
  File "/global/common/software/desi/cori/desiconda/20180709-1.2.6-spec/code/desisim/master/py/desisim/simexp.py", line 294, in simscience
    obsconditions=obsconditions, psfconvolve=psfconvolve)
  File "/global/common/software/desi/cori/desiconda/20180709-1.2.6-spec/code/desisim/master/py/desisim/simexp.py", line 431, in simulate_spectra
    desi.observation.exposure_start = astropy.time.Time(obsconditions['MJD'], format='mjd')
  File "/global/common/software/desi/cori/desiconda/20180709-1.2.6-spec/code/specsim/master/specsim/observation.py", line 115, in exposure_start
    self._update_model()
  File "/global/common/software/desi/cori/desiconda/20180709-1.2.6-spec/code/specsim/master/specsim/observation.py", line 79, in _update_model
    self.pointing, self.observing_model)
  File "/global/common/software/desi/cori/desiconda/20180709-1.2.6-spec/code/specsim/master/specsim/transform.py", line 392, in sky_to_altaz
    altaz_out = sky_coords.transform_to(observing_model)
  File "/global/common/software/desi/cori/desiconda/20180709-1.2.6-spec/conda/lib/python3.6/site-packages/astropy/coordinates/sky_coordinate.py", line 482, in transform_to
    new_coord = trans(self.frame, generic_frame)
  File "/global/common/software/desi/cori/desiconda/20180709-1.2.6-spec/conda/lib/python3.6/site-packages/astropy/coordinates/transformations.py", line 1314, in __call__
    curr_coord = t(curr_coord, curr_toframe)
  File "/global/common/software/desi/cori/desiconda/20180709-1.2.6-spec/conda/lib/python3.6/site-packages/astropy/coordinates/transformations.py", line 914, in __call__
    return supcall(fromcoord, toframe)
  File "/global/common/software/desi/cori/desiconda/20180709-1.2.6-spec/conda/lib/python3.6/site-packages/astropy/coordinates/builtin_frames/cirs_observed_transforms.py", line 58, in cirs_to_altaz
    get_dut1utc(obstime),
  File "/global/common/software/desi/cori/desiconda/20180709-1.2.6-spec/conda/lib/python3.6/site-packages/astropy/coordinates/builtin_frames/utils.py", line 87, in get_dut1utc
    return time.delta_ut1_utc
  File "/global/common/software/desi/cori/desiconda/20180709-1.2.6-spec/conda/lib/python3.6/site-packages/astropy/time/core.py", line 1246, in _get_delta_ut1_utc
    delta = iers_table.ut1_utc(jd1, jd2)
  File "/global/common/software/desi/cori/desiconda/20180709-1.2.6-spec/conda/lib/python3.6/site-packages/astropy/utils/iers/iers.py", line 229, in ut1_utc
    self.ut1_utc_source if return_status else None)
  File "/global/common/software/desi/cori/desiconda/20180709-1.2.6-spec/conda/lib/python3.6/site-packages/astropy/utils/iers/iers.py", line 327, in _interpolate
    self._check_interpolate_indices(i1, i, np.max(mjd))
  File "/global/common/software/desi/cori/desiconda/20180709-1.2.6-spec/conda/lib/python3.6/site-packages/astropy/utils/iers/iers.py", line 602, in _check_interpolate_indices
    raise ValueError(INTERPOLATE_ERROR)
ValueError: interpolating from IERS_Auto using predictive values that are more
than {} days old.

Normally you should not see this error because this class
automatically downloads the latest IERS-A table.  Perhaps you are
offline?  If you understand what you are doing then this error can be
suppressed by setting the auto_max_age configuration variable to
``None``:

  from astropy.utils.iers import conf
  conf.auto_max_age = None

[sbailey]

Odd. Is this reproducible for you? I just tried it and didn't have any problems, including no messages about Downloading these data. If it is reproducible for you, can you post your exact wrap-newexp command and log file location (should be printed in the notebook cell output)?

[weaverba137]

The text I pasted above is the log file. But my point is that this error should never happen at all because that file should be pre-downloaded and cached by the pipeline.

[sbailey]

I agree that it should never happen, but my starting point was to try to figure out if it always happens for you or only sometimes happens for you, and why it didn't happen for me. Digging deeper, astropy.config.get_cache_dir() reports that my cache dir is /global/homes/s/sjbailey/.astropy/cache. I cleared my cache using astropy.utils.data.clear_download_cache() and then reran wrap-newexp, and indeed 3 of the 15 MPI ranks tried to simultaneously download the same file (ugh) though I didn't get the same end failure that you did (lucky).

For the purposes of this data release, I suggest that you file a ticket in desisim to improve this later, but just re-run the notebook to use your new cache and proceed (or otherwise report if it remains a problem).

Related work to add to the desisim ticket: desisurvey.utils.freeze_iers() has code for caching the IERS table.

[dkirkby]

This sounds like the type of problem that desisurvey.utils.freeze_iers() is designed to solve. Is there some reason not to use it here?

[weaverba137]

This time all the wrap-newexp jobs completed successfully. I've already submitted a ticket in desisim. This is probably safe to close, but two questions before we do:

1. Should the desisim ticket be added to 18.6?
2. Should the freeze_iers functionality be moved to desiutil, so that we're not duplicating code?

[sbailey]

1. Should the desisim ticket be added to 18.6?

No.

2. Should the freeze_iers functionality be moved to desiutil, so that we're not duplicating code?

Maybe.

Let's deal with this separately from the 18.6 release and this run of desitest.