IERS URL timing out

[bmorris3]

The IERS tables that describe Earth's orientation ordinarily get downloaded from this URL stored in the utils.iers module:

In [18]: from astropy.utils import iers

In [19]: iers.IERS_A_URL
Out[19]: 'http://maia.usno.navy.mil/ser7/finals2000A.all'

At present (and maybe for some time now), the above URL times out always. I found from the IERS webpage that there's an alternative server which is serving the table appropriately here, at

http://toshi.nofs.navy.mil/ser7/finals2000A.all

This is causing a bunch of mysterious IERS warnings and errors in astroplan. Should we switch the hard-coded URL in astropy to the backup URL until the normal server is restored?

I checked to see if the IERS webpage maintainers have an email address that we can contact, but they left only a phone number on the webpage (for the adventurous: 202-762-1518).

cc @eteq (@kvyh – heads up, astroplan tests may fail due to this problem even if the code you wrote is fine)

[pllim]

Could I be seeing the same thing at #5051(comment)? c/c @taldcroft , @mhvk

[bmorris3]

That's possible, @pllim. The debugging discussion happening in that thread is similar to the one I was having in my own head a few minutes ago.

[mhvk]

The URL seems to work again (just pasted it in my browser). The fact that you are getting "mysterious IERS warnings" does suggest though that the IERS code may need to handle the timeouts more graciously, or have warnings that make clearer what is happening.

[pllim]

For unreliable test, perhaps mark it as xfail under certain conditions? In this case, it would be a timeout error or HTTPError.

[taldcroft]

Well I certainly hope we have not erred in automatically using the IERS web site.

However, please do note that it is easy to opt out of this behavior and return to handling IERS-related transforms manually. See http://astropy.readthedocs.io/en/latest/utils/iers.html. In particular the configuration parameters section and subsequent sections talk about how to disable auto-download or do things manually.

[taldcroft]

It would be technically feasible to try a backup website if the primary fails. I don't know enough about their redundancy to know if that is actually valuable or if it would just mean taking twice as long to fail.

[astrofrog]

We could always use data.astropy.org as a mirror?

[taldcroft]

Can we set a cron job to keep data.astropy.org up to date?

[astrofrog]

I'm sure we could - I guess the question is just on which server to set up the cron job.

[eteq]

@bmorris3 - can you clarify where in the IERS web site it points you to there? All the links I saw there (in admittedly a cursory glance) are at datacenter.iers.org... which @mhvk investigated and got an official answer from the IERS folks that it is a mirror that of the maia copy. I wonder if that should be the fallback, rather than toshi?

[taldcroft]

@eteq et al. - do you think adding support for fallback sites is release-critical? I think I know how to do it, but it's a question of whether we (1) should do this, and (2) if it is in 1.2.

[bmorris3]

@eteq – the IERS webpage directs you to this webpage at the datacenter.iers.org address, which provides these URLS as the "Source:"

http://maia.usno.navy.mil/ser7/ [ser7.dat]

Alternative server:
http://toshi.nofs.navy.mil/ser7/

[mhvk]

I think the datacenter.iers.org would be a good enough backup. I don't think it makes much sense to add yet another copy at data.astropy.org.

That said, most important would seem to be to have a sufficiently informative error message; I'm fine with just sticking to one server.

[astrofrog]

@taldcroft - I think a fallback mechanism and a better error if both fail would make sense. If you think you can do this in the next few hours, I can delay the release candidate - I would really like to get the RC out today or tomorrow morning at the very latest.

[taldcroft]

@astrofrog - I agree with @mhvk that this can wait. I think that the existing warning would be sufficient for programmatic use, it's just that emitted warnings don't show up in testing. And the situation that you encountered today is really just a problem in testing. It failed because the real IERS-A file was already loaded and it didn't pick up the stubbed version for testing.

[pllim]

@bmorris3 is this still an issue? The last update was in 2016, so just want to make sure.

[bmorris3]

@pllim It is, see all of the references to related issues above.

[bsipocz]

However it's a duplicate with #5194. I would suggest to close one of them, maybe the other issue has more concrete suggestion of how to work around, but I really have no preferences which one to close.

[pllim]

Does #8308 address this as well?

[mhvk]

Agreed with @pllim that #8308 addresses at least the basic point here, so closing. Probably best to open a new issue if there are still problems.