MAST download should efficiently fallback on existing downloads

[pllim]

Example use case:

from astroquery.mast import Observations

files = ['jw02727-o002_t062_nircam_clear-f090w_i2d.fits',
         'jw02727-o002_t062_nircam_clear-f277w_i2d.fits'
        ]

for fn in files:
    uri = f"mast:JWST/product/{fn}"
    result = Observations.download_file(uri, local_path=fn)

I already have cached copies of these files.

Problem: When MAST server performance is degraded, it takes a really long time for download_file to complete when I just really want it to grab the cached copy. Even when successful, it would end up just grabbing the cached copy anyway.

Workaround: I have to avoid using download_file altogether, hunt down where that file was downloaded to, and use it directly.

Desired solution: When MAST server takes too long to respond (maybe user can set this timeout?), automatically stops and grab a cached copy. Throw a warning if you like. Throw error as before if no cached copy found.

cc @bmorris3

Related:

• Better caching for CBVCorrector lightkurve/lightkurve#1159

[bsipocz]

Let's not call it a "cached" copy, as it's confusing with the already existing cache, stored elsewhere from the downloaded files.

[bsipocz]

And I would say, it always fall back on the already existing downloads (unless opted out). If that's not the case, then this is clearly a bug.

[pllim]

It does, but only after a looooong wait that I know is unnecessary.

Actually I would also accept a "just get the downloaded copy, don't connect to internet at all" as solution.

[bsipocz]

"just get the downloaded copy, don't connect to internet at all"

yes, and I think that should be the default. If not available, go for the internet. And have a keyword that would opt out.

[bsipocz]

Btw, I'm refactoring the download utilities, and upstream them to pyvo. So many parts of mast will need to be refactored to use that. (and this will be the default behaviour, I really don't want to see unnecessary repeated downloads)

[aripollak]

Maybe an interim workaround for @pllim would be to lower the MAST timeout to something like 1? The default seems to be 600 seconds.

EDIT: Nevermind, I see that downloading a file still requires the request to succeed, and I'm guessing a very low timeout would just cause an exception to be raised in that case and not re-use the cache. I opened #2909 about that.