HTTP 500 error when downloading from offline LTA

[smetsb]

I have the same issue that my LTA request (as described in #344) returns HTTP 500 instead of 202, using sentinelsat v0.13 (installed with pip install). There is no more information what went wrong in the return 500, so how can I proceed ?
Apologies for resubmitting, but #344 was closed, so not sure if my comment was taken.

Example of failure (note this happens for many files I'm trying to download from LTA):
DEBUG:urllib3.connectionpool:https://cophub.copernicus.eu:443 "GET /dhus/odata/v1/Products('5cbb9774-4111-470c-86ef-9ee2cc4d40b3')?$format=json HTTP/1.1" 200 None
INFO:sentinelsat.SentinelAPI:Downloading 5cbb9774-4111-470c-86ef-9ee2cc4d40b3 to /data/cel_vol1/HRL_PHENO/S2/S2_TOA/33UVS/S2B_MSIL1C_20190228T101029_N0207_R022_T33UVS_20190228T125729.zip
WARNING:sentinelsat.SentinelAPI:Product 5cbb9774-4111-470c-86ef-9ee2cc4d40b3 is not online. Triggering retrieval from long term archive.
DEBUG:urllib3.connectionpool:https://cophub.copernicus.eu:443 "GET /dhus/odata/v1/Products('5cbb9774-4111-470c-86ef-9ee2cc4d40b3')/$value HTTP/1.1" 500 0
ERROR:sentinelsat.SentinelAPI:Trying to download an offline product

Exception
SentinelAPILTAError('Trying to download an offline product', <Response [500]>)

[j08lue]

Please try again with the latest master, i.e. pip install --upgrade https://github.com/sentinelsat/sentinelsat/archive/master.zip. There have been some improvements to LTA handling since the last release.

[smetsb]

I've updated my installation with master.zip (also v0.13?), but the issue is not solved.
I've further debugged a bit and I get into the _trigger_offline_retrieval function of the sentinelAPI class. My timeout was set to None, tried to set it to 100, but still no success. The session.auth are ok and the URL seems ok ("https://cophub.copernicus.eu/dhus/odata/v1/Products('4d843621-ee31-4034-babd-0b71aa861318')/$value") but still the session.get returns status_code 500.

Tried it also with COPHUB webclient and get a same error for the product (named S2A_MSIL1C_20190208T110221_N0207_R094_T30TXP_20190214T155730)

[j08lue]

Maybe this is a temporary service outage then?

[smetsb]

Could be, I've contacted the Copernicus EOSupport helpdesk ... as it's already temporary for 3 days ? I'll keep you posted on the outcome.

[kr-stn]

Thank you for the detailed debugging work 👍

From a sentinelsat perspective we can always say - if the product is unavailable via the web-client it will 99.9% also be unavailable with sentinelsat. In your case I think the only option forward is waiting for the support response.

For us it is rather common to see erratic behaviour from the Copernicus APIs. However, we have not found a clear debugging procedure yet that helps to determine whether the issue is with us or with the server.

Going forward we might need to create a section in the documentation that shows 1) How to get the full product URL for the failing download and then 2) how to check the product with the web-client (if one is available). This problem is compounded by the many different DHuS instances with their different support channels, subtly different behaviours and unclear availability of web-clients.

[smetsb]

Got a reply from Copernicus EO Support:

• Please be advised that a user quota on the maximum number of requests per hour per user is set. Note that if you retrieve data directly from the GUI, you can retrieve only 1 product per half hour while, from APIHub 20 products each 12h.
• Moreover, please be informed that during the last period, an increased number of requests arriving at the same time has challenged the current set up causing quota problems of the retrieval of Copernicus Sentinel-2 products from the Long Term Archive (LTA). As result, error messages are received randomly from users when a retrieval request is launched. Please be informed that measures are being taken in order to solve the issue

So it's clearly an issue at server side. I've also requested to consider having another HTTP return code in case the quota for user is reached.
Thanks for your support.

[kr-stn]

I've also requested to consider having another HTTP return code in case the quota for user is reached.

That would indeed be ideal, then we could surface that in sentinelsat to tell users whether it`s a true server error or just the quota.

[neteler]

Please try again with the latest master, i.e. pip install --upgrade https://github.com/sentinelsat/sentinelsat/archive/master.zip. There have been some improvements to LTA handling since the last release.

When will this new code be officially released (I suppose in sentinelsat v0.14)?

[kr-stn]

@neteler Yes - definitely included in the next release.

We are long overdue for a release anyway. This was mostly blocked due to my full schedule. I'll try to get v0.14 out this week.