Timeout MODIS server error when downloading

[agoodman312]

Issue description

I am experiencing a timeout error when I try to download MODIS tiles from MOD44B.006 from 2000-2020, it says Timeout is reached after 10001 milliseconds, error: http server seems to be down! try again later. Aborting!
-I tried increasing the httr timeout time in the source code, following the instructions of previous posts but that did not work
-I tried redoing the analysis throughout the week because I know the MODIS servers are down on wednesdays

Reproducible example

## PLEASE DELETE AND WRITE YOUR OWN
library(MODIStsp)
MODIStsp(gui = FALSE, opts_file = "/user/desktop/test.json")

test.txt

Expected and actual behavior

## PASTE HERE THE OUTPUT OF YOUR EXAMPLE CODE

GDAL version in use: 3.2.1

Listening on http://127.0.0.1:4217
old-style crs object detected; please recreate object with a recent sf::st_crs()
old-style crs object detected; please recreate object with a recent sf::st_crs()
old-style crs object detected; please recreate object with a recent sf::st_crs()
old-style crs object detected; please recreate object with a recent sf::st_crs()
old-style crs object detected; please recreate object with a recent sf::st_crs()
old-style crs object detected; please recreate object with a recent sf::st_crs()
old-style crs object detected; please recreate object with a recent sf::st_crs()
old-style crs object detected; please recreate object with a recent sf::st_crs()
Reading layer OGRGeoJSON' from data source {
"type": "FeatureCollection",
"features": [
{
"type": "Feature",
"properties": {
"X_leaflet_id": 1829,
"layerId": "1829",
"edit_id": "1829"
},
"geometry": {
"type": "Polygon",
"coordinates": [
[
[123.948441, -37.049149],
[129.925003, -35.347056],
[134.671097, -38.439069],
[138.362503, -41.404111],
[142.405472, -44.365588],
[146.448441, -45.731588],
[150.667191, -44.491118],
[152.249222, -40.874584],
[153.479691, -37.747372],
[155.764847, -33.315037],
[156.643753, -29.566888],
[156.467972, -25.515798],
[155.589066, -21.486936],
[151.897659, -18.847162],
[147.678909, -13.446391],
[146.800003, -12.07491],
[143.811722, -9.138029],
[137.659378, -8.443168],
[131.155472, -8.443168],
[119.378128, -12.246747],
[113.753128, -17.511141],
[110.061722, -21.323279],
[107.952347, -27.09153],
[108.655472, -33.900627],
[113.577347, -38.576622],
[121.663284, -39.802734],
[123.948441, -37.049149]
]
]
}
}
]
}'
using driver `GeoJSON'
Simple feature collection with 1 feature and 3 fields
Geometry type: POLYGON
Dimension: XY
Bounding box: xmin: 107.9523 ymin: -45.73159 xmax: 156.6438 ymax: -8.443168
Geodetic CRS: WGS 84
old-style crs object detected; please recreate object with a recent sf::st_crs()
old-style crs object detected; please recreate object with a recent sf::st_crs()
old-style crs object detected; please recreate object with a recent sf::st_crs()
old-style crs object detected; please recreate object with a recent sf::st_crs()
old-style crs object detected; please recreate object with a recent sf::st_crs()
old-style crs object detected; please recreate object with a recent sf::st_crs()
old-style crs object detected; please recreate object with a recent sf::st_crs()
old-style crs object detected; please recreate object with a recent sf::st_crs()
[Wed Sep 29 00:47:56 2021] MODIStsp --> Starting processing
[Wed Sep 29 00:47:56 2021] Accessing http server at: https://e4ftl01.cr.usgs.gov/MOLT/MOD44B.006/
Error in curl::curl_fetch_memory(url, handle = handle): Timeout was reached: [e4ftl01.cr.usgs.gov] Operation timed out after 10003 milliseconds with 0 out of 0 bytes received
Request failed [ERROR]. Retrying in 1 seconds...
[Wed Sep 29 00:48:50 2021] Error: http server seems to be down! Please try again later. Aborting!

System information

# PASTE HERE YOUR OUTPUT

Additional context

Scientific publication

[ranghetti]

Timeout problems may depend on temporary unavailability (server side), archived products (server side) or problems with internet connection (client side).

I'm seeing products can be downloaded correctly, so I can exclude server-side problems:

GDAL version in use: 3.0.4
[Wed Sep 29 10:08:59 2021] MODIStsp --> Starting processing
[Wed Sep 29 10:08:59 2021] Accessing http server at:  https://e4ftl01.cr.usgs.gov/MOLT/MOD44B.006/
[Wed Sep 29 10:09:06 2021] Retrieving list of available ` Terra ` Files for Year 2000
[Wed Sep 29 10:09:17 2021] Downloading Terra Files for date: 2000_03_05 : 1 of:  32
[Wed Sep 29 10:09:18 2021] Downloading Terra Files for date: 2000_03_05 : 2 of:  32
[Wed Sep 29 10:09:21 2021] Downloading Terra Files for date: 2000_03_05 : 3 of:  32
[Wed Sep 29 10:09:47 2021] Downloading Terra Files for date: 2000_03_05 : 4 of:  32

Please try again (supposing there were temporary server-side problems, now solved) or check your internet connection / try using a different machine (supposing client-side problems).

[micha-silver]

I'm also seeing this problem. Using two different machines (with different internet connections) and thru the week. The issue was reported previously. And at that time there was a PR to increase the timeout to 20 seconds. Was this implemented? I still see a 10 sec timeout, with only 5 retries.
Thanks

[ranghetti]

The PR (#220) was reverted with 06c39d3 because it was causing other problems (now I did not remenber what, since I did not annontate them).
If you want to change it on your side you can fork the reporisotry and change the source code similarly to 231fee3 . In the future I will consider to manage the timeout duration as a function argument.

[micha-silver]

body p { margin-bottom: 0cm; margin-top: 0pt; } Thanks very much. I will try On 06/12/2021 11:12, Luigi Ranghetti wrote: The PR (#220) was reverted with 06c39d3 because it was causing other problems (now I did not remenber what, since I did not annontate them). If you want to change it on your side you can fork the reporisotry and change the source code similarly to 231fee3 . In the future I will consider to manage the timeout duration as a function argument. — You are receiving this because you commented. Reply to this email directly, view it on GitHub, or unsubscribe. Triage notifications on the go with GitHub Mobile for iOS or Android. [ { ***@***.***": "http://schema.org", ***@***.***": "EmailMessage", "potentialAction": { ***@***.***": "ViewAction", "target": "#243 (comment)", "url": "#243 (comment)", "name": "View Issue" }, "description": "View this Issue on GitHub", "publisher": { ***@***.***": "Organization", "name": "GitHub", "url": "https://github.com" } } ]

…

-- Micha Silver Ben Gurion Univ. Sde Boker, Remote Sensing Lab cell: +972-523-665918 https://orcid.org/0000-0002-1128-1325