Timeouts greater than 10 s are ignored

[Kodiologist]

With httr 1.4.2:

> httr::GET("https://e4ftl01.cr.usgs.gov/MOTA/MCD19A2.006/2005.01.01", httr::timeout(1))
Error in curl::curl_fetch_memory(url, handle = handle) :
  Timeout was reached: [e4ftl01.cr.usgs.gov] Operation timed out after 1001 milliseconds with 0 out of 0 bytes received
> httr::GET("https://e4ftl01.cr.usgs.gov/MOTA/MCD19A2.006/2005.01.01", httr::timeout(9))

Error in curl::curl_fetch_memory(url, handle = handle) :
  Timeout was reached: [e4ftl01.cr.usgs.gov] Operation timed out after 9000 milliseconds with 0 out of 0 bytes received
> httr::GET("https://e4ftl01.cr.usgs.gov/MOTA/MCD19A2.006/2005.01.01", httr::timeout(10))

Error in curl::curl_fetch_memory(url, handle = handle) :
  Timeout was reached: [e4ftl01.cr.usgs.gov] Operation timed out after 10001 milliseconds with 0 out of 0 bytes received
> httr::GET("https://e4ftl01.cr.usgs.gov/MOTA/MCD19A2.006/2005.01.01", httr::timeout(11))

Error in curl::curl_fetch_memory(url, handle = handle) :
  Timeout was reached: [e4ftl01.cr.usgs.gov] Operation timed out after 10001 milliseconds with 0 out of 0 bytes received
> httr::GET("https://e4ftl01.cr.usgs.gov/MOTA/MCD19A2.006/2005.01.01", httr::timeout(12))

Error in curl::curl_fetch_memory(url, handle = handle) :
  Timeout was reached: [e4ftl01.cr.usgs.gov] Operation timed out after 10001 milliseconds with 0 out of 0 bytes received

I believe this is a duplicate of r-lib/httr#501. Unfortunately, my example is only reproducible so long as this website is so slow.

I think the cause is upstream, where a 10-s timeout is hard-coded: https://github.com/jeroen/curl/blob/395fbcbc13ef7f38167d3837f50c3cad51902068/src/curl.c#L108

[javieriserte]

I'm having the same problem. There is any workaround to this?

[DenchPokepon]

• same problem

[wilsonte-umich]

Bumping this open issue, to try find a resolution.

I also see that both:

httr::GET(url)

and

httr::GET(url, httr::timeout(x)) with x >10

always time out at 10 seconds when a server is slow to respond - which a server I'm using often is, making it very difficult to use httr::GET to request from the server.

[hadley]

httr has been superseded in favour of httr2, so is no longer under active development. If this problem is still important to you in httr2, I'd suggest filing an issue offer there 😄. Thanks for using httr!

[DenchPokepon]

httr::timeout doesn't do what it's supposed to do, because the actual setting for connection timeout is "connectiontimeout", not "timeout_ms" as it's currently written inside httr::timeout function

The workaround for the timeout problem is

GET(url, config(connecttimeout = seconds))

config function is a direct interface to curl settings

[hadley]

Can you provide some evidence? https://curl.se/libcurl/c/CURLOPT_TIMEOUT_MS.html looks what I expect — it's a timeout for the entire connection, and indeed https://curl.se/libcurl/c/CURLOPT_CONNECTTIMEOUT.html says:

The connection timeout set with CURLOPT_CONNECTTIMEOUT is included in the general all-covering CURLOPT_TIMEOUT.

Am I reading the docs wrong?

[DenchPokepon]

I don't really know anything about the original C curl library, but I found this setting (connecttimeout) in the curl defaults for handles in the R curl package https://github.com/jeroen/curl/blob/1bcf1ab3b4c811f4706a03de8e0dc675e1403315/src/handle.c#L159C50-L159C50

I coudn't find timeout_ms anywhere there, so I tried GET(url, config(connecttimeout = seconds)) and it worked on my slow Internet (I assume it's slow because of the slow dns lookup on the ISP side, but it's just a guess)

Before I coudn't use httr or R curl functions at all on my home machine, but now i can

[hadley]

@DenchPokepon oooh that makes sense, because curl is setting a maximum and httr's setting can only make this smaller, not large. I'll fix this in httr2 since httr is superseded.

[DenchPokepon]

@hadley Thank you very much for your work!

I confirm, the current implementation of httr::timeout can indeed make the timeout smaller, but not more than 10 seconds