Timeout reached while waiting for CURLMOPT_MAX_HOST_CONNECTIONS slot

[jeroen]

A user has reported the following issue in R:

If you perform several concurrent requests to a host in a multi, curl will automatically queue them once you reach CURLMOPT_MAX_HOST_CONNECTIONS connections.

However handles that have a CURLOPT_TIMEOUT set will yield a timeout error immediately when it's their turn, because the timeout has been reached waiting in the curl queue.

Is this intended? I would have expected the time to start counting once the request has started.

[bagder]

I think it would be even stranger if it didn't apply.

Basically, the timeout is now "the longest time you allow the transfer to be in progress until it reaches a connected state". If the timeout wouldn't apply, would it be completely timeout-less then while in that state, which basically could last "forever"?

[jeroen]

Hmm I don't exactly know how the queue works, but my interpretation of maximum time the request is allowed to take is that we start counting when the request is actually initiated :-) But perhaps I am thinking about a different context/application.

The goal of the R user was to scrape a lot of pages from many different servers, but not get stuck idling for a long time on very slow hosts. But we dit not expect curl would terminate requests to healthy hosts that were in the queue to be performed.

[bagder]

But for the application, the "request" starts when the handle is added to the multi handle. An application doesn't typically know or need to know the state of the transfer. In your described case here, the application says to libcurl that the entire transfer may not take longer than N so libcurl tries to keep that limit. The fact that it didn't even get to the connect yet when that time is reached is an internal detail that I don't think changes the promise nor what the user asked for. If the user wants to limit the number of connections that therefore can delay the start of the transfer, it should allow a (much) longer time for it.

[jeroen]

OK fair enough! I had assumed the request technically starts when the request is being requested :-) I can see either interpretation could make sense, depending on the sort of application.

I recommended the user to explore the other timeout options to accomplish what he wants.