Sporadic crashes in curl_multi_remove_handle

[Rider-Linden]

I'm encountering a sporadic crash in curl_multi_remove_handle, it appears to be an interaction with timeouts and pipelining very much like what is described in this thread: http://curl.haxx.se/mail/lib-2011-09/0305.html

Unfortunately, the conversation appears to trail off right after "A-ha! Pipelining!". Was any resolution ever reached?

For the record we are seeing this in 7.42.1 and my call stack looks like:

[0] Curl_getoff_all_pipelines [secondlife-bin url.c]
[1] curl_multi_remove_handle [secondlife-bin multi.c]
[2] LLCore::HttpLibcurl::completeRequest(void *,void *,CURLcode) [secondlife-bin _httplibcurl.cpp]
[3] LLCore::HttpLibcurl::processTransport() [secondlife-bin _httplibcurl.cpp]
[4] LLCore::HttpService::threadRun(LLCoreInt::HttpThread *) [secondlife-bin _httpservice.cpp]
[5] boost::detail::function::function_obj_invoker1,boost::_bi::list2,boost::arg<1> > >,LLVisualParam *,int>::invoke(boost::detail::function::function_buffer &,int) [secondlife-bin function_template.hpp]
[6] LLCoreInt::HttpThread::run() [secondlife-bin _thread.h]
[7] boost::detail::thread_data >::run() [secondlife-bin thread.hpp]

I have not been able to get a local reproduction yet... I'm only seeing this as reports from the wild.

[bagder]

I can't recall any specific fix and I can't find any in the git log now either, so I would probably say that we didn't get anywhere on that.

So you are using pipelining too?

[Rider-Linden]

Yes, We're using pipelining primarily for resource retrieval from our CDN (Textures, Meshes, etc.)

Reconstructing what I can from the user logs associated with this crash I'm seeing a number of failed or timedout pulls from the CDN, so I realize that I am making an assumption here.

[bagder]

I guess you can't reproduce it? Can you get more debugging info in the stack trace?

[bagder]

Pipelining is not a very widely used feature so it wouldn't surprise me if there are a few bugs lurking in those corners. HTTP/2 is a much better pipelining so you might want to try to move over to that.

Without any steps to reproduce or further details on this problem there's a not a lot we can do. I'm closing this, but if more or further detail will arise we can always reopen or create a new bug .

[Rider-Linden]

Sorry... I just saw your email...
I was finally able to reproduce and track down what was going on. It appears that this was a regression from an old bug referred to as #1420. On a slow connection with a large number of pipelined requests some of the requests would time out. After the timeout they would be marked for deletion. However, url.c IsPipeliningPossible and bundle checking loop in ConnectionExists would potentially reuse these doomed connections.

There were a number of patches from Carlo Wood from several years ago that addressed the issue, but it would seem that his changes to url.c were lost in the shuffle. I've reintroduced the checks in url.c and will submit a pull request.

[bagder]

Lovely, looking forward to that. Yes, Carlo Wood did a lot of good work but we never got a completed patch set and then he vanished.