XrdHttp loses requests under modest concurrency

[bbockelm]

After applying a workaround for issue #809, I notice that ab will quite consistently fail with only few number of repeated requests:

$ ab -k -s 30 -n 400 -c 100 http://host.example.com/some/path/hello_world
This is ApacheBench, Version 2.3 <$Revision: 1430300 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking hcc-briantest7.unl.edu (be patient)
Completed 100 requests
Completed 200 requests
Completed 300 requests
apr_pollset_poll: The timeout specified has expired (70007)
Total of 398 requests completed

This is a fairly vanilla setup - should be reproducible with the HTTP module on top of a POSIX filesystem.

I haven't been able to diagnose it precisely, but it seems to only occur when this line pops into the log:

180828 16:07:34 880  ofs_Stall: Stall 3: File hello_world is being staged; estimated time to completion 3 seconds for /some/path/hello_world

I admit, I don't understand why the file would ever be considered as being staged with the default OFS plugin. However, the behavior is very much as if there's a callback not occurring.

NOTE this issue is based on an investigation into user complaints about the service. It may be a synthetic benchmark but it shows something actually observed in the wild.

[bbockelm]

I should mention the workaround to get ab working is this one:

--- a/src/XrdHttp/XrdHttpProtocol.cc
+++ b/src/XrdHttp/XrdHttpProtocol.cc
@@ -1376,6 +1376,7 @@ int XrdHttpProtocol::StartSimpleResp(int code, const char *desc, const char *hea
     else ss << "Unknown";
   }
   ss << crlf;
+  ss << "Connection: Keep-Alive" << crlf;
 
   if (bodylen >= 0) ss << "Content-Length: " << bodylen << crlf;
 

Not a complete fix to #809, but good enough to enable testing.

[bbockelm]

The ofs_Stall is not coming from the OSS but from here:

https://github.com/xrootd/xrootd/blob/master/src/XrdOfs/XrdOfsHandle.cc#L203

It appears there's a modest contention on the file descriptor table -- one that does not particularly play well with what appears to be an ad-hoc implementation of a timed lock:

https://github.com/xrootd/xrootd/blob/master/src/XrdOfs/XrdOfsHandle.cc#L504

It's not obvious why one would utilize that instead of a wrapper around pthread_mutex_timedlock; it appears the hand-rolled version has similar guarantees as the standard function but worse performance.

[abh3]

Well, because at the time the code was written (oh about 18 years ago) pthread_timed_lock didn’t exist in Solaris. From: Brian Bockelman Sent: Tuesday, August 28, 2018 6:43 PM To: xrootd/xrootd Cc: Subscribed Subject: Re: [xrootd/xrootd] XrdHttp loses requests under modest concurrency (#810) The ofs_Stall is not coming from the OSS but from here: https://github.com/xrootd/xrootd/blob/master/src/XrdOfs/XrdOfsHandle.cc#L203 It appears there's a modest contention on the file descriptor table -- one that does not particularly play well with what appears to be an ad-hoc implementation of a timed lock: https://github.com/xrootd/xrootd/blob/master/src/XrdOfs/XrdOfsHandle.cc#L504 It's not obvious why one would utilize that instead of a wrapper around pthread_mutex_timedlock; it appears the hand-rolled version has similar guarantees as the standard function but worse performance. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or mute the thread.

[bbockelm]

Ok, found the culprit. It's here:

https://github.com/xrootd/xrootd/blob/master/src/XrdXrootd/XrdXrootdTransit.cc#L418

If a stall occurs (you can tweak the XrdOfsHandle::WaitLock code to never wait and this causes a stall easily), then the XrdXrootdProtocol object is scheduled to re-run at a later point. It's re-run in that function.

However, it doesn't appear to invoke the appropriate callback for realProt->Process as in the other overload of XrdXrootdTransit::Process, meaning the callback doesn't get pushed until the client itself causes activity (such as disconnecting).