-
Notifications
You must be signed in to change notification settings - Fork 149
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[SSI] Problems with timeouts during SSI streaming #829
Comments
Isn't this due to the default or you changed that? |
I set the timeout resolution to 1s, that just affects how often XRootD checks if we exceeded a timeout. |
TO answer Andreas’ question, the defaults must have been changed in the SSI context as SSI sets these two to infinity. Anyway, I’ve long complained about the timeout resolution option. It’s very coarse and I could never see anyone setting it to a value that would actually be adhered to over the long run. That obviously is not the only problem here but is likely a contributing factor.
From: Michael
Sent: Tuesday, September 25, 2018 7:30 AM
To: xrootd/xrootd
Cc: Subscribed
Subject: Re: [xrootd/xrootd] [SSI] Problems with timeouts during SSI streaming (#829)
I set the timeout resolution to 1s, that just affects how often XRootD checks if we exceeded a timeout.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub, or mute the thread.
|
I did change the timing resolution but this is not the problem. The problems are: (a) the behaviour of XRD_REQUESTTIMEOUT and XRD_STREAMTIMEOUT is not documented and it is difficult to understand how they are intended to behave/interact (b) there is no way to independently control the timeout for obtaining the stream handle from the server (which should be short, in the order of 10 or 15 seconds) and the timeout for obtaining a chunk of data from the stream (which can be arbitrarily long, in the order of minutes). |
OK after discussions with Andy and Michal, the key piece of information I was missing is that the concepts of "Request" and "Stream" in SSI do not map to XRD_REQUESTTIMEOUT and XRD_STREAMTIMEOUT.
This could be made clear in the documentation. For the case of independently controlling the the timeout for obtaining the stream handle and the timeout for obtaining a chunk of data from the stream, it seems this is possible by changing the Request timeout (which can be set on a per-request basis) after the stream handle is received. An alternative would be to allow the server to set the waitresp time. Anyway for now I think we understand how it works and have enough to do what we need. |
When using SSI streams, there are two environment variables which control the length of timeouts,
XRD_REQUESTTIMEOUT
andXRD_STREAMTIMEOUT
. It is not clear how these two timeouts are supposed to interact and in some cases the behaviour is strange.Example setup: I have a simple XRootD SSI server which does round-robin requests. The client sends a simple payload and a number of records to return, and the server sends back the payload the requested number of times in an SSI stream. For this test, the server adds a 5-second delay to each chunk of the stream, to better observe how the timeouts interact.
Testing with
test-client stream 1000
, which returns 1000 records in a stream. The requested buffer size is 1024 bytes and the payload is in the order of 100 bytes, so we expect ~10 chunks to be returned on the stream. As we added a 5 sec delay to each chunk, the total processing time is around 50s.CASE 1. XRD_REQUESTTIMEOUT=1, XRD_STREAMTIMEOUT=2
In this case, the server returns the stream handle within the 1s, but it is still the REQUEST timeout which triggers, not the STREAM timeout.
CASE 2. XRD_REQUESTTIMEOUT=3, XRD_STREAMTIMEOUT=2
Same as above, server responds in time, but the REQUEST timeout is triggered, not the STREAM timeout.
CASE 3. XRD_REQUESTTIMEOUT=6, XRD_STREAMTIMEOUT=2
This time the STREAM timeout is triggered, and the error is
Socket error
instead ofOperation expired
.CASE 4. XRD_REQUESTTIMEOUT=6, XRD_STREAMTIMEOUT=7
In this case, the client receives 9 of the 10 chunks, but times out on the last one with
Operation expired
.Here are the server-side logs for this case:
CASE 5. XRD_REQUESTTIMEOUT=6, XRD_STREAMTIMEOUT=60
This gives exactly the same behaviour as above.
CASE 6. XRD_REQUESTTIMEOUT=15, XRD_STREAMTIMEOUT=6
This is the same as above, except it gives a
Socket error
instead ofOperation expired
.CASE 7. XRD_REQUESTTIMEOUT=11, XRD_STREAMTIMEOUT=60
Finally, success.
SUMMARY
The behaviour of XRD_REQUESTTIMEOUT and XRD_STREAMTIMEOUT and their interaction is not documented, so I am trying to figure out how it should work by trial and error.
What I was expecting is that XRD_REQUESTTIMEOUT would place a limit on the time to receive the stream handle from the server, and XRD_STREAMTIMEOUT would place a limit on the time to receive each chunk of the stream. But it seems this is not the case.
It looks like XRD_REQUESTTIMEOUT is used for two distinct purposes, one for the time to receive the stream handle, and also for the time to receive each chunk of the stream. Also XRD_REQUESTTIMEOUT needs to be set to at least double the time taken to process one chunk, as the final chunk of the stream requires two calls to the stream object (one to partially fill the buffer with the last remaining data, then again to set
last = true
and returnnullptr
).XRD_STREAMTIMEOUT is used to control the total amount of time to receive the entire stream, and there is no way to independently control the timeout on each chunk of the stream.
Please can you confirm that this is the intended behaviour of these timeouts?
XRD_STREAMTIMEOUT is not so useful in its current form. We need a way to limit the time taken to process each chunk of the stream, not the total time.
Also we need to be able to control the time taken to process each chunk of the stream independently from the request timeout.
The text was updated successfully, but these errors were encountered: