Close file handle for simple HTTP reads. #664
Conversation
If a client requested a single byte range which was less than the filesize, then the file handle was never explicitly closed. When the TCP connection was closed, all open file handles on the bridge are automatically closed too. Hence, it seems like this is a moot point - a transient "leak". However, we have observed some clients (particularly: CVMFS) that hold open TCP connections for hours or days, resulting in the "leak" of a file handle per read operation. In some cases, the number of file handles leaked was in the tens-of-thousands. In addition to the obvious memory issues, this was observable in the f-stream monitoring as the near-simultaneous closing of 10k file handles caused UDP buffer overflows and dropped monitoring packets.
|
I have a question on this patch. My understanding is thet the original intent was to keep the file open as long as possible so that subsequent opens would avoid going through the full open path (i.e. the underlying xrootd reuses open file descriptors). Will this ptach violate that intent? If so, then the solution may be even worse for the common case where each HTTP get requires a full file open operations which is very expensive. |
|
So, there's three distinct cases:
So, in the end, we're making things behave consistently (in addition to again being able to monitor via the f-stream). I'm not opposed to doing another layer of file descriptors, but probably should fix this first. |
|
Hi, what you say makes a lot of sense to me, as well as the fixes. I find important the fact that you tested it in production. Thank you for sharing |
|
Thinking again at it I'd say that it's correct that the server defends itself against crappy clients. The definitive solution to this issue IMO is a timeout |
Close file handle for simple HTTP reads.
If a client requested a single byte range which was less than the filesize, then the file handle was never explicitly closed.
When the TCP connection was closed, all open file handles on the bridge are automatically closed too. Hence, it seems like this is a moot point - a transient "leak". However, we have observed some clients (particularly: CVMFS) that hold open TCP connections for hours or days, resulting in the "leak" of a file handle per read operation. In some cases, the number of file handles leaked was in the tens-of-thousands.
In addition to the obvious memory issues, this was observable in the f-stream monitoring as the near-simultaneous closing of 10k file handles caused UDP buffer overflows and dropped monitoring packets.