Join GitHub today
GitHub is home to over 31 million developers working together to host and review code, manage projects, and build software together.Sign up
x/net/http2: client can hang forever if headers' size exceeds connection's buffer size and server hangs past request time #23559
What version of Go are you using (
Working on a patch to fix this, I realized that using the context deadline to timeout the i/o calls may not be appropriate.
Any update here?
Keep in mind that there are 3 goroutines per Transport connection to a server:
I think the behavior we want is:
There are cases where this isn't working. I believe the graph and tests demonstrates it.
Unfortunately, the writes almost never time out I don't know if this is specific to my use particular use case or if buffering makes it very long to happen.
I run this CL in production for a very specific service that only makes requests to APNS and Firebase push gateways.
The patch helps to recover from stuck connections and kills it but it's slow to converge, is incomplete and may not work properly for the general use case.
What happens is that there are situations where the APNS gateway is blocking the connection, it behaves like a firewall that drops packet never reset the connection. Most of the time the requests are timing out correctly, the context allow to unlock the client stream goroutine and return to the caller however the connection read loop never times out nor receive any other error causing all requests to time out. (Also not that the APNS gateway doesn't allow pings).
What I see most often is the writes are not timing out. I believe that the only way to guarantee to time out the connection is have a way to unblock the read loop. The reads (ClientConn) are asynchronous to the writes (ClientStream) so we have to find a way to add a deadline on the reads that will follow the writes (requests) or that follow a read (e.g reading the rest of the body) from the client stream side.
The CL takes a simple approach and avoids synchronization between the client stream and the connection read loop. It resets the read deadline and sets a write deadline before every write from the client stream. After the write is done, it sets the read deadline.
I wanted to experiment the following algorithm:
Upon waiting for something from the network on the ClientStream / caller side, push a deadline in a heap data structure (probably under ClientConn.wmu lock) keeping track of the stream that pushes it.
What do you think ?