net/http: receiving DATA after sending (RST_STREAM) kills connection. #16974
Please answer these questions before submitting your issue. Thanks!
What version of Go are you using (
The text was updated successfully, but these errors were encountered:
Sorry. I believe this is still a thing but can be fixed by discarding the response body before the close as to allow the transport to complete the state machine. This isn't how h1 behaves though. I assumed closing the body before reading would automatically discard the body.
Correct, but it doesn't always do that. We've seen the race in production and were able to reproduce after long runs of our server under load locally to get the debug log output that I've posted above and annotated below.
The client closes the Response.Body BEFORE receiving DATA frames from the server. This sends a RST_STREAM to the server and then closes the bodies pipe with the non-nil error
Next the DATA frame for the now closed stream comes in:
The client sees the stream in the list of open streams:
and then attempts to write to the closed pipe:
which returns an err:
produce debug logs:
then return the err:
and close the conn:
Here is the servers log for ref:
Does that help? I was having trouble getting a test together to isolate at the time but was able to reproduce with private code if I didn't drain the body before closing which keeps the pipe in a non-errored state to avoid the path above.
I hope this clears things up.
Are you sure you're running Go 1.7 or tip?
I wrote a test for this, but it doesn't occur like you describe. You wrote:
That line for reference is:
... which sets the pipe's
breakErr error // immediate read error (caller doesn't see rest of b)
It does not set its
Future Writes occur fine.
There's likely still a problem here (we should note that we gave up on that stream and not even try to write), but the problem isn't a you describe, and I still can't make the connection die as you've said.
What version are you running?
Even at golang/net@6c89f96 the code did https://github.com/golang/net/blob/6c89f9617983ee917132513a791d8b5850fe90c5/http2/transport.go#L1585 (BreakWithError) instead of CloseWithError, so the whole code path you described doesn't follow, since that first step you cite doesn't set the sticky error you later cite as causing as a Write error.
What is causing the Write error you see?
We still don't have a repro here.
…ncels If the client canceled a request, the Transport would then send a RST_STREAM, but also would close the Response.Body's pipe. Meanwhile, the server's DATA response could already be on the wire in flight. We'd then read it, attempt to write its bytes to the buffer, find it already closed, bubble up that error, and ultimately close the whole TCP connection (breaking all other open streams). So, don't do that. Keep track of which connections we've sent RST_STREAM to, and ignore DATA frames on those streams. Updates golang/go#16974 (fixes after bundle to std) Change-Id: Ic29a3aefff5241146cd2ca80aafa35fc4fb18b6e Reviewed-on: https://go-review.googlesource.com/32571 Reviewed-by: Tom Bergan <email@example.com> Reviewed-by: Brad Fitzpatrick <firstname.lastname@example.org> Run-TryBot: Brad Fitzpatrick <email@example.com>