net/http, x/net/http2: client loses the request body when retrying after HTTP2 server close connection gracefully(GOAWAY) #39086
Comments
cc @bradfitz Need your help, I had already debug with many days but did not found the root cause. |
/cc |
I ran the reproducer locally and reproduced it. |
I haven't found where the issue is but I do know we are reusing a req.Body on the http2 transport side. What I have determined is that we read the data once and then we reuse the body which has no more data. |
I found http2 transport entrypoint get the request which has no data, and base that I think we are reusing a req.Body on http transport but not http2 transport. I test with adding the following codes in http2 transport entrypoint and got panic: // RoundTripOpt is like RoundTrip, but takes options.
func (t *Transport) RoundTripOpt(req *http.Request, opt RoundTripOpt) (*http.Response, error) {
if !(req.URL.Scheme == "https" || (req.URL.Scheme == "http" && t.AllowHTTP)) {
return nil, errors.New("http2: unsupported scheme")
}
// check request body has data, panic if it's no data
if req.Body != nil && req.GetBody != nil {
body, err := ioutil.ReadAll(req.Body)
if err != nil {
return nil, err
}
if len(body) == 0 {
panic("http2 roundtrip get a requst which body has no data")
}
var ierr error
req.Body, ierr = req.GetBody()
if ierr != nil {
return nil, ierr
}
} |
I found the issue and I have a solution but I need to better understand the flow. |
Took a bit of time but I figured out the problem. Given we never reset the body in the existing request, I am not certain how to solve this other than no returning the ErrSkipAltProtocol if we have already written the body. |
@bradfitz It doesn't look like it. Its a corner case that has always been broken. |
I do have a fix that returns |
I misunderstood that http2 request will never take altRT.RoundTrip before. What about we fix http roundtrip altRT.RoundTrip error process like: if altRT := t.alternateRoundTripper(req); altRT != nil {
if resp, err := altRT.RoundTrip(req); err != ErrSkipAltProtocol {
return resp, err
}
if req.GetBody != nil {
newReq := *req
var ierr error
newReq.Body, ierr = req.GetBody()
if ierr != nil {
return nil, ierr
}
req = &newReq
}
} |
I don't think return an error is acceptable. |
Change https://golang.org/cl/234358 mentions this issue: |
I think I had figured out what happening:
Lines 514 to 518 in bb59a13 And I think #39086 (comment) would be a good solution. |
/cc @bcmills |
Is this a duplicate of #32441? |
@bcmills yes it is. |
Duplicate of #32441 |
What version of Go are you using (
go version
)?Does this issue reproduce with the latest release?
YES, I run the reproduce code in golang docker container with latest image(golang:1.14.2-buster), still occur this bug.
What operating system and processor architecture are you using (
go env
)?go env
OutputWhat did you do?
request declared a Content-Length of N but only wrote 0 bytes
Example codes in Go Playground: https://play.golang.org/p/8eiVRYqRyc6
Already known after I debug
req.GetBody()
in HTTP2 RoundTrip, the bug will not occur again. codes like:go/src/net/http/transport.go
Lines 586 to 596 in 881d540
What did you expect to see?
HTTP client should not got error after received GOAWAY when re-try the request
What did you see instead?
The text was updated successfully, but these errors were encountered: