Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

net/http: Request body is not closed by Transport.RoundTrip on some net.Conn errors #49621

Open
egorbunov opened this issue Nov 16, 2021 · 3 comments

Comments

@egorbunov
Copy link

@egorbunov egorbunov commented Nov 16, 2021

What version of Go are you using (go version)?

$ go version
go version go1.17.3 linux/amd64

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

go env Output
$ go env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/eg/.cache/go-build"
GOENV="/home/eg/.config/go/env"
GOEXE=""
GOEXPERIMENT=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOINSECURE=""
GOMODCACHE="/home/eg/go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/home/eg/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/lib/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/lib/go/pkg/tool/linux_amd64"
GOVCS=""
GOVERSION="go1.17.3"
GCCGO="gccgo"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/usr/lib/go/src/go.mod"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build2074510398=/tmp/go-build -gno-record-gcc-switches"

What did you do?

  1. Wrap net.Listener with error injections
  2. Run http.Server using such listener
  3. Run many concurrent requests targeting such server
  4. On every request, at the client side, wait for request body to be closed

Kind of a minimal example to run described steps: https://play.golang.com/p/lku8lEgiPu6
Yeah, I know that reproducer feels big, but the most of it is to create error injections actually, but
main part (see main fn) is straightforward and simple.

What did you expect to see?

I expect that net/http eventually closes Request.Body as it is told in documentation for RoundTripper: https://pkg.go.dev/net/http#RoundTripper

What did you see instead?

Specified example program hangs indefinitely while waiting for Request.Body to be closed.

Investigation

  • It is possible that encountered behavior has something to deal with errServerClosedIdle error inside net/http, because we have met described hangs only after receiving that error from Client.Do.
  • Also the fact that I am doing Post request may be important because net/http does not retry Post (non-idepotent) request on such error.

In case next assumption is true:

Once request execution got to func (pc *persistConn) roundTrip(req *transportRequest) call, Request.Body closure is expected to be done by func (pc *persistConn) writeLoop().

Then it seems like there is a possibility of not closing body in the writeLoop in case pc is closed concurrently.
Imagine:

                   ---------------------------------------------------------------------------------------> time
gor1               pc.writech <- writeRequest{...}                                   
gor2                   pc.closeWithError(...)
writeLoop                                                              select { case <-pc.closech: return }
                                                                                              

So request was sent successfully into pc.writech (channel has buffer of size 1), but won't be ever read from it because writeLoop exited.

Related places in the code:

  1. func (pc *persistConn) writeLoop() {
  2. pc.writech <- writeRequest{req, writeErrCh, continueCh}
@egorbunov egorbunov changed the title Request body is not closed by Transport.RoundTrip upon underlying net.Conn errors http.Request body is not closed by Transport.RoundTrip upon underlying net.Conn errors Nov 16, 2021
@egorbunov egorbunov changed the title http.Request body is not closed by Transport.RoundTrip upon underlying net.Conn errors http.Request body is not closed by Transport.RoundTrip on some net.Conn errors Nov 16, 2021
@egorbunov egorbunov changed the title http.Request body is not closed by Transport.RoundTrip on some net.Conn errors net/http: Request body is not closed by Transport.RoundTrip on some net.Conn errors Nov 16, 2021
@Asuan
Copy link
Contributor

@Asuan Asuan commented Nov 17, 2021

There are workaround:
add line rw.Header().Add("Content-Length", "1") or rw.Write([]byte("ok")) after line 113 it solve problem.
Looks like problem only in case no body in response from server.

Loading

@heschi heschi added this to the Go1.18 milestone Nov 22, 2021
@heschi
Copy link
Contributor

@heschi heschi commented Nov 22, 2021

cc @neild

Loading

@egorbunov
Copy link
Author

@egorbunov egorbunov commented Nov 22, 2021

There are workaround: add line rw.Header().Add("Content-Length", "1") or rw.Write([]byte("ok")) after line 113 it solve problem. Looks like problem only in case no body in response from server.

Not really because adding correct Content-Length with actual response body of Content-Length bytes does not solve anything and reproducer continues to hang.

Loading

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
3 participants