net/http: HTTP/1 transport connection broken: readLoopPeekFailLocked: <nil>

[jingweno]

Please answer these questions before submitting your issue. Thanks!

1. What version of Go are you using (go version)?

$ go version
go version go1.6 linux/amd64

2. What operating system and processor architecture are you using (go env)?

$ go env
GOARCH="amd64"
GOBIN=""
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/go"
GORACE=""
GOROOT="/usr/local/go"
GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"
GO15VENDOREXPERIMENT="1"
CC="gcc"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0"
CXX="g++"
CGO_ENABLED="1"

3. What did you do?
   If possible, provide a recipe for reproducing the error.
   A complete runnable program is good.
   A link on play.golang.org is best.

Proxying requests using httputil.ReverseProxy with a default http.Transport. A simplified version of the code is here.

4. What did you expect to see?

We expect everything to run fine.

5. What did you see instead?

There were errors net/http: HTTP/1 transport connection broken: readLoopPeekFailLocked: <nil>. The error happened occasionally and seemed to only happen to backend services that responds in gzip. This is the code that emitted the error.

After some investigation, we thought it might be related to keep alive that when all the buffers line up alright but EOF was seen (like from the gzip reader) and the connection was thought to be free again and was put back in the pool. Then when the next request uses that connection it saw the end of the chunked encoding. We disabled keel alive by setting DisableKeepAlives on the transport and the errors hasn't happened for the past 24 hours. We also dug around and thought it might be related to 18072ad (but don't quote us on that).

Let us know if you need more info.

/cc @dpiddy @spofdamon

[danp]

@jingweno could you please also include a sample of the "Unsolicited response received on idle HTTP channel" lines? I believe they said the data was 0\r\n\r\n.

[jingweno]

@dpiddy Good call. The error was Unsolicited response received on idle HTTP channel starting with "0\r\n\r\n"; err=<nil>. It's from here.

Also it would be nice that http.Transport allows taking a log.Logger (like httputil.ReverseProxy) so that we can use our custom logger and don't easily miss any error emitted on transport (we log errors in certain format for better query).

[bradfitz]

Does it happen at tip?

[jingweno]

@bradfitz We'd have to try tip on our staging. Will report result back.

[jingweno]

@bradfitz Just make sure I understand your terms, does tip mean master? Or does that mean particular branches?

[bradfitz]

Yes, master.

[bradfitz]

Any update?

[jingweno]

@bradfitz Unfortunately, we weren't able to reproduce the issue on our staging environment...we may have to wait for 1.7 and then turn on keep alive in prod and see if it fixes it...

Any idea on top of you mind that might have caused this?

[bradfitz]

Pretty sure this is a dup of #15446 which has more info.