Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

net/http: HTTP/1 transport connection broken: readLoopPeekFailLocked: <nil> #15253

Closed
jingweno opened this issue Apr 12, 2016 · 9 comments

Comments

Projects
None yet
4 participants
@jingweno
Copy link

commented Apr 12, 2016

Please answer these questions before submitting your issue. Thanks!

  1. What version of Go are you using (go version)?
$ go version
go version go1.6 linux/amd64
  1. What operating system and processor architecture are you using (go env)?
$ go env
GOARCH="amd64"
GOBIN=""
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/go"
GORACE=""
GOROOT="/usr/local/go"
GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"
GO15VENDOREXPERIMENT="1"
CC="gcc"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0"
CXX="g++"
CGO_ENABLED="1"
  1. What did you do?
    If possible, provide a recipe for reproducing the error.
    A complete runnable program is good.
    A link on play.golang.org is best.

Proxying requests using httputil.ReverseProxy with a default http.Transport. A simplified version of the code is here.

  1. What did you expect to see?

We expect everything to run fine.

  1. What did you see instead?

There were errors net/http: HTTP/1 transport connection broken: readLoopPeekFailLocked: <nil>. The error happened occasionally and seemed to only happen to backend services that responds in gzip. This is the code that emitted the error.

After some investigation, we thought it might be related to keep alive that when all the buffers line up alright but EOF was seen (like from the gzip reader) and the connection was thought to be free again and was put back in the pool. Then when the next request uses that connection it saw the end of the chunked encoding. We disabled keel alive by setting DisableKeepAlives on the transport and the errors hasn't happened for the past 24 hours. We also dug around and thought it might be related to 18072ad (but don't quote us on that).

Let us know if you need more info.

/cc @dpiddy @spofdamon

@danp

This comment has been minimized.

Copy link
Contributor

commented Apr 12, 2016

@jingweno could you please also include a sample of the "Unsolicited response received on idle HTTP channel" lines? I believe they said the data was 0\r\n\r\n.

@jingweno

This comment has been minimized.

Copy link
Author

commented Apr 12, 2016

@dpiddy Good call. The error was Unsolicited response received on idle HTTP channel starting with "0\r\n\r\n"; err=<nil>. It's from here.

Also it would be nice that http.Transport allows taking a log.Logger (like httputil.ReverseProxy) so that we can use our custom logger and don't easily miss any error emitted on transport (we log errors in certain format for better query).

@bradfitz

This comment has been minimized.

Copy link
Member

commented Apr 12, 2016

Does it happen at tip?

@bradfitz bradfitz added this to the Unplanned milestone Apr 12, 2016

@jingweno

This comment has been minimized.

Copy link
Author

commented Apr 12, 2016

@bradfitz We'd have to try tip on our staging. Will report result back.

@jingweno

This comment has been minimized.

Copy link
Author

commented Apr 13, 2016

@bradfitz Just make sure I understand your terms, does tip mean master? Or does that mean particular branches?

@bradfitz

This comment has been minimized.

Copy link
Member

commented Apr 13, 2016

Yes, master.

@bradfitz

This comment has been minimized.

Copy link
Member

commented May 10, 2016

Any update?

@jingweno

This comment has been minimized.

Copy link
Author

commented May 13, 2016

@bradfitz Unfortunately, we weren't able to reproduce the issue on our staging environment...we may have to wait for 1.7 and then turn on keep alive in prod and see if it fixes it...

Any idea on top of you mind that might have caused this?

@bradfitz

This comment has been minimized.

Copy link
Member

commented May 13, 2016

Pretty sure this is a dup of #15446 which has more info.

@bradfitz bradfitz closed this May 13, 2016

@golang golang locked and limited conversation to collaborators May 13, 2017

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
You can’t perform that action at this time.