Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

net/http: Post http://example.com: net/http: HTTP/1 transport connection broken: readLoopPeekFailLocked: EOF #15446

Closed
erikdubbelboer opened this issue Apr 26, 2016 · 5 comments

Comments

Projects
None yet
4 participants
@erikdubbelboer
Copy link
Contributor

commented Apr 26, 2016

Please answer these questions before submitting your issue. Thanks!

  1. What version of Go are you using (go version)?
go version devel +d78c84c Mon Apr 25 23:22:56 2016 +0000 linux/amd64
  1. What operating system and processor architecture are you using (go env)?
GOARCH="amd64"
GOBIN=""
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/home/erik"
GORACE=""
GOROOT="/home/erik/src/github.com/golang/go"
GOTOOLDIR="/home/erik/src/github.com/golang/go/pkg/tool/linux_amd64"
CC="gcc"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build769409056=/tmp/go-build -gno-record-gcc-switches"
CXX="g++"
CGO_ENABLED="1"
  1. What did you do?

Send HTTP POST requests to a host that seems to close the connection some specific time after the response. We see the error Post http://example.com: net/http: HTTP/1 transport connection broken: readLoopPeekFailLocked: EOF multiple times per second when doing around 100 requests per second.

It's hard to debug on live traffic and I haven't been able to reproduce this in a clean example.

The error seems to come from roundTrip() which means the request is already in flight. But it also comes from readLoopPeekFailLocked() which is called from readLoop() in the case of no expected responses.

I'm guessing the connection is closed after a request. Because pc.readLoopPeekFailLocked(err) is called after err is wrapped in a beforeRespHeaderError{err} the peekErr == io.EOF in readLoopPeekFailLocked() is false and the error gets assigned to pc.closed while the connection is already in use. After that I'm not sure, the connection must have somehow already been chosen for the next request. Otherwise it wouldn't make sense that pc.numExpectedResponses == 0 while we are inside the loop to wait for a response.

Is there any debugging (logging things) I can do on the live traffic to help find this bug?

@bradfitz bradfitz self-assigned this Apr 28, 2016

@bradfitz bradfitz added this to the Go1.7 milestone Apr 28, 2016

@bradfitz

This comment has been minimized.

Copy link
Member

commented Apr 28, 2016

Thanks for the report. I think this should be fixable with the information provided.

@jingweno

This comment has been minimized.

Copy link

commented May 16, 2016

@bradfitz Is this fixed on master?

@bradfitz

This comment has been minimized.

Copy link
Member

commented May 16, 2016

Not yet. This bug is still open. Commits will be referenced from here.

@gopherbot

This comment has been minimized.

Copy link

commented May 17, 2016

CL https://golang.org/cl/23160 mentions this issue.

@gopherbot

This comment has been minimized.

Copy link

commented Jul 23, 2016

CL https://golang.org/cl/25153 mentions this issue.

gopherbot pushed a commit that referenced this issue Jul 26, 2016

net/http: make Transport.RoundTrip return raw Conn.Read error on peek…
… failure

From at least Go 1.4 to Go 1.6, Transport.RoundTrip would return the
error value from net.Conn.Read directly when the initial Read (1 byte
Peek) failed while reading the HTTP response, if a request was
outstanding. While never a documented or tested promise, Go 1.7 changed the
behavior (starting at https://golang.org/cl/23160).

This restores the old behavior and adds a test (but no documentation
promises yet) while keeping the fix for spammy logging reported in #15446.

This looks larger than it is: it just changes errServerClosedConn from
a variable to a type, where the type preserves the underlying
net.Conn.Read error, for unwrapping later in Transport.RoundTrip.

Fixes #16465

Change-Id: I6fa018991221e93c0cfe3e4129cb168fbd98bd27
Reviewed-on: https://go-review.googlesource.com/25153
Reviewed-by: Andrew Gerrand <adg@golang.org>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>

@golang golang locked and limited conversation to collaborators Jul 23, 2017

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
You can’t perform that action at this time.