Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

net/http: concurrent HTTP2 client requests to a single nginx server returns GOAWAY NO_ERROR after a while #18112

Closed
fln opened this issue Nov 30, 2016 · 3 comments

Comments

Projects
None yet
4 participants
@fln
Copy link

commented Nov 30, 2016

What version of Go are you using (go version)?

go version go1.7.3 linux/amd64

What operating system and processor architecture are you using (go env)?

GOARCH="amd64"
GOBIN=""
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/home/folnin/go"
GORACE=""
GOROOT="/usr/local/go"
GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"
CC="gcc"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build294525165=/tmp/go-build -gno-record-gcc-switches"
CXX="g++"
CGO_ENABLED="1"

What did you do?

Have multiple goroutines accessing single nginx server over HTTP2.

Default nginx installation will serve up to 1999 streams (1000 requests) over single HTTP2 TCP connection before returning GOAWAY and closing the connection. It is controlled by http2_max_requests directive. If multiple goroutines are making requests when server issues GOAWAY, reconnect is handled ungracefully and some goroutines receive error instead of requested content. If http client is used by single goroutine synchronously such errors never occur.

Example code:

https://play.golang.org/p/mPaBxuuRmm

Please note that this code will not work inside playground because of network access restrictions. It will work if executed locally. Example contains publicly accessible server that is operated by me. The error can be reproduced by connecting to nginx.com CDN as well (URL is commented out in the example) however it has much higher http2_max_requests configured and one have to wait much longer for the error to occur (~15k connections last time I tired).

What did you expect to see?

Application looping forever without errors (except network connectivity issues or server side errors).

What did you see instead?

After ~1000 requests server tries to close TCP connection with GOAWAY, some requests are terminated with an error:

http2: server sent GOAWAY and closed the connection; LastStreamID=1999, ErrCode=NO_ERROR, debug=""

If only one goroutine is making requests this error never occurs.

@bradfitz

This comment has been minimized.

Copy link
Member

commented Nov 30, 2016

@tombergan

This comment has been minimized.

Copy link
Contributor

commented Nov 30, 2016

Thanks for the detailed repro. This sounds like the same problem hypothesized in #18083: we don't gracefully handle GOAWAY(NO_ERROR) if the GOAWAY is received at certain points during the RoundTrip call.

@bradfitz

This comment has been minimized.

Copy link
Member

commented Dec 6, 2016

This should now be fixed by golang/net@8dab929 which will be in Go 1.8.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
You can’t perform that action at this time.