Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

net/http: client.Do(request) panics in time.stopTimer #38219

Open
hamid6 opened this issue Apr 2, 2020 · 5 comments
Open

net/http: client.Do(request) panics in time.stopTimer #38219

hamid6 opened this issue Apr 2, 2020 · 5 comments
Milestone

Comments

@hamid6
Copy link

@hamid6 hamid6 commented Apr 2, 2020

What version of Go are you using (go version)?

$ go version
go version go1.14.1 linux/amd64

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

go env Output
$ go env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/root/.cache/go-build"
GOENV="/root/.config/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/local/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"
GCCGO="gccgo"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build562962017=/tmp/go-build -gno-record-gcc-switches"

What did you do?

Unfortunately it's not possible to post the code here but it should be pretty straightforward to reproduce this.

My application makes concurrent GET requests to another API at around 280 req/sec using goroutines. This application makes about 3 million requests to the API every day which takes about 4-5 hours to finish, however when using the latest version of Go it panics after 10-120 minutes of running (the length of run before panic is not deterministic). Rolling back to Go 1.13.5 fixes this issue.

I have seen other threads here that refer to the same issue in older versions of Go, however this seems to have re-appeared in 1.14.1 again. I'm not sure if it's for a different reason this time.

What did you expect to see?

200 OK

What did you see instead?

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x451cad]

goroutine 646692 [running]:
time.stopTimer(0xc00035fa48, 0x0)
	/usr/local/go/src/runtime/time.go:224 +0x2b
time.(*Timer).Stop(...)
	/usr/local/go/src/time/sleep.go:78
net/http.(*http2ClientConn).roundTrip(0xc001e68780, 0xc001e8c900, 0x0, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/http/h2_bundle.go:7480 +0x9f
net/http.(*http2Transport).RoundTripOpt(0xc0001965a0, 0xc001e8c900, 0x9b3800, 0xc0001af080, 0xc000809770, 0x5)
	/usr/local/go/src/net/http/h2_bundle.go:6948 +0x16f
net/http.(*http2Transport).RoundTrip(...)
	/usr/local/go/src/net/http/h2_bundle.go:6909
net/http.http2noDialH2RoundTripper.RoundTrip(0xc0001965a0, 0xc001e8c900, 0xb53460, 0xc0001965a0, 0x0)
	/usr/local/go/src/net/http/h2_bundle.go:9103 +0x3e
net/http.(*Transport).roundTrip(0xf4e580, 0xc001e8c900, 0xc002c91e60, 0xc001080840, 0x40c7f8)
	/usr/local/go/src/net/http/transport.go:515 +0xd94
net/http.(*Transport).RoundTrip(0xf4e580, 0xc001e8c900, 0xf4e580, 0x0, 0x0)
	/usr/local/go/src/net/http/roundtrip.go:17 +0x35
net/http.send(0xc001e8c900, 0xb52420, 0xf4e580, 0x0, 0x0, 0x0, 0xc00073c2f8, 0xc00002a015, 0x1, 0x0)
	/usr/local/go/src/net/http/client.go:252 +0x43e
net/http.(*Client).send(0xf58ba0, 0xc001e8c900, 0x0, 0x0, 0x0, 0xc00073c2f8, 0x0, 0x1, 0xc001e8c900)
	/usr/local/go/src/net/http/client.go:176 +0xfa
net/http.(*Client).do(0xf58ba0, 0xc001e8c900, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/http/client.go:699 +0x44a
net/http.(*Client).Do(...)
	/usr/local/go/src/net/http/client.go:567
@andybons andybons changed the title http client.Do(request) panics in goroutines net/http: client.Do(request) panics in goroutines Apr 2, 2020
@andybons
Copy link
Member

@andybons andybons commented Apr 2, 2020

I have seen other threads here that refer to the same issue in older versions of Go, however this seems to have re-appeared in 1.14.1 again. I'm not sure if it's for a different reason this time.

Can you point to these threads/issues?

@andybons
Copy link
Member

@andybons andybons commented Apr 2, 2020

@andybons andybons added this to the Unplanned milestone Apr 2, 2020
@bcmills
Copy link
Member

@bcmills bcmills commented Apr 2, 2020

This sounds like #37400 (specifically, the symptom reported in #37490), but the fix for that (by @ianlancetaylor) was released in Go 1.14.1 (#37494).

@bcmills bcmills modified the milestones: Unplanned, Backlog Apr 2, 2020
@bcmills bcmills changed the title net/http: client.Do(request) panics in goroutines net/http: client.Do(request) panics in time.stopTimer Apr 2, 2020
@bcmills bcmills modified the milestones: Backlog, Go1.15 Apr 2, 2020
@bcmills
Copy link
Member

@bcmills bcmills commented Apr 2, 2020

Milestoning to 1.15 based on similarlity to #37400.

@OneOfOne
Copy link
Contributor

@OneOfOne OneOfOne commented Apr 3, 2020

IMO this should be 1.14.2, this is a massive regression from 1.13.x, I ran into the same issue with the exact backtrace.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
4 participants
You can’t perform that action at this time.