net/http: client.Do(request) panics in time.stopTimer

[codekhol]

What version of Go are you using (go version)?

$ go version
go version go1.14.1 linux/amd64

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

go env Output

$ go env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/root/.cache/go-build"
GOENV="/root/.config/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/local/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"
GCCGO="gccgo"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build562962017=/tmp/go-build -gno-record-gcc-switches"

What did you do?

Unfortunately it's not possible to post the code here but it should be pretty straightforward to reproduce this.

My application makes concurrent GET requests to another API at around 280 req/sec using goroutines. This application makes about 3 million requests to the API every day which takes about 4-5 hours to finish, however when using the latest version of Go it panics after 10-120 minutes of running (the length of run before panic is not deterministic). Rolling back to Go 1.13.5 fixes this issue.

I have seen other threads here that refer to the same issue in older versions of Go, however this seems to have re-appeared in 1.14.1 again. I'm not sure if it's for a different reason this time.

What did you expect to see?

200 OK

What did you see instead?

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x451cad]

goroutine 646692 [running]:
time.stopTimer(0xc00035fa48, 0x0)
	/usr/local/go/src/runtime/time.go:224 +0x2b
time.(*Timer).Stop(...)
	/usr/local/go/src/time/sleep.go:78
net/http.(*http2ClientConn).roundTrip(0xc001e68780, 0xc001e8c900, 0x0, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/http/h2_bundle.go:7480 +0x9f
net/http.(*http2Transport).RoundTripOpt(0xc0001965a0, 0xc001e8c900, 0x9b3800, 0xc0001af080, 0xc000809770, 0x5)
	/usr/local/go/src/net/http/h2_bundle.go:6948 +0x16f
net/http.(*http2Transport).RoundTrip(...)
	/usr/local/go/src/net/http/h2_bundle.go:6909
net/http.http2noDialH2RoundTripper.RoundTrip(0xc0001965a0, 0xc001e8c900, 0xb53460, 0xc0001965a0, 0x0)
	/usr/local/go/src/net/http/h2_bundle.go:9103 +0x3e
net/http.(*Transport).roundTrip(0xf4e580, 0xc001e8c900, 0xc002c91e60, 0xc001080840, 0x40c7f8)
	/usr/local/go/src/net/http/transport.go:515 +0xd94
net/http.(*Transport).RoundTrip(0xf4e580, 0xc001e8c900, 0xf4e580, 0x0, 0x0)
	/usr/local/go/src/net/http/roundtrip.go:17 +0x35
net/http.send(0xc001e8c900, 0xb52420, 0xf4e580, 0x0, 0x0, 0x0, 0xc00073c2f8, 0xc00002a015, 0x1, 0x0)
	/usr/local/go/src/net/http/client.go:252 +0x43e
net/http.(*Client).send(0xf58ba0, 0xc001e8c900, 0x0, 0x0, 0x0, 0xc00073c2f8, 0x0, 0x1, 0xc001e8c900)
	/usr/local/go/src/net/http/client.go:176 +0xfa
net/http.(*Client).do(0xf58ba0, 0xc001e8c900, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/http/client.go:699 +0x44a
net/http.(*Client).Do(...)
	/usr/local/go/src/net/http/client.go:567

[andybons]

I have seen other threads here that refer to the same issue in older versions of Go, however this seems to have re-appeared in 1.14.1 again. I'm not sure if it's for a different reason this time.

Can you point to these threads/issues?

[andybons]

@bradfitz @bcmills @rsc

[bcmills]

This sounds like #37400 (specifically, the symptom reported in #37490), but the fix for that (by @ianlancetaylor) was released in Go 1.14.1 (#37494).

[bcmills]

Milestoning to 1.15 based on similarlity to #37400.

[OneOfOne]

IMO this should be 1.14.2, this is a massive regression from 1.13.x, I ran into the same issue with the exact backtrace.

[ianlancetaylor]

I can tell from the line numbers that the stack backtrace in the original report is from 1.14, not 1.14.1, so I'm fairly certain that this was fixed by the fix for #37494.

@OneOfOne Do you still see this when using the 1.14.4 release? Can you show us the backtrace that you get? Thanks.

[gopherbot]

Timed out in state WaitingForInfo. Closing.

(I am just a bot, though. Please speak up if this is a mistake or you have the requested information.)