Skip to content

net/http: occasional nil pointer dereference in (*Client).send #37598

Closed
@noneymous

Description

@noneymous

What version of Go are you using (go version)?

go 1.11 and 1.14

Does this issue reproduce with the latest release?

yes

What operating system and processor architecture are you using (go env)?

go env Output
$ go env
set GO111MODULE=
set GOARCH=amd64
set GOBIN=
set GOCACHE=C:\Users\USER\AppData\Local\go-build
set GOENV=C:\Users\USER\AppData\Roaming\go\env
set GOEXE=.exe
set GOFLAGS=
set GOHOSTARCH=amd64
set GOHOSTOS=windows
set GOINSECURE=
set GONOPROXY=
set GONOSUMDB=
set GOOS=windows
set GOPATH=C:\go
set GOPRIVATE=
set GOPROXY=https://proxy.golang.org,direct
set GOROOT=c:\go
set GOSUMDB=sum.golang.org
set GOTMPDIR=
set GOTOOLDIR=c:\go\pkg\tool\windows_amd64
set GCCGO=gccgo
set AR=ar
set CC=gcc
set CXX=g++
set CGO_ENABLED=1
set GOMOD=
set CGO_CFLAGS=-g -O2
set CGO_CPPFLAGS=
set CGO_CXXFLAGS=-g -O2
set CGO_FFLAGS=-g -O2
set CGO_LDFLAGS=-g -O2
set PKG_CONFIG=pkg-config
set GOGCCFLAGS=-m64 -mthreads -fmessage-length=0 -fdebug-prefix-map=C:\Users\USER\AppData\Local\Temp\go-build702297486=/tmp/go-build -gno-record-gcc-switches

What did you do?

Wrote a web crawler using net/http to crawl large networks with lots of different web services of all ages and technologies. The crawler mostly works great, but ocasionally I'm running into nil pointer dereference panics. I cannot reproduce the issue. Looking at the stacktrace it seems to be burried within net/http and independent of my code.

2020-02-29 18:52:11.810 +0100 	Critical 	Scan scope crashed unexpectedly: runtime error: invalid memory address or nil pointer dereference
Stacktrace:
	| goroutine 4239417 [running]:
	| runtime/debug.Stack(0xa1f, 0xc001186d00, 0x1d)
	| 	C:/Go/src/runtime/debug/stack.go:24 +0xa4
	| my-project/log.StacktraceIndented(0xbc8784, 0x1, 0x52, 0x0)
	| 	D:/go/src/my-project/log/logger.go:347 +0x3b
	| my-project/scans/webenum.(*Scanner).Run.func1(0xc000481440, 0xc002e0fdb0)
	| 	D:/go/src/my-project/scans/webenum/webenum.go:175 +0x10f
	| panic(0xb2a700, 0xa88be0)
	| 	C:/Go/src/runtime/panic.go:967 +0x16b
	| net/http.send(0xc000317500, 0xcc32e0, 0xc00107e600, 0xbf8ec712e19c8a1c, 0x67b0b803359d, 0x10b9800, 0xc0005fe868, 0xbf8ec712e19c8a1c, 0x1, 0x0)
	| 	C:/Go/src/net/http/client.go:271 +0x503
	| net/http.(*Client).send(0xc0002dabd0, 0xc000317500, 0xbf8ec712e19c8a1c, 0x67b0b803359d, 0x10b9800, 0xc0005fe868, 0x0, 0x1, 0xc0013f0710)
	| 	C:/Go/src/net/http/client.go:176 +0x101
	| net/http.(*Client).do(0xc0002dabd0, 0xc000317500, 0x0, 0x0, 0x0)
	| 	C:/Go/src/net/http/client.go:699 +0x451
	| net/http.(*Client).Do(...)
	| 	C:/Go/src/net/http/client.go:567
	|......

C:/Go/src/net/http/client.go:271 is where the actual panic araises due to a nil pointer dereference, by accessing .Body on the response of resp, err = rt.RoundTrip(req). Obviously, for some reason, resp is nil, due to some unhanled error. So, execution continues until it ends up with a panic calling resp.Body. Here is a copy of the source code of client.go lines 252-275 for quick reference:

252:	resp, err = rt.RoundTrip(req)
253:	if err != nil {
254:		stopTimer()
255:		if resp != nil {
256:			log.Printf("RoundTripper returned a response & error; ignoring response")
257:		}
258:		if tlsErr, ok := err.(tls.RecordHeaderError); ok {
259:			// If we get a bad TLS record header, check to see if the
260:			// response looks like HTTP and give a more helpful error.
261:			// See golang.org/issue/11111.
262:			if string(tlsErr.RecordHeader[:]) == "HTTP/" {
263:				err = errors.New("http: server gave HTTP response to HTTPS client")
264:			}
265:		}
266:		return nil, didTimeout, err
267:	}
268:	if !deadline.IsZero() {
269:		resp.Body = &cancelTimerBody{
270:			stop:          stopTimer,
271:			rc:            resp.Body,
272:			reqDidTimeout: didTimeout,
273:		}
274:	}
275:	return resp, nil, nil

Maybe some package expert has an idea, why rt.RoundTrip(req) might return nil without error? Obviously, this edge case should not happen, because resp is required down the road...

What did you expect to see?

I'm expecting anything but panics. Either an error or a normal response.

What did you see instead?

A panic

Metadata

Metadata

Assignees

No one assigned

    Labels

    FrozenDueToAgeNeedsInvestigationSomeone must examine and confirm this is a valid issue and not a duplicate of an existing one.

    Type

    No type

    Projects

    No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions