Skip to content

net/http: document that it's illegal to reuse a Request concurrently from multiple goroutines #21780

Closed
@twmb

Description

@twmb

Incorrect errRequestCanceled errors occur when reusing http.Request structs.

There is a small race in reusing Transport's reqCanceler map, where a finishing request can override an active request dial's canceler. I suspect this is what #19653 was running into, and Brad hinted at this.

The problem:

The code centers around transport.go's RoundTrip block of code here, L400-L413.

For two requests a and b that execute back to back:
a sees a body's EOF: L1622-L1624,
b begins, enters RoundTrip, tries to getConn, no idle conns exist, b sets the request's canceler (L940, important), and a large select happens: L948-L986
a's persistConn running gets notified of the bodyEOF, sets its transport's request canceler for this request to nil (L1656) overriding the canceler that was just set by B, and goes to tryPutIdleConn (assuming other conditions are successful): L1655-L1661
a''s tryPutIdleConn sees a waitingDialer: L704-L706
b's dial select receives the persistConn that just ran a: L976-L986
b's RoundTrip getConn returns successfully and goes into pconn.roundTrip (L400-L413, the main block)
roundTrip tries to replace the transports request canceler for b with the persistConn's cancelRequest: L1889-L1891
replaceReqCancel fails because the prior replacement deleted the request from the transport's reqCanceler map, and if the request does not exist in the map, the canceler for it cannot be replaced: L856-L865
This causes roundTrip to return errRequestCanceled: L1893

Also note that this is only the http1 stack and is harder to notice against URLs that have redirects.

What version of Go are you using (go version)?

1.8.3, 1.9, tip

Does this issue reproduce with the latest release?

yes

What operating system and processor architecture are you using (go env)?

GOARCH="amd64"
GOBIN=""
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/home/twmb/go"
GORACE=""
GOROOT="/home/twmb/go/go"
GOTOOLDIR="/home/twmb/go/go/pkg/tool/linux_amd64"
GCCGO="gccgo"
CC="gcc"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build465265467=/tmp/go-build -gno-record-gcc-switches"
CXX="g++"
CGO_ENABLED="1"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"

What did you do?

https://play.golang.org/p/BP7YCP2D4J
go test -bench . -benchtime 10s (should happen quickly)

What did you expect to see?

No panic.

What did you see instead?

A panic.

Metadata

Metadata

Assignees

No one assigned

    Labels

    DocumentationIssues describing a change to documentation.FrozenDueToAgeNeedsFixThe path to resolution is known, but the work has not been done.

    Type

    No type

    Projects

    No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions