Skip to content

net/http: fatal error: concurrent map iteration and map write transport.go #58494

New issue
New issue
Closed
Closed
net/http: fatal error: concurrent map iteration and map write transport.go#58494
Labels
FrozenDueToAgeWaitingForInfoIssue is not actionable because of missing required information, which needs to be provided.
@johnbalvin

Description

@johnbalvin
johnbalvin
opened on Feb 13, 2023

What version of Go are you using (go version)?

$ 1.20

Does this issue reproduce with the latest release?

yes

What operating system and processor architecture are you using (go env)?

set GO111MODULE= set GOARCH=amd64 set GOBIN= set GOCACHE=C:\Users\user\AppData\Local\go-build set GOENV=C:\Users\user\AppData\Roaming\go\env set GOEXE=.exe set GOEXPERIMENT= set GOFLAGS= set GOHOSTARCH=amd64 set GOHOSTOS=windows set GOINSECURE= set GOMODCACHE=C:\Users\user\go\pkg\mod set GONOPROXY= set GONOSUMDB= set GOOS=windows set GOPATH=C:\Users\user\go set GOPRIVATE= set GOPROXY=https://proxy.golang.org,direct set GOROOT=C:\Program Files\Go set GOSUMDB=sum.golang.org set GOTMPDIR= set GOTOOLDIR=C:\Program Files\Go\pkg\tool\windows_amd64 set GOVCS= set GOVERSION=go1.20 set GCCGO=gccgo set GOAMD64=v1 set AR=ar set CC=gcc set CXX=g++ set CGO_ENABLED=0 set GOMOD=D:\upwork\andrew\lawplus\go.mod set GOWORK= set CGO_CFLAGS=-O2 -g set CGO_CPPFLAGS= set CGO_CXXFLAGS=-O2 -g set CGO_FFLAGS=-O2 -g set CGO_LDFLAGS=-O2 -g set PKG_CONFIG=pkg-config set GOGCCFLAGS=-m64 -fno-caret-diagnostics -Qunused-arguments -Wl,--no-gc-sections -fmessage-length=0 -fdebug-prefix-map=C:\Users\user\AppData\Local\Temp\go-build3669119433=/tmp/go-build -gno-record-gcc-switches Output
$ go env

What did you do?

Making multiple requests at the same time.
the transport information I'm using is:

transport := &http.Transport{
		MaxIdleConnsPerHost: 30,
		DisableKeepAlives:   true,
		TLSClientConfig: &tls.Config{
			Renegotiation:      tls.RenegotiateOnceAsClient,
			InsecureSkipVerify: true,
		},
	}

Not a reproducable error because the code is big and it happens on random times

What did you expect to see?

Code runnign well

What did you see instead?

fatal error: concurrent map iteration and map write

goroutine 941361 [running]:
net/http.Header.sortedKeyValues(0xc003d58780, 0x1b4bb20?)
        /usr/local/go/src/net/http/header.go:179 +0x12a
net/http.Header.writeSubset(0xc013a1a960?, {0x1b4bb20, 0xc00fa89f00}, 0x0?, 0x0)
        /usr/local/go/src/net/http/header.go:201 +0xd1
net/http.(*Request).write(0xc01180e300, {0x1b4bb20, 0xc00fa89f00}, 0x0, 0xc003b4a300, 0x0)
        /usr/local/go/src/net/http/request.go:658 +0x8d6
net/http.(*persistConn).writeLoop(0xc013706480)
        /usr/local/go/src/net/http/transport.go:2413 +0x171
created by net/http.(*Transport).dialConn
        /usr/local/go/src/net/http/transport.go:1766 +0x173d


goroutine 776511 [select]:
net/http.(*Transport).getConn(0xc006a0b040, 0xc065c66a40, {{}, 0xc000512240, {0x170d77c, 0x5}, {0xc00d9a5940, 0x1f}, 0x0})
        /usr/local/go/src/net/http/transport.go:1382 +0x5c6
net/http.(*Transport).roundTrip(0xc006a0b040, 0xc0085b8100)
        /usr/local/go/src/net/http/transport.go:590 +0x79e
net/http.(*Transport).RoundTrip(0xc0085b8100?, 0x1b51260?)
        /usr/local/go/src/net/http/roundtrip.go:17 +0x19
net/http.send(0xc0085b8000, {0x1b51260, 0xc006a0b040}, {0x8?, 0x1676500?, 0x24b21c0?})
        /usr/local/go/src/net/http/client.go:252 +0x5f7
net/http.(*Client).send(0xc0035c8c00, 0xc0085b8000, {0x0?, 0xf?, 0x24b21c0?})
        /usr/local/go/src/net/http/client.go:176 +0x9b
net/http.(*Client).do(0xc0035c8c00, 0xc0085b8000)
        /usr/local/go/src/net/http/client.go:716 +0x8fb
net/http.(*Client).Do(...)
        /usr/local/go/src/net/http/client.go:582
lawplus/shared/requests.make(0xc0085b8000, 0x1, 0x0, 0x4a817c800)
        /workspace/shared/requests/request.go:120 +0x665
lawplus/shared/requests.Get(0xc0085b8000, 0x1, 0x170d77c?, 0x39?)
        /workspace/shared/requests/request.go:27 +0xe6

goroutine 941344 [IO wait]:
internal/poll.runtime_pollWait(0x7f20965b6f28, 0x72)
        /usr/local/go/src/runtime/netpoll.go:306 +0x89
internal/poll.(*pollDesc).wait(0xc003b2a500?, 0xc0045b1000?, 0x0)
        /usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x32
internal/poll.(*pollDesc).waitRead(...)
        /usr/local/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc003b2a500, {0xc0045b1000, 0x1000, 0x1000})
        /usr/local/go/src/internal/poll/fd_unix.go:167 +0x299
net.(*netFD).Read(0xc003b2a500, {0xc0045b1000?, 0xc005fc3700?, 0x61?})
        /usr/local/go/src/net/fd_posix.go:55 +0x29
net.(*conn).Read(0xc012799490, {0xc0045b1000?, 0xc005fc3820?, 0x40d8e8?})
        /usr/local/go/src/net/net.go:183 +0x45
crypto/tls.(*atLeastReader).Read(0xc01234d170, {0xc0045b1000?, 0xc01234d170?, 0x0?})
        /usr/local/go/src/crypto/tls/conn.go:788 +0x3d
bytes.(*Buffer).ReadFrom(0xc015345090, {0x1b4da20, 0xc01234d170})
        /usr/local/go/src/bytes/buffer.go:202 +0x98
crypto/tls.(*Conn).readFromUntil(0xc015344e00, {0x1b51180?, 0xc012799490}, 0x0?)
        /usr/local/go/src/crypto/tls/conn.go:810 +0xe5
crypto/tls.(*Conn).readRecordOrCCS(0xc015344e00, 0x0)
        /usr/local/go/src/crypto/tls/conn.go:617 +0x116
crypto/tls.(*Conn).readRecord(...)
        /usr/local/go/src/crypto/tls/conn.go:583
crypto/tls.(*Conn).Read(0xc015344e00, {0xc0045b6000, 0x1000, 0xc005fc3cc8?})
        /usr/local/go/src/crypto/tls/conn.go:1288 +0x16f
net/http.(*persistConn).Read(0xc010f16900, {0xc0045b6000?, 0x405c66?, 0x60?})
        /usr/local/go/src/net/http/transport.go:1943 +0x4e
bufio.(*Reader).fill(0xc008f1ff80)
        /usr/local/go/src/bufio/bufio.go:106 +0xff
bufio.(*Reader).Peek(0xc008f1ff80, 0x1)
        /usr/local/go/src/bufio/bufio.go:144 +0x5d
net/http.(*persistConn).readLoop(0xc010f16900)
        /usr/local/go/src/net/http/transport.go:2107 +0x1ac
created by net/http.(*Transport).dialConn
        /usr/local/go/src/net/http/transport.go:1765 +0x16ea

goroutine 941288 [select]:
net/http.(*persistConn).writeLoop(0xc013b13680)
        /usr/local/go/src/net/http/transport.go:2410 +0xf2
created by net/http.(*Transport).dialConn
        /usr/local/go/src/net/http/transport.go:1766 +0x173d

goroutine 941298 [select]:
net/http.(*persistConn).writeLoop(0xc013b137a0)
        /usr/local/go/src/net/http/transport.go:2410 +0xf2
created by net/http.(*Transport).dialConn
        /usr/local/go/src/net/http/transport.go:1766 +0x173d

goroutine 940845 [IO wait]:
internal/poll.runtime_pollWait(0x7f20965b67a8, 0x72)
        /usr/local/go/src/runtime/netpoll.go:306 +0x89
internal/poll.(*pollDesc).wait(0xc006a57300?, 0xc006b49800?, 0x0)
        /usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x32
internal/poll.(*pollDesc).waitRead(...)
        /usr/local/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc006a57300, {0xc006b49800, 0x1800, 0x1800})
        /usr/local/go/src/internal/poll/fd_unix.go:167 +0x299
net.(*netFD).Read(0xc006a57300, {0xc006b49800?, 0xc004278700?, 0x61?})
        /usr/local/go/src/net/fd_posix.go:55 +0x29
net.(*conn).Read(0xc006851908, {0xc006b49800?, 0xc004278820?, 0x668120?})
        /usr/local/go/src/net/net.go:183 +0x45
crypto/tls.(*atLeastReader).Read(0xc01148b428, {0xc006b49800?, 0xc01148b428?, 0x0?})
        /usr/local/go/src/crypto/tls/conn.go:788 +0x3d
bytes.(*Buffer).ReadFrom(0xc006f0b090, {0x1b4da20, 0xc01148b428})
        /usr/local/go/src/bytes/buffer.go:202 +0x98
crypto/tls.(*Conn).readFromUntil(0xc006f0ae00, {0x1b51180?, 0xc006851908}, 0x0?)
        /usr/local/go/src/crypto/tls/conn.go:810 +0xe5
crypto/tls.(*Conn).readRecordOrCCS(0xc006f0ae00, 0x0)
        /usr/local/go/src/crypto/tls/conn.go:617 +0x116
crypto/tls.(*Conn).readRecord(...)
        /usr/local/go/src/crypto/tls/conn.go:583
crypto/tls.(*Conn).Read(0xc006f0ae00, {0xc002ddc000, 0x1000, 0x1400600?})
        /usr/local/go/src/crypto/tls/conn.go:1288 +0x16f
net/http.(*persistConn).Read(0xc013158120, {0xc002ddc000?, 0x405c66?, 0x60?})
        /usr/local/go/src/net/http/transport.go:1943 +0x4e
bufio.(*Reader).fill(0xc0079c98c0)
        /usr/local/go/src/bufio/bufio.go:106 +0xff
bufio.(*Reader).Peek(0xc0079c98c0, 0x1)
        /usr/local/go/src/bufio/bufio.go:144 +0x5d
net/http.(*persistConn).readLoop(0xc013158120)
        /usr/local/go/src/net/http/transport.go:2107 +0x1ac
created by net/http.(*Transport).dialConn
        /usr/local/go/src/net/http/transport.go:1765 +0x16ea

goroutine 776510 [select]:
net/http.(*persistConn).roundTrip(0xc013b13680, 0xc0561e8000)
        /usr/local/go/src/net/http/transport.go:2638 +0x994
net/http.(*Transport).roundTrip(0xc00606e780, 0xc025ca2e00)
        /usr/local/go/src/net/http/transport.go:603 +0x7fa
net/http.(*Transport).RoundTrip(0xc025ca2e00?, 0x1b51260?)
        /usr/local/go/src/net/http/roundtrip.go:17 +0x19
net/http.send(0xc025ca2d00, {0x1b51260, 0xc00606e780}, {0x8?, 0x1676500?, 0x24b21c0?})
        /usr/local/go/src/net/http/client.go:252 +0x5f7
net/http.(*Client).send(0xc004204fc0, 0xc025ca2d00, {0x4cb77f?, 0x9f876c?, 0x24b21c0?})
        /usr/local/go/src/net/http/client.go:176 +0x9b
net/http.(*Client).do(0xc004204fc0, 0xc025ca2d00)
        /usr/local/go/src/net/http/client.go:716 +0x8fb
net/http.(*Client).Do(...)
        /usr/local/go/src/net/http/client.go:582
lawplus/shared/requests.make(0xc025ca2d00, 0x0, 0x0, 0xdf8475800)
        /workspace/shared/requests/request.go:120 +0x665

goroutine 776506 [runnable]:
net/http.Header.Clone(...)
        /usr/local/go/src/net/http/header.go:106
net/http.cloneOrMakeHeader(0xc003d58780?)
        /usr/local/go/src/net/http/clone.go:69 +0x14c
net/http.(*Client).makeHeadersCopier(0xc003b4a360, 0xc01180e200)
        /usr/local/go/src/net/http/client.go:745 +0x45
net/http.(*Client).do(0xc003b4a360, 0xc01180e200)
        /usr/local/go/src/net/http/client.go:603 +0x276
net/http.(*Client).Do(...)
        /usr/local/go/src/net/http/client.go:582
lawplus/shared/requests.make(0xc01180e200, 0xb, 0xc006986090, 0x5d21dba00)
        /workspace/shared/requests/request.go:120 +0x665


goroutine 941297 [IO wait]:
internal/poll.runtime_pollWait(0x7f2096e33298, 0x72)
        /usr/local/go/src/runtime/netpoll.go:306 +0x89
internal/poll.(*pollDesc).wait(0xc0035e7b00?, 0xc002ddac00?, 0x0)
        /usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x32
internal/poll.(*pollDesc).waitRead(...)
        /usr/local/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc0035e7b00, {0xc002ddac00, 0x1300, 0x1300})
        /usr/local/go/src/internal/poll/fd_unix.go:167 +0x299
net.(*netFD).Read(0xc0035e7b00, {0xc002ddac00?, 0xc002ddac05?, 0x11a?})
        /usr/local/go/src/net/fd_posix.go:55 +0x29
net.(*conn).Read(0xc0119dd360, {0xc002ddac00?, 0x1b5b020?, 0xc013c85330?})
        /usr/local/go/src/net/net.go:183 +0x45
crypto/tls.(*atLeastReader).Read(0xc0077802e8, {0xc002ddac00?, 0xc0077802e8?, 0x0?})
        /usr/local/go/src/crypto/tls/conn.go:788 +0x3d
bytes.(*Buffer).ReadFrom(0xc013c85410, {0x1b4da20, 0xc0077802e8})
        /usr/local/go/src/bytes/buffer.go:202 +0x98
crypto/tls.(*Conn).readFromUntil(0xc013c85180, {0x1b51180?, 0xc0119dd360}, 0x1300?)
        /usr/local/go/src/crypto/tls/conn.go:810 +0xe5
crypto/tls.(*Conn).readRecordOrCCS(0xc013c85180, 0x0)
        /usr/local/go/src/crypto/tls/conn.go:617 +0x116
crypto/tls.(*Conn).readRecord(...)
        /usr/local/go/src/crypto/tls/conn.go:583
crypto/tls.(*Conn).Read(0xc013c85180, {0xc00424d000, 0x1000, 0x655f2f?})
        /usr/local/go/src/crypto/tls/conn.go:1288 +0x16f
net/http.(*persistConn).Read(0xc013b137a0, {0xc00424d000?, 0x405c66?, 0x60?})
        /usr/local/go/src/net/http/transport.go:1943 +0x4e
bufio.(*Reader).fill(0xc004d25ec0)
        /usr/local/go/src/bufio/bufio.go:106 +0xff
bufio.(*Reader).Peek(0xc004d25ec0, 0x1)
        /usr/local/go/src/bufio/bufio.go:144 +0x5d
net/http.(*persistConn).readLoop(0xc013b137a0)
        /usr/local/go/src/net/http/transport.go:2107 +0x1ac
created by net/http.(*Transport).dialConn
        /usr/local/go/src/net/http/transport.go:1765 +0x16ea
goroutine 941142 [chan receive]:
net/http.(*persistConn).addTLS(0xc0137bd7a0, {0x1b59f68?, 0xc0035c8c90}, {0xc00d9a5940, 0x1b}, 0x0)
        /usr/local/go/src/net/http/transport.go:1550 +0x365
net/http.(*Transport).dialConn(0xc006a0b040, {0x1b59f68, 0xc0035c8c90}, {{}, 0xc000512240, {0x170d77c, 0x5}, {0xc00d9a5940, 0x1f}, 0x0})
        /usr/local/go/src/net/http/transport.go:1746 +0x11b7
net/http.(*Transport).dialConnFor(0x16f6773?, 0xc0081036b0)
        /usr/local/go/src/net/http/transport.go:1456 +0xb0
created by net/http.(*Transport).queueForDial
        /usr/local/go/src/net/http/transport.go:1425 +0x3ea


goroutine 941153 [IO wait]:
internal/poll.runtime_pollWait(0x7f2096e32c08, 0x72)
        /usr/local/go/src/runtime/netpoll.go:306 +0x89
internal/poll.(*pollDesc).wait(0xc004bdb500?, 0xc002de5d40?, 0x0)
        /usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x32
internal/poll.(*pollDesc).waitRead(...)
        /usr/local/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc004bdb500, {0xc002de5d40, 0x240, 0x240})
        /usr/local/go/src/internal/poll/fd_unix.go:167 +0x299
net.(*netFD).Read(0xc004bdb500, {0xc002de5d40?, 0x0?, 0x205?})
        /usr/local/go/src/net/fd_posix.go:55 +0x29
net.(*conn).Read(0xc00ef38df0, {0xc002de5d40?, 0x47c725?, 0x0?})
        /usr/local/go/src/net/net.go:183 +0x45
crypto/tls.(*atLeastReader).Read(0xc02127ab40, {0xc002de5d40?, 0xc02127ab40?, 0x0?})
        /usr/local/go/src/crypto/tls/conn.go:788 +0x3d
bytes.(*Buffer).ReadFrom(0xc002801090, {0x1b4da20, 0xc02127ab40})
        /usr/local/go/src/bytes/buffer.go:202 +0x98
crypto/tls.(*Conn).readFromUntil(0xc002800e00, {0x1b51180?, 0xc00ef38df0}, 0xc004481710?)
        /usr/local/go/src/crypto/tls/conn.go:810 +0xe5
crypto/tls.(*Conn).readRecordOrCCS(0xc002800e00, 0x0)
        /usr/local/go/src/crypto/tls/conn.go:617 +0x116
crypto/tls.(*Conn).readRecord(...)
        /usr/local/go/src/crypto/tls/conn.go:583
crypto/tls.(*Conn).readHandshake(0xc002800e00)
        /usr/local/go/src/crypto/tls/conn.go:1020 +0x6d
crypto/tls.(*Conn).clientHandshake(0xc002800e00, {0x1b59ef8, 0xc01414d860})
        /usr/local/go/src/crypto/tls/handshake_client.go:184 +0x257
crypto/tls.(*Conn).handshakeContext(0xc002800e00, {0x1b59f68, 0xc0035c8c90})
        /usr/local/go/src/crypto/tls/conn.go:1463 +0x32e
crypto/tls.(*Conn).HandshakeContext(...)
        /usr/local/go/src/crypto/tls/conn.go:1406
net/http.(*persistConn).addTLS.func2()
        /usr/local/go/src/net/http/transport.go:1544 +0x70
created by net/http.(*persistConn).addTLS
        /usr/local/go/src/net/http/transport.go:1540 +0x345

goroutine 941154 [select]:
crypto/tls.(*Conn).handshakeContext.func2()
        /usr/local/go/src/crypto/tls/conn.go:1439 +0x98
created by crypto/tls.(*Conn).handshakeContext
        /usr/local/go/src/crypto/tls/conn.go:1438 +0x205

When looking at the code where the error is being produced, I get the error is here:
image
so probally it's because go is trying to write/iterate to the header which is underline type is a map, and it's not using any mutex

image

image

Metadata

Metadata

Assignees

No one assigned

    Labels

    FrozenDueToAgeWaitingForInfoIssue is not actionable because of missing required information, which needs to be provided.

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions