Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

x/net/http2: header modifications after WriteHeader(http.StatusEarlyHints) cause a map race #67940

Closed
Timer opened this issue Jun 12, 2024 · 5 comments
Labels
NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided.
Milestone

Comments

@Timer
Copy link

Timer commented Jun 12, 2024

Go version

go version go1.22.1 linux/arm64

Output of go env in your module/workspace:

GO111MODULE=''
GOARCH='arm64'
GOBIN=''
GOCACHE='/home/timer/.cache/go-build'
GOENV='/home/timer/.config/go/env'
GOEXE=''
GOEXPERIMENT=''
GOFLAGS=''
GOHOSTARCH='arm64'
GOHOSTOS='linux'
GOINSECURE=''
GOMODCACHE='/home/timer/go/pkg/mod'
GONOPROXY=''
GONOSUMDB=''
GOOS='linux'
GOPATH='/home/timer/go'
GOPRIVATE=''
GOPROXY='https://proxy.golang.org,direct'
GOROOT='/usr/local/go1.22.1.linux-arm64'
GOSUMDB='sum.golang.org'
GOTMPDIR=''
GOTOOLCHAIN='auto'
GOTOOLDIR='/usr/local/go1.22.1.linux-arm64/pkg/tool/linux_arm64'
GOVCS=''
GOVERSION='go1.22.1'
GCCGO='gccgo'
AR='ar'
CC='gcc'
CXX='g++'
CGO_ENABLED='1'
GOMOD='/home/timer/Development/proxy/https-terminator/go.mod'
GOWORK='/home/timer/Development/proxy/go.work'
CGO_CFLAGS='-O2 -g'
CGO_CPPFLAGS=''
CGO_CXXFLAGS='-O2 -g'
CGO_FFLAGS='-O2 -g'
CGO_LDFLAGS='-O2 -g'
PKG_CONFIG='pkg-config'
GOGCCFLAGS='-fPIC -pthread -Wl,--no-gc-sections -fmessage-length=0 -ffile-prefix-map=/tmp/go-build1654315752=/tmp/go-build -gno-record-gcc-switches'

What did you do?

We run a HTTP/2 server that flushes an early headers map (status 103 - http.StatusEarlyHints) using func (http.ResponseWriter) WriteHeader(statusCode int).

After calling WriteHeader, we may need to mutate the Header map for the final 2xx response based on information we discover. In our application, we simply clear the Headers after flushing the early hint:

			r.W.Header().Set("Server", "...")
			r.W.Header().Set("Link", "...")

			r.W.WriteHeader(http.StatusEarlyHints)

			// remove the headers after sending the early hints to avoid
			// sending duplicate headers when the final response is sent:
			r.W.Header().Del("Server")
			r.W.Header().Del("Link")

What did you see happen?

After >60 minutes of runtime and millions of requests (this is hard to reproduce), we were faced with this crash:

fatal error: concurrent map read and map write
goroutine 44382232 [running]:
golang.org/x/net/http2.encodeHeaders(0xc09549f798?, 0x1412ef5?, {0x0?, 0xc0416e72b9?, 0xc07fbd2f18?})
golang.org/x/net@v0.24.0/http2/write.go:343 +0x156
golang.org/x/net/http2.(*writeResHeaders).writeFrame(0xc04d077c00, {0x15edb08, 0xc08528a000})
golang.org/x/net@v0.24.0/http2/write.go:217 +0xe9
golang.org/x/net/http2.(*serverConn).writeFrameAsync(0xc08528a000, {{0x15e9140?, 0xc04d077c00?}, 0xc01a73d860?, 0xc0326e0c60?}, 0xc070d666a8?)
golang.org/x/net@v0.24.0/http2/server.go:852 +0x74
created by golang.org/x/net/http2.(*serverConn).startFrameWrite in goroutine 44380821
golang.org/x/net@v0.24.0/http2/server.go:1266 +0x390
goroutine 1 [semacquire, 87 minutes]:

This appears to be due to writeFrameAsync running on a separate goroutine and the encodeHeaders function doing a naive iteration over the values (vv := h[k], line 343 from above).

What did you expect to see?

Encoding headers needs to be concurrent-safe now that a server can send and mutate its headers multiple times before sending the final response (e.g. 103, 103, 103, 200).

@gopherbot gopherbot added this to the Unreleased milestone Jun 12, 2024
@prattmic prattmic changed the title x/net: header modifications after WriteHeader(http.StatusEarlyHints) cause a map race x/net/http2: header modifications after WriteHeader(http.StatusEarlyHints) cause a map race Jun 12, 2024
@prattmic prattmic added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Jun 12, 2024
@prattmic
Copy link
Member

cc @neild @tombergan

@ruyi789
Copy link

ruyi789 commented Jun 12, 2024

r.W.Header().Del("Server")
Header doesn't need to be deleted? Because Response creates a new Header each time.

@neild
Copy link
Contributor

neild commented Jun 13, 2024

WriteHeader should block until the headers have been sent, to avoid precisely this type of race. I've looked through this code path a few times now, and I don't see a way for the header map to be touched after WriteHeader returns, but I might be missing something.

Do you possibly have a middleware layer wrapping the ResponseWriter that might be deferring the WriteHeader call to a later time?

If you can run this under the race detector (go build -race), it might make the failure more reproducible.)

@neild neild added the WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided. label Jun 17, 2024
@gopherbot
Copy link
Contributor

Timed out in state WaitingForInfo. Closing.

(I am just a bot, though. Please speak up if this is a mistake or you have the requested information.)

@gopherbot gopherbot closed this as not planned Won't fix, can't repro, duplicate, stale Jul 17, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided.
Projects
None yet
Development

No branches or pull requests

6 participants