x/net/http2: header modifications after WriteHeader(http.StatusEarlyHints) cause a map race #67940
Comments
|
Similar Issues
(Emoji vote if this was helpful or unhelpful; more detailed feedback welcome in this discussion.) |
prattmic
changed the title
prattmic
added
the
NeedsInvestigation
|
cc @neild @tombergan |
|
r.W.Header().Del("Server") |
|
Do you possibly have a middleware layer wrapping the If you can run this under the race detector ( |
neild
added
the
WaitingForInfo
|
Timed out in state WaitingForInfo. Closing. (I am just a bot, though. Please speak up if this is a mistake or you have the requested information.) |
|
No, we are not wrapping the |
|
Got the race! Sharing it here while I continue to evaluate it. |
|
I may be missing it, but I don't see where The http2 For 1xx status codes, it calls
And the Perhaps the header map sent in to |
|
cc @neild in case you unsubscribed from the issue after being closed 🙏, hope you don't mind. |
seankhliao
removed
the
WaitingForInfo
|
Thanks for the race detector output; that was very useful. The problem is that while writeHeaders usually waits for the write to complete before returning, it stops waiting if the stream is reset by the peer. The race occurs when the header write has already started at the time the peer's reset is processed. (This is the I'm still trying to figure out the right fix; there's a fairly complex set of interactions between the different goroutines involved in writing a frame. Most of this complexity is in service of efficiency, and I suspect the more obvious fixes leave us with all the complexity and none of the efficiency. |
|
If it's of any anecdotal value, for latency/efficiency sake, we'd prefer Also, I did not catch that In either case, the most trivial fix appears to be moving the |
Go version
go version go1.22.1 linux/arm64
Output of
go envin your module/workspace:What did you do?
We run a HTTP/2 server that flushes an early headers map (status 103 - http.StatusEarlyHints) using
func (http.ResponseWriter) WriteHeader(statusCode int).After calling WriteHeader, we may need to mutate the Header map for the final 2xx response based on information we discover. In our application, we simply clear the Headers after flushing the early hint:
What did you see happen?
After >60 minutes of runtime and millions of requests (this is hard to reproduce), we were faced with this crash:
This appears to be due to
writeFrameAsyncrunning on a separate goroutine and theencodeHeadersfunction doing a naive iteration over the values (vv := h[k], line 343 from above).What did you expect to see?
Encoding headers needs to be concurrent-safe now that a server can send and mutate its headers multiple times before sending the final response (e.g. 103, 103, 103, 200).
The text was updated successfully, but these errors were encountered: