Description
What version of Go are you using (go version)?
$ go version go version go1.19 darwin/amd64
Does this issue reproduce with the latest release?
yes
What operating system and processor architecture are you using (go env)?
go env Output
$ go env
What did you do?
We have a proxy server in Go that proxies over HTTP/2 to a third-party server. We're seeing that the third-party server sometimes runs out of flow control tokens and can't send anymore data back to the client. There's no clear reproduction path, but it seems to happen when the proxy is under heavy load and seeing an increase in canceled requests, due to higher latency.
Looking for potential causes in the http2 package, I noticed that in the case where the transport reads a data frame for a stream that was already canceled, it returns the flow control tokens, but it also increments the flow control counter for the connection:
As far as I can tell, the tokens haven't been subtracted at this point, so we shouldn't be adding to the window? I ran the test case below, with some logging added to the lines linked above, e.g.:
cc.inflow.add(int32(f.Length))
cc.logf("http2: Transport received DATA frame for canceled stream (window: %d)", cc.inflow.available())
func TestFlowControlDesync(t *testing.T) {
ct := newClientTester(t)
sync := make(chan struct{})
ct.client = func() error {
ctx, cancel := context.WithCancel(context.Background())
req, _ := http.NewRequest("GET", "https://dummy.tld/", nil)
req = req.WithContext(ctx)
_, err := ct.tr.RoundTrip(req)
if err != nil {
t.Fatal(err)
}
<-sync
cancel()
return nil
}
ct.server = func() error {
ct.greet()
window := initialWindowSize
f, err := ct.fr.ReadFrame()
if err != nil {
t.Fatal(err)
}
wuf := f.(*WindowUpdateFrame)
window += int(wuf.Increment)
f, err = ct.fr.ReadFrame()
if err != nil {
t.Fatal(err)
}
hf := f.(*HeadersFrame)
var buf bytes.Buffer
enc := hpack.NewEncoder(&buf)
enc.WriteField(hpack.HeaderField{Name: ":status", Value: "200"})
ct.fr.WriteHeaders(HeadersFrameParam{
StreamID: hf.StreamID,
EndHeaders: true,
EndStream: false,
BlockFragment: buf.Bytes(),
})
close(sync)
for {
data := make([]byte, maxFrameSize)
window -= len(data)
ct.fr.WriteData(hf.StreamID, false, data)
f, err := ct.fr.ReadFrame()
if err != nil {
t.Fatal(err)
}
switch fr := f.(type) {
case *WindowUpdateFrame:
window += int(fr.Increment)
t.Logf("server window: %v", window)
}
time.Sleep(time.Second)
}
}
ct.run()
}
What did you expect to see?
I'd expect flow control to be returned to the server, but the transport's counter shouldn't increase in this case.
What did you see instead?
The flow control counter keeps increasing. I think will lead to flow control desync between client and server.