x/net/http2: internal error: attempt to send frame on a closed stream

[joliveirinha]

What version of Go are you using (go version)?

1.19.3

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

amd64 linux

What did you do?

This is not an easy issue to reproduce because it seems requiring a lot of concurrency and high load of requests.
Under such high load scenario, we observe a panic in the go routine that is serving the http2 connection.

Because this panic occurs in the path of the serve(), and there is no panic recover, when it occurs, the connection is closed.
However, if the panic occurred in the scope of the runHandler, it would have been caught and recovered.
So, it is not clear to me why this particular is not covered by a panic handler.

Looking into the specific case, it seems that what is happening is that some frames are being queued to be written, and in parallel something triggered the close of a stream.
During the loop of the serve() method, frames are dequeued to be written, which encounter a frame for a stream that was already closed, thus resulting in the panic.

What did you expect to see?

The connection just fails because the panic occurs in the execution path of the serve() which doesn't recover from panics as it occurs if it was done within the runHandler method.

The http2 connection to recover from the panic and keep the http2 connection in a stable state and reusable.

What did you see instead?

panic: internal error: attempt to send frame on a closed stream: [FrameWriteRequest stream=907097, ch=false, writer=http2.writeWindowUpdate]

goroutine 136 [running]:
golang.org/x/net/http2.(*serverConn).startFrameWrite(0xc0003f2300, {{0x1386c48, 0xc053eb8ce8}, 0xc00db48000, 0x0})
    golang.org/x/net/http2/server.go:1200 +0x439
golang.org/x/net/http2.(*serverConn).scheduleFrameWrite(0xc0003f2300)
    golang.org/x/net/http2/server.go:1323 +0x105
golang.org/x/net/http2.(*serverConn).writeFrame(0xc0003f2300, {{0x1386c48?, 0xc06bcbf5d0?}, 0x0?, 0x0?})
    golang.org/x/net/http2/server.go:1176 +0x1f3
golang.org/x/net/http2.(*serverConn).sendWindowUpdate32(0xc0003f2300, 0x0, 0xea)
    golang.org/x/net/http2/server.go:2369 +0xb8
golang.org/x/net/http2.(*serverConn).sendWindowUpdate(0xc0003f2300, 0xc00098b410?, 0xc0000dd759?)
    golang.org/x/net/http2/server.go:2353 +0x95
golang.org/x/net/http2.(*serverConn).closeStream(0xc0003f2300, 0xc00db48000, {0x137de60, 0xc00007e550})
    golang.org/x/net/http2/server.go:1591 +0x194
golang.org/x/net/http2.(*serverConn).wroteFrame(0xc0003f2300, {{}, {{0x1385398, 0xc016c23dd0}, 0xc00db48000, 0xc01c2b4360}, {0x0, 0x0}})
    golang.org/x/net/http2/server.go:1266 +0x165
golang.org/x/net/http2.(*serverConn).serve(0xc0003f2300)
    golang.org/x/net/http2/server.go:908 +0xc72
golang.org/x/net/http2.(*Server).ServeConn(0xc0007601c0, {0x1391f50?, 0xc0001ec000}, 0xc000018ef0)
    golang.org/x/net/http2/server.go:504 +0xbc5

[joliveirinha]

@bradfitz not sure if this is something you can help quickly identifying the root cause.

[seankhliao]

and you're sure this isn't because your code has a data race?

[joliveirinha]

I don't want to discard that possibility.

But I find it difficult. Mainly because:

1. this is a very are event
2. the panic happens on internal data structures of the net/http2 package when handling frames to write to the conn
3. I don't see how our usage of public API (response writer and request) can trigger this
4. Also, reading the code and the place where it is, it seems to take into consideration that panics can occur.
5. A quick search and we can find some past issues related to this part of the code.

Because this part of the code of complex, I am trying to understand:

1. what is the order of events that can cause a stream to be closed while there are pending frames to be written, and that afterwards causing this panic.
2. if 1) is a valid scenario that should be covered by the internals of this package, and if so, should it handle gracefully?