Netty 4.2.5.Final race condition handling GOAWAY frame

[ethanhere]

Sorry for reposting the similar question because the previous question (#15836) was closed. However this is the updated question:
We have seen the following error in our application log:

[2025-10-22 20:48:26.797] - [httpclient-nioEventLoopGroup-2-13] TRACE i.m.h.client.netty.DefaultHttpClient - Request Body
[2025-10-22 20:48:26.797] - [httpclient-nioEventLoopGroup-2-13] TRACE i.m.h.client.netty.DefaultHttpClient - ----
[2025-10-22 20:48:26.797] - [httpclient-nioEventLoopGroup-2-8] TRACE i.m.h.client.netty.DefaultHttpClient - "test-name-space:instance:1"
[2025-10-22 20:48:26.797] - [httpclient-nioEventLoopGroup-2-8] TRACE i.m.h.client.netty.DefaultHttpClient - ----
[2025-10-22 20:48:26.800] - [httpclient-nioEventLoopGroup-2-8] TRACE i.m.h.client.netty.DefaultHttpClient - Http2PoolEntry[5, pool=Pool[httpclient-nioEventLoopGroup-2-8]] became unavailable
[2025-10-22 20:48:26.800] - [httpclient-nioEventLoopGroup-2-8] DEBUG i.m.h.client.netty.DefaultHttpClient - Server sent GOAWAY frame. errorCode=0 base64(content)=
[2025-10-22 20:48:26.802] - [httpclient-nioEventLoopGroup-2-8] TRACE i.m.h.client.netty.DefaultHttpClient - HTTP Client exception (Connection closed before response was received) occurred for request : POST https://host/endpoint
[2025-10-22 20:48:26.810] - [main] DEBUG i.m.h.c.i.HttpClientIntroductionAdvice - Client [myapp.SvcClient] received HTTP error response: Client 'svc': Connection closed before response was received
io.micronaut.http.client.exceptions.ResponseClosedException: Client 'svc': Connection closed before response was received

The application(client side) and the service are both implemented using java and micronaut (micronaut platform 4.9.4 and netty 4.2.5.Final).

After carefully exam the log and other information, we found the following:
The client sent a request with body "test-name-space:instance:1", and 3 milliseconds later, it received GOAWAY frame and connection pool entry 5 removed from the pool.
Then HTTP Client exception happens.
We think the HTTP Client exception is raised because there is a request added to the connection Http2PoolEntry 5 at the moment it is closed, or right before the closure.
This looks like a race condition: the client side adds a request while the server side sent GOAWAY frame while not knowing each other's simultaneous action.

According to https://datatracker.ietf.org/doc/html/rfc7540#section-6.8, the inherent race condition is discussed:
"
There is an inherent race condition between an endpoint starting new
streams and the remote sending a GOAWAY frame. To deal with this
case, the GOAWAY contains the stream identifier of the last peer-
initiated stream that was or might be processed on the sending
endpoint in this connection. For instance, if the server sends a
GOAWAY frame, the identified stream is the highest-numbered stream
initiated by the client.

Once sent, the sender will ignore frames sent on streams initiated by
the receiver if the stream has an identifier higher than the included
last stream identifier. Receivers of a GOAWAY frame MUST NOT open
additional streams on the connection, although a new connection can
be established for new streams.

If the receiver of the GOAWAY has sent data on streams with a higher
stream identifier than what is indicated in the GOAWAY frame, those
streams are not or will not be processed. The receiver of the GOAWAY
frame can treat the streams as though they had never been created at
all, thereby allowing those streams to be retried later on a new
connection.
"

I think netty on the client side, has the last stream id, therefore it knows if the in-flight requst will be processed or not when receiving GOAWAY frame. Would netty retry that request by opening another connection? or it can somehow signal upper level code, such as micronaut to try?

[slandelle]

Hi,

This looks like something that should be reported directly to the micronaut project, not here.

Regards

[ethanhere]

I don't understand: netty should know the last stream id. how is that exposed to micronaut? If exposed, please explain; if not exposed, then micronaut cannot decide weather to retry. i agree that there is a pooling in micronaut therefore micronaut code can find or create a connection.

[normanmaurer]

@yawkat ^^