Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Server might not send GOAWAY when some streams are being processed #2777

Merged
merged 5 commits into from
Apr 28, 2023

Conversation

pderop
Copy link
Contributor

@pderop pderop commented Apr 20, 2023

Motivation:

Sometimes, when initiating a graceful shutdown on an HTTP2 server, it may happen that some connections are closed without sending GOAWAY.

This may (not always) happen under the following conditions:

  • the server is configured with H2/HTTP11 or H2C/HTTP11
  • some streams are currently being processed on the server
  • at this point a graceful shutdown is initiated

Under these conditions, the channelGroup that is associated to the HttpServer contains both stream channel as well as their corresponding parent socket channels. So, when the graceful shutdown is initiated, the ServerTransport.disposeNow(Duration) method is iterating on all channels registered in the channel group.

Let's say we have one http connection, and one currently active stream with id=3, so in the channelGroup, we have something like:

0 = NioSocketChannel [id: 0xfce41a80, L:/xxx.xxx.xxx.xxx:58780 - R:/xxx.xxx.xxx.xxx:58782]
1 = Http2MultiplexHandler$Http2MultiplexHandlerStreamChannel [id: 0xfce41a80, L:/xxx.xxx.xxx.xxx:58780 - R:/xxx.xxx.xxx.xxx:58782](H2 - 3)

Now, assume that during the first iteration over the channelGroup, disposeNow first finds the first channel (the parent channel: NioSocketChannel with id: 0xfce41a80 ), then this parent channel will be registered in the channelsToMono map (with an empty ArrayList).

Now, for the second iteration, disposeNow method is now getting the second channel (the stream channel: Http2MultiplexHandler$Http2MultiplexHandlerStreamChannel with id: 0xfce41a80. In this case the parent won't be inserted in the channelsToMono, because the parent has already been inserted during the first iteration.

So, during the second iteration, we won't close the stream here.

But the stream is being processed and there is a channel operation which is active so the current operation will be added to the list here, and because of this the parent channel also won't be closed here, and the socket will be closed when the JVM exits without having sent a GOAWAY message.

This PR first modifies the existing doTestGracefulShutdown in HttpServerTests, and the method is now checking that GOAWAY are received by the client. But since map iterator ordering can vary, this test may still succeed even without any patch.

Then the PR is doing a patch in the ServerTransport.disposeNow method in order to sort the channelGroup so we first handle Stream channels before handling parent ServerChannel.

Side note: when the server is configured only with H2 or with H2C, then only the stream channels (Http2MultiplexHandlerStreamChannel) are registered in the channelGroup, and the corresponding parent channels are not registered, so we don't have the problem.
However, it may be a problem to not have the parent channels registered, because in this case, if no active streams are currently being processed, then no GOAWAY message will be sent at all, which may be a problem, because if I'm correct we should always send a GOAWAY. But this is another use case, which might be addressed in another PR, if necessary.

Related to #2735

@pderop pderop added the type/bug A general bug label Apr 20, 2023
@pderop pderop added this to the 1.0.32 milestone Apr 20, 2023
@pderop pderop self-assigned this Apr 20, 2023
@pderop pderop changed the title Server might not send GOAWAY when some streams are processing Server might not send GOAWAY when some streams are being processed Apr 20, 2023
@pderop
Copy link
Contributor Author

pderop commented Apr 24, 2023

In the last commit, the impact of the patch is now minimal, we only take care to sort the channelGroup in order to ensure that streams channel are handled first, before any parent server channels.

@pderop pderop requested a review from violetagg April 24, 2023 11:50
@pderop
Copy link
Contributor Author

pderop commented Apr 24, 2023

@violetagg , can you take a look ?

@pderop
Copy link
Contributor Author

pderop commented Apr 27, 2023

@violetagg ,

can you retake a look, I just fixed a typo, and the channel group list is not sorted anymore.

@pderop
Copy link
Contributor Author

pderop commented Apr 28, 2023

@violetagg , thanks.

@pderop pderop merged commit 7990570 into reactor:1.0.x Apr 28, 2023
@pderop pderop deleted the 1.0.x-goaway-not-sent branch April 28, 2023 17:58
pderop added a commit that referenced this pull request Apr 28, 2023
pderop added a commit that referenced this pull request Apr 28, 2023
@violetagg violetagg changed the title Server might not send GOAWAY when some streams are being processed Server might not send GOAWAY when some streams are being processed May 4, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
type/bug A general bug
Projects
None yet
2 participants