Jetty Http2 client discards the response fames when there is GOAWAY and sends RST_STREAM

[sukawanth]

Jetty version 9.4.27.v20200227

Java version OpenJdk 14

OS type/version Linux master 4.1.12-112.16.4.el7uek.x86_64 /Deployed in Kubernetes cluster

Description
We are using Jetty Http2 client in our application. We have a scenario where the server will terminate the connection by sending GOAWAY frame with no error along with the response header and data frames. But Jetty client cancels the request by sending RST_STREAM and terminates the connection by sending GOAWAY. In the application side client also throws "java.nio.channels.AsynchronousCloseException" without committing response.

{"thread":"@d8d9199-4177","level":"ERROR","loggerName":"Exception","message":"Internal Server Error","thrown":{"commonElementCount":0,"name":"java.nio.channels.AsynchronousCloseException","extendedStackTrace":[{"class":"org.eclipse.jetty.http2.client.http.HttpConnectionOverHTTP2","method":"close","file":"HttpConnectionOverHTTP2.java","line":144,"exact":false,"location":"http2-http-client-transport-9.4.27.v20200227.jar!/","version":"9.4.27.v20200227"}],"suppressed":[{"commonElementCount":0,"localizedMessage":"\nError has been observed at the following site(s):\n\t|_ checkpoint ⇢ Request to POST

[sbordet]

You do not say what the problem is, you just describe a normal scenario.

Why do you think what you describe is an issue?

[gregw]

@sukawanth What we see in your trace does indicate your server is a little strange. We see

Client <-------------> Server
 <---- Settings    // probably response to settings in the upgrade
 <---- Settings    // strange second settings frame here, but no harm
 Settings ---->    // client responding to server settings frame
Headers ---->      // POST
Data ---->         // POST body... but can't tell if that is complete?
Settings ---->     // More settings? rather strange???? @sbordet any idea why?
<---- GOAWAY       // The first go away! Maybe it didn't like the settings?
RST ---->           // Client resets stream! because of go away
GOAWAY ---->       // Client responds to go away
Headers ---->     // Another POST???? is this the same connection?  Can you turn on ports?

Very strange there is more conversation after the goaway and its response. Also the multiple settings are confusing.
Can you display source and destination port as well as I think multiple connections may be in play.

[sukawanth]

hi @gregw yes you are correct there are multiple connections in play. I filter out one connection and attached the pic. We are facing a scenario where server is gracefully shutting down and is gracefully terminating all the connections established to it. So when new connections are established to the server and request is sent to it, server is accepting connection and responding to request with 200OK along with GO AWAY frame. this is observed for grace period after which connections are not accepted at server.

@sbordet we expect that the response should be processed which is received with GOAWAY frame in same TCP packet. why jetty client is sending RST frame for the request? And for the client side out code is not receiving response instead seeing connection
error.

[sbordet]

The problem seems on the server.

The client sends a "magic" + its settings + window_update
The server replies with its own settings + settings reply acknowledging the client one + window_update
The client sends a POST request
The server sends a GOAWAY and then the response?!?

What server is it (product and version)?

What do you exactly mean by "scenario where server is gracefully shutting down and is gracefully terminating all the connections established to it"?

If the server is shutting down, the GOAWAY is expected, although the last stream id contained in the GOAWAY is bogus.
What do the server logs say?

Can you please write down exactly what you are doing, what do you expect to happen, and what actually happens instead?

[sukawanth]

Hi @sbordet, Thanks for the reply

Below is more details explanation is the issue:

Environment:
we are using Spring cloud gateway as a reverse proxy in our environment. SCG receives a request and uses jetty http2 client library to send request to backend server.
Backend server is Envoy proxy V1.11.0.
We are using jetty round robin connection pool to establish connections to backend.
This solution is deployed in kubernetes cluster as docker containers.
We have a single container for SCG which receives request and send to backend. Backend has multiple containers which map to one FQDN. We have customized round robin connection pool to resolve backend FQDN and create connections to all resolved IPs. And we have a scheduler in SCG to refresh the FQDN-IP mapping and update the connection pool with changed ip list.

Test scenario:
While traffic is running, one of the back end container is gracefully shutdown. So jetty client received GOAWAY on that connection with promised-stream-id as (2Pow31)-1. As per the HTTP2 specification, in this scenario jetty client shall wait for some time untill final GOAWAY is received from server.
We see server sending GOAWAY frame with promised stream id as (2Pow31)-1 indicating graceful shutdown and sending response to pending request later.

Expected behavior of jetty client:
upon receiving GOAWAY frame with promised stream id as (2Pow31)-1 from server jetty client shall wait for responses for pending requests on that connection or until final GOAWAY is received before closing connection.
Process any response received before final GOAWAY from server.

Obseravtion:
Upon receiving GOAWAY frame with promised stream id as (2Pow31)-1 from server, jetty client is cancelling the pending request on the connection and sending GOAWAY to server.

Question is why jetty is not waiting for responses on connection after receiving GOAWAY frame with promised stream id as (2Pow31)-1

[sbordet]

Ok, so the HTTP/2 client implementation does not wait for responses after it receives a GOAWAY, that's why it resets the stream.
The spec contemplates for the scenario that the client should wait for responses, so we will implement it.

[sbordet]

@sukawanth are you able to build and test branch jetty-9.4.x-5310-http2_goaway to test whether the fixes work for you?