Hyper Hung Client Example

Issue

We have run into shutdown hangs in our tonic gRPC server that we have isolated to a single change:

hyperium/hyper#3729

There are two separate failure modes this repository shows:

Non Upgraded Connections Can Block Shutdown

This is contained in the non_upgrade_connections_can_block_shutdown() function.

An upstream service of ours is connecting thousands of times to a downstream service using multiplexing on a single client.

In our lower volume staging environment, it is not uncommon that many of those connections may not be used given a short enough time period.

Those non-upgraded connections are blocking tonic's serve_with_incoming_shutdown. This was first tracked down when restarting the upstream service "fixed" the shutdown hang.

Hung Clients Also Can Block Shutdown

This is contained in the client_not_responding_example function.

We do not believe we saw this in production, but came across it while trying to originally isolate a unit test for this issue.

If you connect a client to a tonic server, and then hang it so it can not fully complete the network communication, that will also block a tonic server's serve_with_incoming_shutdown.

This test case uses a single threaded tokio runtime on a thread and a blocking channel read, but you can imagine this occurring in production with a less synthetic set of circumstances

Steps to reproduce

1. cargo run - This will run non_upgrade_connections_can_block_shutdown and will fail with:

thread 'main' (1700557) panicked at src/main.rs:118:5:
assertion failed: !shutdown_timed_out
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

2. Change the commented function in main to be client_not_responding_example instead and re-run. It will fail with:

Got a request: Request { metadata: MetadataMap { headers: {"te": "trailers", "content-type": "application/grpc", "user-agent": "tonic/0.14.6"} }, message: HelloRequest { name: "Tonic" }, extensions: {tonic::transport::server::conn::TcpConnectInfo, axum::routing::url_params::UrlParams} }

thread 'main' (1701915) panicked at src/main.rs:204:5:
assertion failed: !shutdown_timed_out
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

3. Open Cargo.toml and uncomment the patch.crates-io section. This will move hyper to a branch with this commit which reverts the PR in question
4. cargo update -p hyper
5. Re-run both example, and they will both now pass