Multiple concurrent http requests to non-existing HTTP/2 endpoint takes a long time to time-out

[SaxoAnders]

Description

When doing multiple concurrent HTTP/2 requests to an endpoint that does not exists (e.g. because a gRPC service was just shut down), each request is timed out one at a time, which causes the last requests to time out very slowly.

Issue can be reproduced with the following snippet:

var httpClient = new HttpClient();

Console.WriteLine(DateTime.Now);
var tasks = Enumerable.Range(1, 10).Select(async x =>
{
    try
    {
        var request = new HttpRequestMessage(HttpMethod.Get, "http://localhost:8080/test")
        {
            Version = new Version(2, 0),
            VersionPolicy = HttpVersionPolicy.RequestVersionExact
        };

        await httpClient.SendAsync(request);
    }
    catch (Exception) { }
    Console.WriteLine(DateTime.Now);
});

Task.WaitAll(tasks.ToArray());
Console.WriteLine("Done");

Expected output is that all requests time out at approximately the same time, instead I see something like:

23-04-2025 14:27:27
23-04-2025 14:27:31
23-04-2025 14:27:35
23-04-2025 14:27:39
23-04-2025 14:27:44
23-04-2025 14:27:48
23-04-2025 14:27:52
23-04-2025 14:27:56
23-04-2025 14:28:00
23-04-2025 14:28:04
23-04-2025 14:28:08
Done

Removing either Version = new Version(2, 0) or VersionPolicy = HttpVersionPolicy.RequestVersionExact makes all requests time out after ~4 seconds.

Regression?

I am not aware of any .NET version that does not have this issue.

Analysis

My guess is that this is caused by HTTP/2 trying to multiplex multiple requests through the same connection, but I think if first requests times out because endpoint does not exist, then all request in queue should just be timed out immediately.

[dotnet-policy-service]

Tagging subscribers to this area: @dotnet/ncl
See info in area-owners.md if you want to be subscribed.

[MihaZupan]

The current policy of the SocketsHttpHandler connection pool is to give every request a chance to attempt a new connection when possible.
If a connection attempt fails, we will intentionally not fail all pending requests. For example the next request might obtain a different endpoint from DNS which does work fine.

If you wish for requests to time out earlier, you could choose to decrease the HttpClient.Timeout or SocketsHttpHandler.ConnectTimeout.

[SaxoAnders]

@MihaZupan
Looks like setting Timeout gets me the behavior I want. Thank you!!

Will close issue, sorry for the inconvenience.

[SaxoAnders]

@MihaZupan
I was a bit too quick to close this. Setting HttpClient.Timeout, also changes the request timeout, this is not what I want.

We are doing thick client side load balancing, so I want long timeout for requests that reaches the endpoint, but when the endpoint goes down there is no reason to have all in-flight request to try their luck, one at a time.

Would their be any interest in exposing an API to change the default behavior?

Or maybe HttpClient could be changed to scale better for HTTP/2 in this case. E.g. if the queue of requests is long, and requests are getting connection timeouts, then maybe HttpClient could increase the number of concurrent connections it is using?

[MihaZupan]

Depending on the exact behavior you wish to see, you could consider:

• Decreasing the ConnectTimeout to time out attempts earlier.
• Call HttpClient.CancelPendingRequests() when you observe such failures to immediately abort all requests.
• Hook up a custom ConnectCallback that throws for some amount of time after the first failed connection.

Would their be any interest in exposing an API to change the default behavior?

The fact that we only fail 1 vs multiple requests on connection failure is an implementation detail of the connection management logic. We wouldn't want to expose an API for it as that'd lock us into a specific behavior.

If the queue of requests is long, and requests are getting connection timeouts, then maybe HttpClient could increase the number of concurrent connections it is using?

This might be feasible, but also only if opted into EnableMultipleHttp2Connections = true.
A similar discussion around more aggressively opening HTTP/2 connections (not necessarily because of failures) is #66944.