[API Proposal]: Configurable HTTP connection eviction

[MihaZupan]

Background and motivation

Prototype: main...MihaZupan:http-connectionEviction

SocketsHttpHandler exposes logic for establishing new connections (namely the ConnectCallback), and some properties to control how long connections are kept around (PooledConnectionLifetime, PooledConnectionIdleTimeout).

Since DNS is only consulted when a connection is established, a common approach to ensure that the application responds to DNS changes is to configure the PooledConnectionLifetime, forcing the handler to periodically recycle connections.
But in the majority of cases, nothing actually changed, and you're forced to throw away perfectly good connections that have already been warmed up - doing all the handshakes, and possibly expanding TCP windows. This is clearly not ideal for latency.

There is also no good way to currently force connections to be shut down, short of tearing down the transport streams from under the client, or manually load balancing requests across multiple handlers.

I'm proposing we expose a callback that gives the user the option to decide whether connections should be evicted early.
The callback would not be guaranteed to run on every request. Instead, it would happen sometimes in the background. My current prototype works by periodically calling these callbacks from maintenance timers we already have.
If the user returned false in the callback, that would mark the connection for eviction, and from then on we wouldn't schedule new requests on that connection, but pending ones would be allowed to finish normally. Existing scavenging logic would eventually throw away any idle marked connections.
The callback may be invoked multiple times in parallel for a given handler instance, but only once at a time for a given connection.

To make it possible to associate custom state with a connection, we should expose the long ConnectionId, a unique identifier for a given connection within the current process. The concept is an existing one since we already expose connection IDs via public telemetry.

The RemoteEndPoint property would expose the same info we currently report in telemetry, which we extract from the socket if possible (hence nullable).

API Proposal

namespace System.Net.Http;

public partial sealed class SocketsHttpHandler
{
    // Existing
    // public Func<SocketsHttpConnectionContext, CancellationToken, ValueTask<Stream>>? ConnectCallback { get; set; }
    // public Func<SocketsHttpPlaintextStreamFilterContext, CancellationToken, ValueTask<Stream>>? PlaintextStreamFilter { get; set; }

    [Experimental]
    public Func<SocketsHttpConnectionEvictionContext, CancellationToken, Task<bool>>? ShouldEvictConnection { get; set; }
}

[Experimental]
public sealed class SocketsHttpConnectionEvictionContext // New type
{
    // No public ctor

    public TimeSpan Age { get; }
    public long ConnectionId { get; }
    public Version HttpVersion { get; }
    public DnsEndPoint DnsEndPoint { get; }
    public IPEndPoint? RemoteEndPoint { get; }
}

public partial sealed class SocketsHttpConnectionContext // Existing type used for ConnectCallback
{
    [Experimental]
    public long ConnectionId { get; }
}

public partial sealed class SocketsHttpPlaintextStreamFilterContext // Existing type used for PlaintextStreamFilter
{
    [Experimental]
    public long ConnectionId { get; }
}

API Usage

Evict connections when DNS changes

var handler = new SocketsHttpHandler
{
    PooledConnectionLifetime = Timeout.InfiniteTimeSpan,
    ShouldEvictConnection = async (context, ct) =>
    {
        if (context.RemoteEndPoint is IPEndPoint connectionIp)
        {
            var addresses = await _cache.GetOrCreateAsync(
                context.DnsEndPoint.Host,
                async ct => await Dns.GetHostAddressesAsync(context.DnsEndPoint.Host, connectionIp.AddressFamily, ct),
                cancellationToken: ct);

            return !addresses.Contains(connectionIp.Address);
        }

        // This was a custom stream created via ConnectCallback (unreachable with default settings), fall back to a lifetime limit
        return context.Age > TimeSpan.FromMinutes(10);
    }
};

Evict based on information from a custom transport stream used via ConnectCallback

ConcurrentDictionary<long, WriteCountingStream> connections = [];

new SocketsHttpHandler
{
    ConnectCallback = async (context, ct) =>
    {
        var socket = new Socket(SocketType.Stream, ProtocolType.Tcp) { NoDelay = true };
        try
        {
            socket.SetSocketOption(SocketOptionLevel.Socket, SocketOptionName.KeepAlive, true);
            // ...

            var networkStream = new NetworkStream(socket, ownsSocket: true);
            var stream = new WriteCountingStream(networkStream, context.ConnectionId, connections);

            connections.TryAdd(context.ConnectionId, stream);

            return stream;
        }
        catch
        {
            socket.Dispose();
            throw;
        }
    },
    ShouldEvictConnection = async (context, ct) =>
    {
        if (context.Age.TotalSeconds > 5 && // Grace period
            connections.TryGetValue(context.ConnectionId, out WriteCountingStream stream))
        {
            double averageBitrate = stream.TotalBytesWritten / context.Age.TotalSeconds * 8;
            return averageBitrate is < 10 * 1024 or > 1024 * 1024 * 1024; // Only keep connections with bitrates between 10 kbit and 1 gbit.
        }

        return false;
    }
};

Evict connections with many consecutive 502s

Requires that ConnectionId be exposed on the request/response message (follow-up proposal #130108), otherwise it takes lots of ceremony to obtain the data from EventSource telemetry.

HttpMessageHandler handler = new SocketsHttpHandler
{
    PooledConnectionLifetime = TimeSpan.FromMinutes(10), // Eviction decisions complement each other
    ShouldEvictConnection = async (context, ct) =>
    {
        if (_cache.TryGetValue(context.ConnectionId, out StrongBox<int> failureCounter))
        {
            return failureCounter.Value > 10;
        }

        return false;
    }
};

handler = new Http502ResponseCounterHandler(handler, _cache);

sealed class Http502ResponseCounterHandler(HttpMessageHandler handler, IMemoryCache cache) : DelegatingHandler(handler)
{
    protected override async Task<HttpResponseMessage> SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
    {
        var response = await base.SendAsync(request, cancellationToken);

        if (request.ConnectionId is long connectionId)
        {
            var counter = cache.GetOrCreate(connectionId, _ => new StrongBox<int>());

            if (response.StatusCode == HttpStatusCode.BadGateway)
            {
                Interlocked.Increment(ref counter.Value);
            }
            else if (counter.Value > 0)
            {
                Interlocked.Decrement(ref counter.Value);
            }
        }

        return response;
    }
}

Alternative Designs

• The RemoteEndPoint is typed as IPEndPoint? right now, but we could consider making it EndPoint? instead in case we later expose APIs on the ConnectCallback context that allow users to populate the info even if the stream isn't a NetworkStream. That would also help the gap in telemetry where we can't currently report the info.
• HttpVersion seems like a reasonable name to me, but we've also used NegotiatedHttpVersion on SocketsHttpPlaintextStreamFilterContext.
• I left the details about when/how often we run the callback as an implementation detail right now with the assumption that you normally have some grace period, so there's no need to recheck everything 10 times a second and immediately react. The prototype currently uses Min(5 seconds, existing cleanup timer interval). The callback implementor can always rate limit calls themselves, or we could make it configurable in the future.

Risks

We could be accidentally exposing too much internal info. We're adding [Experimental] for now to keep the option to tweak the API in the future.

[dotnet-policy-service]

Tagging subscribers to this area: @karelz, @dotnet/ncl
See info in area-owners.md if you want to be subscribed.