QuicConnection.ConnectAsync() connection failure causes memory leak.

[xlievo]

Description

In an automatic reconnection use case, the following code is executed in a timer. If the connection fails, it may cause a memory leak, possibly due to underlying handles not being released.

try
{
    var quicCon = await QuicConnection.ConnectAsync(QuicOptions, cancel.Token);
}
catch
{
    //It will attempt to reconnect again.
}

Reproduction Steps

internal class Program
{
    static async Task Main()
    {
        await Task.Run(async () =>
        {
            if (!QuicConnection.IsSupported)
            {
                throw new PlatformNotSupportedException("QUIC protocol is not supported on this platform.");
            }

            CancellationTokenSource cancel = new();
            using var pingTimer = new PeriodicTimer(TimeSpan.FromSeconds(1));

            while (await pingTimer.WaitForNextTickAsync(cancel.Token))
            {
                try
                {
                    await using var quicCon = await QuicConnection.ConnectAsync(new()
                    {
                        RemoteEndPoint = new IPEndPoint(IPAddress.Loopback, 9999),
                        DefaultCloseErrorCode = 0x0A,
                        DefaultStreamErrorCode = 0x0B,
                        ClientAuthenticationOptions = new SslClientAuthenticationOptions
                        {
                            ApplicationProtocols = [SslApplicationProtocol.Http3],
                            RemoteCertificateValidationCallback = (sender, certificate, chain, errors) => true,
                            TargetHost = "localhost"
                        },
                    }, cancel.Token);

                    break;
                }
                catch
                {
                    //It will attempt to reconnect again.
                }
            }
        });

        Console.WriteLine("exit;");
        Console.ReadLine();
    }
}

Expected behavior

Normal memory.

Actual behavior

The memory will continue to grow without any signs of being reclaimed.

Regression?

No response

Known Workarounds

No response

Configuration

dotnet 9.0 win-x64 AOT publish

Other information

No response

[dotnet-policy-service]

Tagging subscribers to this area: @dotnet/ncl
See info in area-owners.md if you want to be subscribed.

[xlievo]

And QuicConnection.ConnectAsync() cannot be used on Android either. #111019. Although these are two separate issues, I am only concerned about the progress and in which future version they might be fixed...

[rzikm]

VS shows this

At the same time, looking at the QUIC_PERF_COUNTER_CONN_ACTIVE perf counter, it seems like we are closing the connections properly on our end, so this seems like a bug in MsQuic itself. cc: @nibanks

Can you open an issue at microsoft/msquic repo?

[nibanks]

You'll notice that this function allocates memory from a pool (i.e., lookaside list):

DATAPATH_RX_IO_BLOCK*
CxPlatSocketAllocRxIoBlock(
    _In_ CXPLAT_SOCKET_PROC* SocketProc
    )
{
    CXPLAT_DATAPATH_PARTITION* DatapathProc = SocketProc->DatapathProc;
    DATAPATH_RX_IO_BLOCK* IoBlock;
    CXPLAT_POOL* OwningPool;

    if (SocketProc->Parent->UseRio) {
        OwningPool = &DatapathProc->RioRecvPool;
    } else {
        OwningPool = &DatapathProc->RecvDatagramPool.Base;
    }

    IoBlock = CxPlatPoolAlloc(OwningPool);

    if (IoBlock != NULL) {
        IoBlock->Route.State = RouteResolved;
        IoBlock->OwningPool = OwningPool;
        IoBlock->ReferenceCount = 0;
        IoBlock->SocketProc = SocketProc;
    }

    return IoBlock;
}

You'll be getting this from a global, per-processor pool DatapathProc->RecvDatagramPool.Base. This means the memory will purposefully be held on to, to optimize future datapath allocations.

In otherwords, this is by design.

[nibanks]

I will also say, that in more recent versions of msquic.dll, we've added support for CxPlatAddDynamicPoolAllocator which adds logic to periodically prune these pools, so eventually (8 allocs per second are pruned) they will drain if not being used. I think this logic is only in main right now.

[rzikm]

In otherwords, this is by design.

I get that there is a pool, but the private bytes of the process grew from some 30 MB to close to 1 GB, which seems a bit excessive to me.

Is there an upper limit on the pool? the two heap snapshots I took show difference of 8000+ live instances allocated from that location.

[rzikm]

What I mean to say, is that even with a pool, the app would reach a steady-state where the private bytes do not increase anymore (modulo noise), but the memory on the repro application keeps on growing. That makes me suspect that maybe the instances are not returned to the pool correctly, or there is some other bug at play.

[ManickaP]

@xlievo we believe this is MsQuic issue. I cannot transfer the issue there. So I suggest to re-file it in https://github.com/microsoft/msquic.

I'll close this for now.