Possible object rooting issue with runtime-async

[BrennanConroy]

Description

dotnet/aspnetcore#66200 is enabling runtime-async for aspnetcore dlls and tests. We have 1 test that is consistently failing with the change.

https://github.com/dotnet/aspnetcore/blob/df9f19afc8db182a48528d3e06d67344ac6670e6/src/Servers/Kestrel/test/InMemory.FunctionalTests/HttpConnectionManagerTests.cs#L24
This test verifies that Kestrel isn't rooting a connection after the connection is closed even if application code is still running.

I grabbed a gcroot of the KestrelConnection during the test run to see what was holding onto it in case that's helpful:

Details

000001f388d711b8 (strong handle)
          -> 01f38a802020     System.Object[] 
          -> 01f3ac3706e0     System.Collections.Generic.Dictionary<System.Int32, System.Collections.Generic.List<System.Runtime.InteropServices.PosixSignalRegistration+Token>> (static variable: System.Action<System.Object>.<>9__19_1)
          -> 01f3ac3707a8     System.Collections.Generic.Dictionary<System.Int32, System.Collections.Generic.List<System.Runtime.InteropServices.PosixSignalRegistration+Token>>+Entry[] 
          -> 01f3ac370748     System.Collections.Generic.List<System.Runtime.InteropServices.PosixSignalRegistration+Token> 
          -> 01f3ac370808     System.Runtime.InteropServices.PosixSignalRegistration+Token[] 
          -> 01f3ac3706a8     System.Runtime.InteropServices.PosixSignalRegistration+Token 
          -> 01f3ac370668     System.Action<System.Runtime.InteropServices.PosixSignalContext> 
          -> 01f3ac36fe88     Microsoft.Extensions.Hosting.Internal.ConsoleLifetime 
          -> 01f3ac36ff80     Microsoft.Extensions.Logging.Logger 
          -> 01f3ac370040     Microsoft.Extensions.Logging.ScopeLogger[] 
          -> 01f3ac36ffb0     Microsoft.AspNetCore.Server.Kestrel.InMemory.FunctionalTests.HttpConnectionManagerTests+CallbackLoggerProvider+CallbackLogger 
          -> 01f3aadbd4c0     System.Action<Microsoft.Extensions.Logging.EventId> 
          -> 01f3aadbcea8     Microsoft.AspNetCore.Server.Kestrel.InMemory.FunctionalTests.HttpConnectionManagerTests+<>c__DisplayClass0_0 
          -> 01f3aadbced0     System.Threading.SemaphoreSlim 
          -> 01f3ad400028     System.Threading.SemaphoreSlim+TaskNode 
          -> 01f3ad400080     System.Threading.Tasks.Task+CancellationPromise<System.Boolean> 
          -> 01f3ad400148     System.Runtime.CompilerServices.AsyncTaskMethodBuilder<System.Boolean>+AsyncStateMachineBox<System.Threading.SemaphoreSlim+<WaitUntilCountOrTimeoutAsync>d__33> 
          -> 01f3add58398     System.Action 
          -> 01f3add58350     System.Runtime.CompilerServices.AsyncHelpers+RuntimeAsyncTask<System.Threading.Tasks.VoidTaskResult> 
          -> 01f3ad4001f8     Continuation_16_0_2 
          -> 01f3add582d8     Continuation_80_8_2_32_5 
          -> 01f3ab921310     Microsoft.AspNetCore.Server.Kestrel.InMemory.FunctionalTests.TestTransport.TestServer 
          -> 01f3ab9218b8     Microsoft.AspNetCore.Server.Kestrel.InMemory.FunctionalTests.TestTransport.InMemoryTransportFactory 
          -> 01f3ab9218d8     System.Threading.Channels.UnboundedChannel<Microsoft.AspNetCore.Connections.ConnectionContext> 
          -> 01f3ab921dd0     System.Threading.Channels.WaitingReadAsyncOperation 
          -> 01f3ac537f30     System.Runtime.CompilerServices.AsyncHelpers+RuntimeAsyncTask<Microsoft.AspNetCore.Connections.ConnectionContext> 
          -> 01f3ac535868     System.Runtime.CompilerServices.AsyncHelpers+RuntimeAsyncTask<System.Threading.Tasks.VoidTaskResult> 
          -> 01f3ac537fb8     Continuation_48_0_5 
          -> 01f3ac535808     Continuation_56_8_5 
          -> 01f3add56a68     Microsoft.AspNetCore.Server.Kestrel.InMemory.FunctionalTests.TestTransport.InMemoryTransportConnection 
          -> 01f3ac535a48     System.Collections.Generic.List<System.Collections.Generic.KeyValuePair<System.Type, System.Object>> 
          -> 01f3ac535a68     System.Collections.Generic.KeyValuePair<System.Type, System.Object>[] 
          -> 01f3ac535918     Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Infrastructure.KestrelConnection<Microsoft.AspNetCore.Connections.ConnectionContext> 

Reproduction Steps

Enable runtime-async and then run the Kestrel test:

dotnet test -c Release --filter CriticalErrorLoggedIfApplicationDoesntComplete

Expected behavior

Test passes.

Actual behavior

Test fails.

| Xunit.Sdk.TrueException: Assert.True() Failure
| Expected: True
| Actual:   False
|    at Xunit.Assert.True(Nullable`1 condition, String userMessage) in /_/src/xunit.assert/Asserts/BooleanAsserts.cs:line 141

Regression?

No response

Known Workarounds

No response

Configuration

No response

Other information

No response

[dotnet-policy-service]

Tagging subscribers to this area: @agocke, @dotnet/gc
See info in area-owners.md if you want to be subscribed.

[dotnet-policy-service]

Tagging subscribers to this area: @agocke
See info in area-owners.md if you want to be subscribed.

[VSadov]

The affected testcases looks very similar to

runtime/src/libraries/System.Net.Http/tests/FunctionalTests/HttpClientHandlerTest.Finalization.cs

Line 29 in 263d5b9

public async Task IncompleteResponseStream_ResponseDropped_CancelsRequestToServer()

There was an existing retention of task context that in combination with runtime async was holding too much and had to be fixed in #124491

Perhaps we have a similar issue here.
(also possible that the fix did not cover all the occurrences of the same pattern)

[wtgodbe]

We've been able to reproduce another runtime-async hang in aspnetcore E2E tests that may be related to this rooting issue.

Repro: ServerFormsTest.InputTextAreaWorksWithMutatingSetter hangs indefinitely with runtime-async=on but fails normally (clean assertion timeout) without it.

• With runtime-async: test host process hangs until blame-hang kills it (60s+)
• Without runtime-async: test hits a WaitAssert timeout and fails cleanly (~77s)

This was found running the aspnetcore runtime-async PR. Out of 260 ServerExecutionTest E2E tests, 258 passed and only this one hung — suggesting a specific async pattern in the Blazor forms/input binding code triggers the issue rather than a blanket incompatibility.