System.Threading.Channels.Tests.SyncMultiReaderUnboundedChannelTests.ManyProducerConsumer_ConcurrentReadWrite_Success

[MichalStrehovsky]

[FAIL] System.Threading.Channels.Tests.SyncMultiReaderUnboundedChannelTests.ManyProducerConsumer_ConcurrentReadWrite_Success(numReaders: 1, numWriters: 1)
Assert.Equal() Failure: Values differ
Expected: 50005000
Actual:   45067347
   at System.Threading.Channels.Tests.ChannelTestBase.ManyProducerConsumer_ConcurrentReadWrite_Success(Int32 numReaders, Int32 numWriters) in /_/src/libraries/System.Threading.Channels/tests/ChannelTestBase.cs:line 205
   at System.Reflection.DynamicInvokeInfo.InvokeWithFewArguments(IntPtr, Byte&, Byte&, Object[], BinderBundle, Boolean) in /_/src/coreclr/nativeaot/System.Private.CoreLib/src/System/Reflection/DynamicInvokeInfo.cs:line 504

Build Information

Build: https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_build/results?buildId=1434067
Build error leg or test failing: System.Threading.Channels.Tests.SyncMultiReaderUnboundedChannelTests.ManyProducerConsumer_ConcurrentReadWrite_Success
Pull request:

Error Message

DO NOT USE JSON BELOW IF THIS IS A BUILD BREAK otherwise build analysis will allow pull requests to merge that break the build worse. For a build break, do not use this issue form. Make a regular new issue.

Fill the error message using step by step known issues guidance.

{
  "ErrorMessage": "Expected: 50005000",
  "BuildRetry": false,
  "ExcludeConsoleLog": false
}

Known issue validation

Build: 🔎 https://dev.azure.com/dnceng-public/public/_build/results?buildId=1434067
Error message validated: [Expected: 50005000]
Result validation: ✅ Known issue matched with the provided build.
Validation performed at: 5/27/2026 5:32:50 AM UTC

Report

Build	Repository	Test	Pull Request
1434067	dotnet/runtime	System.Threading.Channels.Tests.SyncMultiReaderUnboundedChannelTests.ManyProducerConsumer_ConcurrentReadWrite_Success

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
0	1	1

[dotnet-policy-service]

Tagging subscribers to this area: @agocke, @dotnet/ilc-contrib
See info in area-owners.md if you want to be subscribed.

[MichalStrehovsky]

I'm able to repro locally in dozen or so runs.

I let GPT-5.5 rip with this prompt:

Details

I'm investigating a test failure in System.Threading.Channels.Tests.

I modified the test to FailFast at the point of failure. I updated my machine to automatically collect last 16 crashdumps to D:\git\runtime1\Dumps. There is one dump from my last attempt available.

The test is failing with:

Process terminated. 50005000 != 32561629
   at System.RuntimeExceptionHelpers.FailFast(String, Exception, String, RhFailFastReason, IntPtr, IntPtr)
   at System.Threading.Channels.Tests.ChannelTestBase.ManyProducerConsumer_ConcurrentReadWrite_Success(Int32 numReaders, Int32 numWriters)

I'm executing the test as:

d:\git\runtime1\artifacts\bin\System.Threading.Channels.Tests\Release\net11.0\publish\System.Threading.Channels.Tests.exe -notrait category=IgnoreForCI -notrait category=OuterLoop -notrait category=failing

To rebuild the test, I use:

dotnet.cmd build src\libraries\System.Threading.Channels\tests\System.Threading.Channels.Tests.csproj -c Release -p:TestNativeAot=true -f net11.0 -p:TargetArchitecture=x86

To rebuild the AOT compiler and runtime used to build the test I use:

build.cmd clr.aot -c Release -arch x86

Note the System.Threading.Channels.Tests.csproj build might not correctly rebuild if you change the compiler or AOT runtime (it might look up-to-date for MSBuild). It's good to do a dummy change to one of the test source files to make sure it really rebuilds.

I have cdb.exe available at C:\cdb-x86.

Root cause the issue. The existing dump might not provide sufficient information to troubleshoot. You can update the test or the native AOT runtime or compiler to help troubleshooting this (add instrumentation or whatever else could help).

The failure is not deterministic, but it does repro in a dozen runs or so.

After some lazy workaround first, it came up with:

Root cause: RuntimeAsyncAwaitState.Push/Pop only stacked StackState. It did not isolate the thread-static SentinelContinuation, CachedTaskContinuation, and CachedValueTaskSourceContinuation. With sync IValueTaskSource continuations, NativeAOT can inline a nested runtime-async root while another runtime-async thunk/dispatch is active on the same thread. The nested root then reuses/clobbers those shared continuation fields, so the resumed reader can incorrectly stop while channel items remain.

Fix now: nested runtime-async roots save/clear those thread-static continuation fields on Push and restore them on Pop. Inline execution remains intact; no ThreadPool hop workaround.

Why x86 NativeAOT only? It’s not an x86 atomics or Channels issue. NativeAOT is using the runtime-async path here, and x86 made the sync-inline interleaving reproducible. The bug is in runtime-async’s thread-static state handling and may be latent elsewhere; x86 NativeAOT is just the reliable trigger we have.

Validation: x86 NativeAOT Channels repro passed 1000/1000 after this fix.

diff --git a/src/coreclr/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncHelpers.CoreCLR.cs b/src/coreclr/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncHelpers.CoreCLR.cs
index 64169a69906..d632f66ba55 100644
--- a/src/coreclr/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncHelpers.CoreCLR.cs
+++ b/src/coreclr/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncHelpers.CoreCLR.cs
@@ -227,6 +227,10 @@ private ref struct RuntimeAsyncStackState
             public ExecutionContext? RootExecutionContext;
             public SynchronizationContext? RootSynchronizationContext;
 
+            public Continuation? PreviousSentinelContinuation;
+            public RuntimeAsyncTaskContinuation? PreviousCachedTaskContinuation;
+            public ValueTaskSourceContinuation? PreviousCachedValueTaskSourceContinuation;
+
             public unsafe RuntimeAsyncStackState* Next;
 
             public void Push(Thread thread)
@@ -285,6 +289,17 @@ public void CaptureContexts()
             public void Push(RuntimeAsyncStackState* stackState)
             {
                 stackState->Next = StackState;
+                if (StackState != null)
+                {
+                    stackState->PreviousSentinelContinuation = SentinelContinuation;
+                    stackState->PreviousCachedTaskContinuation = CachedTaskContinuation;
+                    stackState->PreviousCachedValueTaskSourceContinuation = CachedValueTaskSourceContinuation;
+
+                    SentinelContinuation = null;
+                    CachedTaskContinuation = null;
+                    CachedValueTaskSourceContinuation = null;
+                }
+
                 StackState = stackState;
                 stackState->Push(CurrentThread ??= Thread.CurrentThread);
             }
@@ -293,8 +308,16 @@ public void Push(RuntimeAsyncStackState* stackState)
             public void Pop()
             {
                 Debug.Assert(CurrentThread != null);
-                StackState->Pop(CurrentThread);
-                StackState = StackState->Next;
+                RuntimeAsyncStackState* currentStackState = StackState;
+                currentStackState->Pop(CurrentThread);
+                StackState = currentStackState->Next;
+
+                if (StackState != null)
+                {
+                    SentinelContinuation = currentStackState->PreviousSentinelContinuation;
+                    CachedTaskContinuation = currentStackState->PreviousCachedTaskContinuation;
+                    CachedValueTaskSourceContinuation = currentStackState->PreviousCachedValueTaskSourceContinuation;
+                }
             }
         }

@VSadov @jakobbotsch does this sound plausible?

[jakobbotsch]

No, not very. This change just removes all caching entirely. But the caching should be robust against multiple dispatchers on the same thread. The sentinel continuation is only used during the suspension sequence while no user code is executed, and the other two are cleared out when they are used.
I can take a closer look.

[VSadov]

I'm able to repro locally in dozen or so runs.

Does it happen on x64 or x86 only?

[MichalStrehovsky]

I'm not seeing it on x64. So either it doesn't happen there or happens a lot more rarely.

FWIW, we also have #127900 that was being hit on x86 in the same test. That one didn't repro locally. I still have the x86 EXE from ~May 4th around and I'm not seeing either the NullReference, nor the count mismatch.

Hard to say whether the diff above would be able to throw off the timing by so much that it no longer reproes despite not fixing a bug, or there is a bug somewhere.

[jakobbotsch]

I suspect there is a subtle GC or hijacking related issue here. Some of my observations:

• I can still repro it even disabling all the caching
• What happens is that this task exits early. This task is normally cycling between a ValueTaskSourceContinuation (created by the thunk for WaitToReadAsync) and the continuation that represents resuming after WaitToReadAsync there. However, sometimes we end up with a ValueTaskSourceContinuation that has Next == null, and that results in the task finishing even though it should have returned to the lambda.
• Indeed this issue is almost surely the same as [ci-scan] Known Build Error: NullReferenceException in AsyncHelpers.RuntimeAsyncTask.HandleSuspended on NativeAOT x86 #127900. The symptom has just changed with recent changes in Use cached continuation in Await as well when used with ValueTaskSource #128399 -- before that PR we would NRE when Next == null in this scenario.
• Adding the following gives some more information:

diff --git a/src/coreclr/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncHelpers.CoreCLR.cs b/src/coreclr/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncHelpers.CoreCLR.cs
index 64169a69906..be0f0f7f685 100644
--- a/src/coreclr/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncHelpers.CoreCLR.cs
+++ b/src/coreclr/System.Private.CoreLib/src/System/Runtime/CompilerServices/AsyncHelpers.CoreCLR.cs
@@ -1,6 +1,7 @@
 // Licensed to the .NET Foundation under one or more agreements.
 // The .NET Foundation licenses this file to you under the MIT license.
 
+using System.Collections.Generic;
 using System.Diagnostics;
 using System.Diagnostics.CodeAnalysis;
 using System.Reflection;
@@ -8,6 +9,7 @@
 using System.Threading;
 using System.Threading.Tasks;
 using System.Threading.Tasks.Sources;
+using Internal;
 
 #if NATIVEAOT
 using Internal.Runtime;
@@ -215,6 +217,7 @@ public static partial class AsyncHelpers
             public ICriticalNotifyCompletion? CriticalNotifier;
             public INotifyCompletion? Notifier;
             public ValueTaskSourceContinuation? ValueTaskSourceContinuation;
+            public int SuspendedWithHashCode;
             public RuntimeAsyncTaskContinuation? TaskContinuation;
 
             // When we suspend in the leaf, the contexts are captured into these fields.
@@ -227,6 +230,8 @@ public static partial class AsyncHelpers
             public ExecutionContext? RootExecutionContext;
             public SynchronizationContext? RootSynchronizationContext;
 
+            public Continuation SuspendingCont;
+
             public unsafe RuntimeAsyncStackState* Next;
 
             public void Push(Thread thread)
@@ -559,6 +564,8 @@ private static unsafe T TransparentAwaitValueTaskOfT<T>(ValueTask<T> valueTask)
                 nextCont = vtsCont;
             }
 
+            state.StackState->SuspendedWithHashCode = RuntimeHelpers.GetHashCode(nextCont);
+
             sentinelContinuation.Next = nextCont;
             state.CaptureContexts();
             AsyncSuspend(nextCont);
@@ -753,6 +760,14 @@ internal unsafe bool HandleSuspended(ref RuntimeAsyncAwaitState state)
                     else if (stackState->ValueTaskSourceContinuation is { } valueTaskSourceCont)
                     {
                         Debug.Assert(headContinuation == valueTaskSourceCont);
+                        if (valueTaskSourceCont.Next == null)
+                        {
+                            Environment.FailFast(
+                                $"Should not be null! " +
+                                $"{((long)valueTaskSourceCont.ResumeInfo):x} ({RuntimeHelpers.GetHashCode(valueTaskSourceCont):x}), " +
+                                $"{((long)state.StackState->SuspendingCont.ResumeInfo):x} ({RuntimeHelpers.GetHashCode(state.StackState->SuspendingCont):x}) " +
+                                $"Suspended with: {state.StackState->SuspendedWithHashCode:x}");
+                        }
                         object? source = valueTaskSourceCont.Source;
                         Debug.Assert(source != null);
                         // The awaiter must inform the ValueTaskSource on whether the continuation
@@ -891,6 +906,7 @@ private unsafe void DispatchContinuations()
                         if (newContinuation != null)
                         {
                             newContinuation.Next = nextContinuation;
+                            awaitState.StackState->SuspendingCont = newContinuation;
                             HandleSuspended(ref awaitState);
 
                             awaitState.Pop();

It gives:

Process terminated. Should not be null! 171db08 (3ff274a), 159d63c (24d696) Suspended with: 3ff274a
   at System.RuntimeExceptionHelpers.FailFast(String, Exception, String, RhFailFastReason, IntPtr, IntPtr)

This output does not make sense. It indicates that 1) we suspended with the ValueTaskSourceContinuation (expected, this is the head continuation), 2) we returned the user code continuation (expected, this should be the tail continuation) but 3) the head continuation does not point to the tail continuation. This linking is set up by jitted code during the suspension sequence, so these fields should just have been written shortly before this.

[jakobbotsch]

I am not sure support for hijacking with async continuation was ever implemented in NativeAOT, so that seems likely to be the problem. x86 has different hijacking compared to non-x86.