Description
We run MassTransit with the Amazon SQS transport. Since moving to .NET 10, SQS receive endpoints occasionally stop polling while the process and the bus keep running, usually with nothing in the logs (MassTransit/MassTransit#6226). Investigating that symptom led to System.Threading.Channels: the transport funnels SQS client calls through a bounded channel with one waiting reader and many writers, and under CPU contention that channel intermittently fails.
Stress rig that reproduces it: https://github.com/PooryaCareExpert/repro — many independent "batchers" (a bounded(100) channel with SingleReader = true, FullMode.Wait, AllowSynchronousContinuations = false; one parked reader plus 10 paced writers, mirroring MassTransit's SQS Batcher<TEntry>) in CPU-constrained Linux containers.
Reproduction Steps
git clone https://github.com/PooryaCareExpert/repro
cd repro
docker compose --profile 9 --profile 10 --profile 11 up --build -d --scale stress-9-0=5 --scale stress-10-0=5 --scale stress-11-0=5
docker compose --profile 9 --profile 10 --profile 11 logs -f
Hits are probabilistic; with 12–16 instances sharing a 14-core cpuset the first hit typically lands within the hour. Knobs are in the repo README; captured runs are in RESULTS.md.
Actual behavior
Three failure shapes, all on the same bounded channel:
1. Writer side — WriteAsync throws:
System.NullReferenceException: Object reference not set to an instance of an object.
at System.Threading.Channels.ChannelUtilities.DangerousSetOperations[TAsyncOp,TResult](TAsyncOp head, TResult result)
at System.Threading.Channels.BoundedChannel`1.BoundedChannelWriter.WriteAsync(T item, CancellationToken cancellationToken)
at StressRepro.Batcher.Produce()
2. Reader side — the single reader's WaitToReadAsync await throws, killing the consume loop (in MassTransit this kills the endpoint):
System.InvalidOperationException: The asynchronous operation has not completed.
at System.Threading.Channels.AsyncOperation.ThrowIncompleteOperationException()
at System.Threading.Channels.AsyncOperation`2.GetResult(Int16 token)
at StressRepro.Batcher.Consume()
3. Silent — the parked reader is never woken and nothing is thrown: consumption on that channel stops permanently while writers park forever. This matches the MassTransit#6226 reports where an endpoint dies with no log line at all.
On 11.0.0-preview.6 the reader has additionally thrown OperationCanceledException carrying a cancellation token the application never created.
Shapes 2/3 usually land on a channel that had already produced a shape-1 writer exception (10 of 11 same-channel pairings across two runs).
Results
25 container instances (details in the repo's RESULTS.md):
| Runtime |
Writer NREs |
Reader kills / strands |
| 9.0.18 |
0 |
0 — control, clean on all instances |
| 10.0.10 |
28 |
1 (shape 2) |
| 11.0.0-preview.6 |
50 |
10 (shape 3) |
Expected behavior
No exceptions out of WriteAsync / WaitToReadAsync, and a parked reader always wakes when items are written.
Regression?
The identical rig built for net9.0 runs clean in every control run; .NET 10.0.10 and 11.0.0-preview.6.26359.118 both hit.
Known Workarounds
Targeting net9.0.
Configuration
- .NET 10.0.10 and 11.0.0-preview.6.26359.118 (affected); 9.0.18 (clean)
- linux/amd64 containers (Docker Desktop on Windows 11, x64 host), 8-CPU quota, shared
cpuset
- Also observed in production: x64 AWS Fargate (1 vCPU), .NET 10, MassTransit Amazon SQS
Other information
In production the writer-side NullReferenceException appears roughly every other day across 5 environments, always the same DangerousSetOperations → BoundedChannelWriter.WriteAsync stack (reached via MassTransit's DeleteMessage batching; each hit means a failed SQS delete and a duplicate delivery). Reader-side endpoint deaths are rarer and match the stopped-endpoint reports in MassTransit/MassTransit#6226.
Description
We run MassTransit with the Amazon SQS transport. Since moving to .NET 10, SQS receive endpoints occasionally stop polling while the process and the bus keep running, usually with nothing in the logs (MassTransit/MassTransit#6226). Investigating that symptom led to
System.Threading.Channels: the transport funnels SQS client calls through a bounded channel with one waiting reader and many writers, and under CPU contention that channel intermittently fails.Stress rig that reproduces it: https://github.com/PooryaCareExpert/repro — many independent "batchers" (a
bounded(100)channel withSingleReader = true,FullMode.Wait,AllowSynchronousContinuations = false; one parked reader plus 10 paced writers, mirroring MassTransit's SQSBatcher<TEntry>) in CPU-constrained Linux containers.Reproduction Steps
git clone https://github.com/PooryaCareExpert/repro cd repro docker compose --profile 9 --profile 10 --profile 11 up --build -d --scale stress-9-0=5 --scale stress-10-0=5 --scale stress-11-0=5 docker compose --profile 9 --profile 10 --profile 11 logs -fHits are probabilistic; with 12–16 instances sharing a 14-core
cpusetthe first hit typically lands within the hour. Knobs are in the repo README; captured runs are in RESULTS.md.Actual behavior
Three failure shapes, all on the same bounded channel:
1. Writer side —
WriteAsyncthrows:2. Reader side — the single reader's
WaitToReadAsyncawait throws, killing the consume loop (in MassTransit this kills the endpoint):3. Silent — the parked reader is never woken and nothing is thrown: consumption on that channel stops permanently while writers park forever. This matches the MassTransit#6226 reports where an endpoint dies with no log line at all.
On 11.0.0-preview.6 the reader has additionally thrown
OperationCanceledExceptioncarrying a cancellation token the application never created.Shapes 2/3 usually land on a channel that had already produced a shape-1 writer exception (10 of 11 same-channel pairings across two runs).
Results
25 container instances (details in the repo's RESULTS.md):
Expected behavior
No exceptions out of
WriteAsync/WaitToReadAsync, and a parked reader always wakes when items are written.Regression?
The identical rig built for
net9.0runs clean in every control run; .NET 10.0.10 and 11.0.0-preview.6.26359.118 both hit.Known Workarounds
Targeting
net9.0.Configuration
cpusetOther information
In production the writer-side
NullReferenceExceptionappears roughly every other day across 5 environments, always the sameDangerousSetOperations→BoundedChannelWriter.WriteAsyncstack (reached via MassTransit'sDeleteMessagebatching; each hit means a failed SQS delete and a duplicate delivery). Reader-side endpoint deaths are rarer and match the stopped-endpoint reports in MassTransit/MassTransit#6226.