Proposal: cheaper thread pool hops for async continuations (run-next slot + deferred worker requests)

[thomhurst]

Summary

Async code has to choose between two kinds of continuations. Inline continuations are cheap, but they run user code on the thread that completes the operation, and that code can block it. Asynchronous continuations (RunContinuationsAsynchronously, ConfigureAwait hops, channel and pipe readers, and so on) avoid that, but each one pays for a thread pool hop. This proposes a change to the portable thread pool that makes that hop much cheaper, taking inspiration from the schedulers in Go and Rust's Tokio. In the measurements below the hop is about 7x cheaper for a raw work item and about 2x cheaper for an await ping-pong, with 5–40x less CPU.

This is a proposal for discussion, not a PR. Every behavior change in the linked branch is off by default behind config switches. Feedback is welcome on whether the direction is worth pursuing, and from the thread pool owners on the liveness design in particular.

Diff (branch vs main at 366f2fd): 366f2fd...thomhurst:runtime:threadpool-runnext-deferred-wake

What the current hop costs

A single chain of continuations bounced between two TaskCompletionSources with RunContinuationsAsynchronously costs about 530–730 ns per hop and about 2,400–4,100 ns of CPU per hop. Roughly 5–9 cores burn to run one sequential chain. The same chain with inline continuations costs about 50 ns.

Here is why. When a worker queues a continuation to its local queue, ThreadPoolWorkQueue.Enqueue always calls EnsureWorkerRequested, which wakes a worker. The worker that wakes spins, and it often steals the item before the queuing thread gets back to its dispatch loop. The chain then moves between cores with cold caches, and each worker that wakes and finds work requests yet another.

Inspiration from other runtimes

Other runtimes with work-stealing schedulers already handle the "continuation queued by a running task" case specially:

• Go puts a goroutine that was just made runnable into the P's runnext slot rather than its run queue, so it runs next on the same thread and inherits the remaining time slice. Other Ps steal runnext only after it has had a few microseconds to be picked up. wakep only starts a new M when no M is already spinning, and a spinning M that finds work wakes another only if it was the last spinner. sysmon is the backstop for Ps stuck in syscalls or long-running code.
• Tokio (Rust) has a per-worker LIFO slot for the most recently woken task. Pushing into an empty slot does not notify other workers; a notification only happens when an older task is displaced into the run queue.

The .NET thread pool already has per-thread LIFO local queues, but every local enqueue wakes another worker, and nothing protects the newest item from being stolen. The change below adapts the two ideas above, plus the backstops .NET needs because sync-over-async and blocking work items are common.

The change

1. Run-next slot (DOTNET_ThreadPool_RunNext). The most recently queued local work item goes into a per-thread slot. The owning thread takes the slot first when its current work item returns. Other threads can steal it only after it has been seen waiting for a short time (RunNextStealDelayUs, default 5 µs). During a burst only the first item goes into the slot, and the rest go to the normal queue. To keep the clock out of the queuing path, waiting time is measured from when an idle thread first sees the item.

2. Deferred worker requests (DOTNET_ThreadPool_DeferWorkerRequests, requires 1). When a worker queues into an empty slot and another worker is already spinning, no wake-up is requested, similar to Go's spinning-M rule and Tokio's notify-on-displacement rule. This is the part that gives most of the win. Liveness comes from these backstops:

• A spinner watches deferred work. While any thread has an item in its slot, one spinning worker keeps spinning past the usual limit. It requests a worker for any slot item that has waited longer than StalledWorkUs (default 50 µs), and for any work in a queue's array part, which can be stolen right away.
• No lost wake-up between spinner and queuing thread. The queuing thread reads the spinner count after publishing its item with a full fence. The last spinner to stop decrements the count with a full fence and then re-checks for deferred items. So either the queuing thread sees that no worker is spinning and requests one, or the spinner sees the item.
• Blocking hooks. Just before a thread blocks, it marks its slot item as stealable right away and requests a worker. The hooks are in WaitHandle waits, Thread.Sleep(n != 0), and ThreadBlockingInfo.Scope (Lock contention and Monitor/Condition waits). Task.Wait already moves local work to the global queue. The hook does not change the queue itself, because it can run from inside a queue operation: SpinLock contention calls Thread.Sleep(1).
• Gate thread. It checks for stalled local work every 1 ms as a last resort, playing a role similar to Go's sysmon.

3. Lazy Dispatch requests (DOTNET_ThreadPool_LazyDispatchRequest). After its first dequeue, Dispatch only requests another worker if work is visible. Results were mixed (see below), so it is probably not worth keeping.

Results

Windows 11, 20 logical processors (laptop, likely a hybrid core layout), Release build of main at 366f2fd with and without the change. Medians of 3 rounds × 5 runs, as wall ns / CPU ns per operation. The benchmark harness is a small console app and can be shared. Laptop numbers drift ±20% between batches, so only the large differences below should be trusted.

Scenario	base	RunNext	RunNext + Defer	+ Lazy
Raw IThreadPoolWorkItem re-queueing itself, 1 chain	148 / 1,750	108 / 1,125	19 / 39	19 / 39
Same, 4 chains	54 / 977	31 / 422	5.6 / 39	5.4 / 39
TCS ping-pong (RunContinuationsAsynchronously), 1 pair	566 / 2,812	368 / 1,953	279 / 547	279 / 625
TCS ping-pong, 4 pairs	183 / 1,562	116 / 781	93 / 391	139 / 781
TCS ping-pong, 20 pairs	43	45	47	92 / 2,188
Fan-out: 1 producer completes 1,024 TCS, consumers do ~200 ns of work each	297	330	345	350
Channel SPSC (default and sync continuations)	79 / 64	70 / 63	90 / 63	77 / 63
Task.Yield chains (global queue), 1 and 20	222 / 161	224 / 157	225 / 160	223 / 161
TCS ping-pong, inline continuations (reference)	50	48	50	50

Notes:

• Fan-out: a dedicated 3-round rerun put RunNext + Defer even with base. Best runs were about 320–345 ns for every variant, and medians overlapped at 352–450 ns. The gap in the table looks like noise, but it is the first case to recheck on a quieter machine.
• Lazy Dispatch requests help parallel chains but are 2x slower in the saturated 20-pair case.
• What remains of the async ping-pong cost (~279 ns against ~50 ns inline) is mostly the real await suspend and resume, which the inline version avoids because its awaits complete synchronously. The pool's own cost per hop is now about 19 ns, close to the 2 interlocked operations for push and pop.

Latency cost of deferring

These numbers measure how long a continuation waits when the thread that queued it keeps running after queuing it. Median µs per scenario, after the pool has been idle for 100 ms or 3 s:

The queuing thread then…	base	RunNext + Defer
spins on the CPU for 50 ms	6–25	62–156
blocks in native code the runtime doesn't see (P/Invoke Sleep)	6–7	64–75
calls Thread.Sleep	8	10–16
waits on Monitor.Wait	28	16–18
waits on a contended Lock	7–8	17–31
waits on ManualResetEventSlim	18–33	60

This is the main trade-off. A continuation queued by a work item that then runs a long time without blocking now waits for the stall threshold, roughly 50–150 µs, instead of ~10 µs. The threshold can be tuned.

Validation so far

• Stress test: a 120 s mix of fan-out, SemaphoreSlim ping-pong, sync-over-async Task.Wait (including the case where the waited task is taken back out of the local queue and run inline), Parallel.For with nested waits, nested Task.Result, and Task.Yield storms. Every variant passes with 2 instances running at once and a watchdog for hangs.
• Library tests: System.Threading.ThreadPool.Tests, System.Threading.Tests, System.Threading.Thread.Tests, System.Threading.Tasks.Tests, System.Threading.Channels.Tests, System.Threading.Tasks.Parallel.Tests and System.Threading.Timer.Tests all pass, both with the switches off and with RunNext + DeferWorkerRequests on (3,693 tests per run, 0 failures).

Open questions

1. Is a 50–150 µs worst case acceptable for continuations queued by long-running or natively-blocked work items, or does it need a tighter backstop? On Windows the gate thread cannot do better than the ~1–15 ms timer resolution.
2. The blocking hooks cover managed waits only. Sync IO and other P/Invokes rely on the spinner and gate backstops. Are there other choke points worth hooking?
3. Should this be measured with the dotnet/performance thread pool benchmarks and ASP.NET Core plaintext/JSON load tests on Linux before going further? The numbers here all come from a single laptop.
4. Making asynchronous continuations this cheap could let more libraries (Channels, Pipelines, SemaphoreSlim) complete asynchronously by default. Their consumers would then never block producers, at close to inline cost. Is that worth exploring as a follow-up?

Trying it

DOTNET_ThreadPool_RunNext=1
DOTNET_ThreadPool_DeferWorkerRequests=1
# optional tuning
DOTNET_ThreadPool_RunNextStealDelayUs=5
DOTNET_ThreadPool_StalledWorkUs=50

With the switches off, the logic is unchanged apart from a null check on the slot in queue operations, a spinner count in LowLevelLifoSemaphore, and a static bool check in the blocking hooks.