Turbopack build: whole-app module graph construction is effectively serial and dominates large-app production builds

[ejc3]

On a large Next.js App Router application (2,070 routes, roughly 16,000 first-party TypeScript files, module graph of ~31,000 modules), next build with Turbopack spends 60% of its wall time in a phase that uses almost no parallelism, while the rest of the build parallelizes well.

Environment: Next.js 16.3.3, Node 22, Linux, 166 logical cores, production build, experimental.turbopackFileSystemCacheForBuild at its default. Timings below are from a cold build traced with NEXT_TURBOPACK_TRACING=turbopack and read back through next internal trace / query_spans.

The compile ("Creating an optimized production build" through "Compiled successfully") takes 338.5s. Its breakdown by corrected duration:

• next_api::instrumentation instrumentation endpoint -> next_api::project whole app module graph: 203.6s
  • turbopack_core::module_graph module: 30,696 spans, corrected durations totaling 195.5s (average 6.4ms per module).
• next_api::project emitting: 42.1s
• next_api::app app endpoint HTML: 1,778 spans totaling 39.2s (all pages combined)
• next_api::app app endpoint route: 398 spans totaling 23.2s
• chunking the instrumentation output: 8.9s

The phase is highly concurrent in task terms but starved in CPU terms. Its raw span time is 30,036s inside 203.6s of wall (cpuDuration 3,003,601,571,200 ticks at 100 ticks/us vs correctedDuration 20,361,263,794), i.e. on the order of 150 tasks in flight on average. Yet external process CPU sampling shows the build averaging only ~22 cores over the compile (with peaks near 128 during page compilation), so during graph construction the in-flight tasks appear to spend most of their time waiting rather than executing. Adding hardware does not shorten the phase.

A possible contributor visible in the same trace: allocator pressure. TurboMalloc live-bytes samples on the phase peak at 28.6 GB with maxPressure=2 recorded, and the task engine's eviction threshold scales down under memory pressure (default TURBO_ENGINE_EVICT_MIN_BYTES is 128 MiB). If eviction and recomputation kick in mid-construction, that would convert concurrency into churn. Not yet verified; happy to A/B with the env knobs if that is a plausible mechanism.

Two secondary observations from the same trace, possibly related:

1. Allocator live bytes (TurboMalloc samples in the trace) peak at 28.6 GB during graph construction and stay ~26 GB until project shutdown.
2. The persistent filesystem cache inverts at moderate drift: a warm rebuild after one changed file compiles in ~20s, but after ~370 changed files it takes longer than building with no cache at all (6.4 min vs 4.3-5.1 min cold), with the cache directory growing from 4.4 GB to 6.9 GB. If graph revalidation shares the serial behavior above, that would explain it.

Ask: what bounds throughput of whole-app module graph construction for large graphs (coordination, allocator pressure, lock contention)? The phase sets a hard floor on build time for large apps that no amount of cores changes, and any guidance on knobs or instrumentation to narrow it down would be appreciated. Happy to provide the 6 GB trace file and reproduce with other tracing presets.

[bgw]

We'll need to see the trace to make a proper determination of what's causing the performance problems here. We can handle that over our Slack channel.

There's a good chance that something else is causing your builds to take 338.5s, we rarely see builds that slow when using 30-core machines.

Regarding parallelism

While we haven't observed slower overall wall times due to additional cores, in our testing, we've found diminishing returns after about 16 cores. We haven't done much testing beyond ~30 cores as we have limited access to machines with that many cores, and that's the largest machine size that most of our customers use.

There's a giant DashMap at the core of Turbopack's caching layer, and we can get increased lock contention across shards on extremely large core count machines. Within the team, we have discussed using alternate data structures with more fine-grained locking for these extreme cases, but that would likely increase memory consumption and make performance worse for the average user with a more moderate number of cores. This remains an open area of research for us.

There are also just some steps in the build that aren't as trivially parallelized as others. We've fixed many of these, but we might be able to find stuff in your trace.

Eviction

the task engine's eviction threshold scales down under memory pressure (default TURBO_ENGINE_EVICT_MIN_BYTES is 128 MiB). If eviction and recomputation kick in mid-construction, that would convert concurrency into churn.

We run eviction during next dev, not next build, and we try to run it when we're otherwise idle. It sounds like your issue is with next build so this doesn't seem relevant here.

A couple things for you to check

• We have an environment variable that can limit the number of tokio worker threads that we create. In the extreme case of 166 cores, Turbopack might actually perform better if you lower this by setting TURBO_TASKS_AVAILABLE_PARALLELISM to something like 64 or 32.
  • Alternatively, you could more tightly enforce this at the OS level with something like https://man7.org/linux/man-pages/man1/taskset.1.html (I'm not very familiar with this stuff though)
• We suspect that we're highly sensitive to memory latency. Does this 166 core machine have multiple NUMA nodes?
  • Would you be able to test disabling scheduling across NUMA nodes with something like https://www.man7.org/linux/man-pages/man8/numactl.8.html? (Also not very familiar with this)
• If this is a dedicated CI machine, try seeing if there's a smaller machine type that you can use. Turbopack would likely benefit from fewer, but faster cores, than many slower cores.

It's also probably possible to set some/all of these policies by running under systemd-run, e.g.:

sudo systemd-run -p AllowedCPUs=... -p NUMAPolicy=... -p NUMAMask=...

[ejc3]

Follow-up with discriminating measurements on the module-graph phase (same app as above: ~2,070 routes, ~30k modules; 166 logical cores, Linux).

We ran cold builds varying one knob at a time:

arm	compile
defaults	3.8 min
TURBO_ENGINE_READ_CONCURRENCY=512	3.9 min
TURBO_ENGINE_READ_CONCURRENCY=8	4.0 min
TURBO_TASKS_AVAILABLE_PARALLELISM=32	4.1 min

So the disk-read semaphore is not limiting (64→512 no change; even 8 barely hurts), and dropping to 32 workers (1/5 of cores) costs only ~8% — the phase is not CPU-parallel-bound.

strace -c -f over a 60 s window inside the graph phase:

 99.45%  7341.73 s   3,145,772 calls   futex
  0.14%    10.48 s     118,420 calls   sched_yield
  0.00%              12 calls          openat

~7,342 thread-seconds of futex wait in a 60 s window across ~166 worker threads ≈ ~120 workers asleep on locks at any instant, which matches the observed ~20 busy cores. Negligible file IO.

Reading the v16.3.3 sources, this looks consistent with synchronous lock waits inside task execution rather than IO or compute:

• task-state accesses take a synchronous DashMap shard write lock even for reads (turbo-tasks-backend/src/backend/mod.rs ~:442-464, incl. the TODO(sokra) about the mutable-read double lock), and a worker blocked synchronously never returns Poll::Pending, so PriorityRunner never releases its slot or spawns a replacement — which also explains why the task trace reports concurrency ≈ target_workers while the OS sees ~20 busy cores;
• the sched_yield volume is consistent with AggregationUpdateQueue's retry_loop busy-wait (backend/operation/aggregation_update.rs, yield_now up to 60k retries);
• the whole-app module graph entry point is a root aggregation function, so activeness/dirty counter propagation from the full ~30k-module subgraph funnels toward a small set of upper tasks, concentrating writers on hot shards.

Happy to run further instrumented builds (env-var arms, off-CPU stack samples) if useful — the 6 GB task trace offer from the issue body still stands.

[ejc3]

Results on your suggestions, plus a first pass over the trace ourselves.

Machine note first: the parallelism arms below ran on a single-socket AMD EPYC 9D64 (88 cores / 176 threads), one NUMA node — so cross-node memory latency is ruled out on this box. (The original trace came from a similar-class single-socket host; I'll state topology alongside any future numbers.)

TURBO_TASKS_AVAILABLE_PARALLELISM sweep (cold builds, one change per arm):

arm	compile
default (176 threads)	3.8 min
taskset to 88 physical cores	4.1 min
64	4.1 min
32	4.1 min
16	4.3 min

Monotonically worse as parallelism drops — so on this workload, reducing workers doesn't help, which argues against raw lock-throughput being the primary limiter (fewer contenders would have won). Together with the flat TURBO_ENGINE_READ_CONCURRENCY arms, our current read: the module-graph phase is runnable-frontier-bound — effective parallelism ~15–22 regardless of resources — and the futex volume is mostly idle workers with nothing runnable (the 118k sched_yields in 60 s still look like the aggregation retry loop, but as a symptom, not the cap).

From the trace (whole app module graph, 203.6 s corrected): 195.5 s is 30,696 recursive module_graph module spans — each stays open until its entire import subtree completes (avg ~1 s open, ~6 ms exclusive each), so open-span counts overstate concurrency by design. The real work spans underneath (turbopack process module, parse ecmascript) show high internal concurrency (e.g. 87 s CPU in 1.0 s corrected for one process-module aggregate) — when work is runnable it parallelizes fine; there just isn't a wide runnable set for most of the 195 s.

Next we'll get an on-CPU profile of the phase to see what the ~20 busy cores are executing (parse/resolve vs task-engine bookkeeping). And yes on the trace handoff via Slack — it's packaged.

[ejc3]

Closing the loop on the promised on-CPU look, plus the synthesis of all the arms.

Thread-level sampling during the graph phase (perf tracepoints weren't available on this host, so: per-thread CPU-time deltas from /proc/<pid>/task/*/stat at 2 s intervals, stacks via a thread dumper): in every sample window only ~8 tokio worker threads accrued meaningful CPU — consistent with the OS-level ~20-busy-cores reading and the worker-sweep insensitivity. The busy threads' stacks were dominated by the allocator (calloc / _int_malloc) and raw syscall frames; one caveat, the Rust frames above libc didn't symbolize with the tooling available here, so I can't name the calling functions from stacks — but the trace's memory track independently shows the phase allocating +25.5 GB net (851 MB → 26.3 GB live), so allocation-heavy bookkeeping is measured, not inferred.

The arithmetic that summarizes everything we measured: the phase contains 30,696 module tasks; each averages ~6.4 ms of exclusive time but stays open ~1 s — roughly a 150:1 ratio of engine/wait time to work per module. Summing the actual work spans (parse, process-module) gives a few hundred CPU-seconds for the whole app — i.e. tens of seconds at even modest parallelism — while the wall is 200+ s. And when work is runnable it runs wide (we saw ~87x inside process-module aggregates). So the picture is: plenty of parallel capacity, plenty of parallelizable work, but a narrow runnable frontier most of the time, with the busy threads spending their cycles in allocation/bookkeeping rather than compilation.

Combined with the earlier arms (read-semaphore 8/64/512 flat; workers 16→176 within ~13% and fewer is worse; eviction and NUMA ruled out), the candidates that remain are the structural ones you alluded to: the single-driver graph traversal draining discovery bursts serially (e.g. a ~1,700-import barrel file releasing its children all at once), write-locked task-state touches even for reads, and counter propagation toward the root aggregation. Happy to run any discriminating experiment you want on the 176-thread host — and the full trace remains available through the private channel.

[ejc3]

We built the v16.3.3 napi bindings from source and A/B'd three engine-level patches against an unmodified from-source control on the same 176-thread host, same ~2,080-route app, two cold builds per arm (__INTERNAL_CUSTOM_TURBOPACK_BINDINGS to swap binaries; run-to-run noise ±15-20 s):

arm	compile (2 runs)	verdict
stock npm binary	4.3 / 3.9 min	baseline
from-source, unpatched	3.7 / 4.1 min	control valid (within noise of stock)
LD_PRELOAD mimalloc (override, whole process)	4.0 / 3.8 min	null — C-side allocator not a factor (Rust side already mimalloc)
LEAF_NUMBER: 16 → 1<<24 (no intermediate aggregating nodes)	12.8 / 13.0 min	3.2x slower, output correct — the aggregation tree is load-bearing compression for activeness/dirty propagation, not deletable overhead
traversal-driver batching (drain all ready completions per wakeup via now_or_never before re-awaiting)	4.1 / 4.1 min	null — per-completion executor round-trips in the single-threaded walk driver are not the cap

Combined with the earlier knob sweeps (workers 16→176 within ~13%, read-concurrency flat, NUMA n/a single-node), everything cheap is now exonerated: allocator, lock pressure relief via fewer workers, driver wakeup overhead, and aggregation removal (which is strongly counterproductive). What remains consistent with all measurements is per-task engine latency times dependency-chain depth: one turbo-task per module edge, each paying the fixed schedule/execute/complete/notify cost, with the runnable frontier bounded by the import graph's structure rather than by any resource. If that's right, the wins are structural — fewer, coarser tasks per module (we notice a next-taskless crate in the tree that suggests you're already thinking this way) or speculative/breadth-first discovery that widens the frontier ahead of resolution.

Happy to run any candidate patch you want measured on this host — the from-source A/B loop is now ~20 minutes per arm.

[ejc3]

We instrumented the v16.3.3 engine (relaxed-atomic counters at four lifecycle points, from-source build, __INTERNAL_CUSTOM_TURBOPACK_BINDINGS) and then patched what the numbers indicted. Same app (~2,080 routes, ~30.7k modules), same 176-thread single-NUMA host as our earlier reports.

Where a 270 s compile's thread-time goes (176 threads = 47,520 thread-seconds of capacity; instrumented build ~10-15% slower than stock, segments overlap — aggregation includes its nested lock waits):

segment	thread-seconds	count	avg
PriorityRunner::schedule (one global Mutex<BinaryHeap>)	9,255	22.8 M	406 µs
aggregation queue processing (inline in tasks)	7,844	319.6 M	24.5 µs
task shard-lock wait, single (access_mut)	5,017	999.2 M	5.0 µs
task shard-lock wait, pair	901	176.5 M	5.1 µs
event notify	40	30.7 M	1.3 µs

Volume context: 8.59 M task executions, 94.3 M output/cell reads, 59.2 M connect_child operations — i.e. ~280 tasks, ~33,000 task-lock acquisitions, and ~1,930 child connections per module. Per-op costs are healthy; the op counts are the story.

Patch experiment: shard the scheduler queue (64 sub-heaps, approximate-emptiness counter, try_lock scan + blocking fallback; global priority order relaxed to per-shard — patch attached, 131 lines):

• Wall: 3.5 / 3.5 min vs 3.7 / 4.1 min control (~10%, the most repeatable pair we've measured; output identical, peak RSS +~2 GB).
• Mechanism confirmed by re-instrumenting the patched binding: schedule fell 9,255 → 2,493 thread-s (406 → 109 µs/op) — but single-task lock waits doubled (5,017 → 10,405 thread-s) and pair waits doubled too. The freed threads immediately re-queued on the next lock in the pipeline; total engine-segment time barely moved.

Earlier same-rig arms for completeness: LD_PRELOAD mimalloc = null; traversal-driver batching = null; LEAF_NUMBER 16 → 1<<24 = 3.2x slower with correct output (the aggregation tree is load-bearing compression, not removable overhead).

Our takeaway: the engine is convoy-limited at whichever lock is next in line — relieving one moves the wait, so micro-fixes cap out around ~10% on this shape. The op-count-per-module (280 tasks/module) is the structural lever; that matches where next-taskless seems headed. The scheduler shard is still probably worth having on high-core hosts (it's small, and 406 µs per schedule on a global mutex won't age well as core counts grow) — happy to clean it up as a PR if you want it, and to run any other candidate on this rig (~20 min per arm).

[lukesandberg]

Thanks, we have some work in flight to improve the semantics of PriorityRunner (e.g. #97390 that should land soon)

lock contention has been a target in the past but we agree with the basic conclusion, reducing per module tasks count is the better approach.

One interesting experiment to try is disabling the filesystem cache. Proper functioning of the task does require slightly more datastructures to be updated (more lock acquisitions), so experimental.turbopackFileSystemCacheForBuild=false might be a win which would be an interesting datapoint.

From examining the trace file there is a lot of unaccounted for time when processing modules which could be lock contention but the other detail here is 'module context explosion' according to your count there are 30k modules but per our count we process 100K modules. Some of this is caused by referencing large dependency trees from multiple context (SSR and client). e.g. the nav_manifest module is processed twice along with all of its dependencies. Part of this is our bug to fix for sure, but you might want to see if a simple "use client" on the nav_manifest (or one of its importers) makes sense for your application.

[ejc3]

One correction to our last two comments: we cited next-taskless as a sign you're moving toward coarser task granularity — that was inference from the crate name, and having now read it, it's wrong: it's sans-io utilities shared with the webpack/rspack paths, unrelated to engine granularity. Apologies for the noise. The op-count observation stands on the measurements themselves (~280 task executions / ~33k task-lock acquisitions per module, with per-op costs healthy), independent of any guess about your roadmap.

One related datum from the same counters that's worth having when weighing granularity changes: reads outnumber task executions ~11:1 (94.3 M reads vs 8.59 M executions), so the task cache is doing real dedup work — the win isn't fewer cached nodes, it's fusing intra-module chains (module → parse → analyze → per-reference resolution) so the cache boundary sits at module granularity rather than per-step.

[ejc3]

Update, and it starts with a correction to my own framing.

Everything we have reported in this thread was measured on 16.3.3. Going to open PRs against canary, I found #96808 ("execute scheduled tasks inline when they are read", merged 2026-08-24), whose description says it avoids "a futile acquisition of the scheduler's queue lock, which is otherwise the most contended lock in the system." That is the same conclusion our counters reached, arrived at independently and already shipped. v16.3.3 diverged before it (compare says diverged, 344 behind), so none of our numbers include it and canary does.

So I am not going to post a "your scheduler is the bottleneck" patch, because you already know and have acted. What follows is: the one change that is still orthogonal to that fix and is open as a PR, an offer to measure your fix at a scale you have said you cannot reach, and the findings that do not depend on the scheduler-queue measurement. Same app and host as every comment above (~2,080 routes, ~30.7k modules, 88c/176t single-NUMA EPYC, production build), each arm against an unpatched from-source control on the same machine in the same session.

An offer: we can measure #96808 at 176 threads

@bgw said the team has limited access to machines beyond ~30 cores. We have a 176-thread host, a real 2,000-route app, a scripted from-source bring-up, and a paired A/B harness that swaps engine binaries with __INTERNAL_CUSTOM_TURBOPACK_BINDINGS. We would like to run canary against 16.3.3 on it and report what #96808 is worth at that scale, with the same counter instrumentation we used above, and post the numbers here whatever they say. If there is a specific arm you want (the inline_execution_stats feature counters, a particular commit pair, an env-knob sweep), name it and we will run that instead of guessing.

One PR: remove a task boundary that never caches

#98091 (draft)

ModuleAssetContext::process_resolve_result executes ~223k times per build with a near-zero cache-hit rate, so it pays task creation, cell allocation, dependency edges and completion bookkeeping for values nothing reads twice. The PR calls the body directly from the internal call site that holds a concrete context, keeps the trait method as a wrapper, and leaves the inner process() boundary (the one that actually deduplicates module creation) untouched. Measured 279 s against a 298 s control, two runs each, on 16.3.3; run-to-run noise on this host is +/-15-20 s, so treat the mechanism as the claim and the 6% as soft.

How it relates to #96808: that change makes a queued-then-read task execute on the reading thread instead of taking the queue detour, but the task still exists, still allocates, still records edges. This removes the task. Whether they compose on canary is unmeasured, and we would rather establish that in the PR than assert it here.

The instructive part is the failure that set our gate. We tried the same move on a second function and it measured 304 s against the same control, a clear regression: that function's cache was hitting and inlining re-did shared work. A cached function can be deduplicating across many inputs even with one caller in the source, so call-site counting is not a safe signal. Our rule became one call site and a measured near-zero hit rate; a third candidate chosen that way removed its 123k executions and moved the wall clock by nothing, because those calls were never on the critical path. Two of three "obviously fusable" candidates were not worth doing and one was harmful.

What we are holding back, and why

We also have a diff that shards the PriorityRunner queue into 64 independently-locked heaps. On 16.3.3 it measured ~10% on its own and scheduler thread-time fell by roughly three quarters. We are not opening it, for two reasons, and the second is the real one:

1. It deliberately fails your two priority_runner tests that assert strict highest-priority-first execution, because relaxing that global order is where the win comes from. We built the order-preserving variant (pop scans all 64 shard heads for the global maximum); it passes both tests and costs 42 s, roughly four times the win. So the win largely is the relaxation, which is a policy question, not a bug.
2. It was measured against the pre-#96808 queue. Posting a 10% number against a lock you have since taken pressure off would be misleading. If our canary measurement shows the scheduler still dominating at 176 threads, we will bring the diff and the policy question back with current numbers. If it shows #96808 closed it, we will say that here.

One smaller thing, unrelated to the queue, and I want to correct myself on it since I had it wrong when I first drafted this: the Node worker pool's bootup_semaphore is not a permanent serialization. It starts at one permit and gains one per completed boot, so concurrency ramps 1, 2, 4 and so on. That is deliberate, from vercel/turborepo#7661, and it has a measured payoff on a 16-core machine (16 process startups totalling 9.09 s down to 7 totalling 1.26 s). What we measured is that on a 176-thread host the ramp appears to cost more than it saves: starting the semaphore at 64 instead engaged 28 workers and gave 286.5 s against a 303/297 s control. We are not proposing a change to a tuned heuristic on the strength of one host, but if the slow-start is worth revisiting for very wide machines we are happy to run whatever arm you want with the counters attached.

React Compiler is the largest single line item, and the Rust port measured slower

Unaffected by any of the above. On this app it runs 10,862 times per build for 2,515 thread-seconds of transform across the app's own 9,773 files. Libraries and node_modules are already excluded by the existing gates, so this is all first-party source. The hot path is the inference pass's lattice merge and clone.

• experimental.turbopackRustReactCompiler was slower than the Babel path here: 346 +/- 22 s against ~300 s, three runs each. Same algorithm; the difference looks like placement, in that the Rust path runs inline inside each per-file task and loses the overlap the Babel worker pool gets. Worth flagging because "port it to Rust" reads as an automatic win and on this app it was not.
• compilationMode: 'annotation' is a real lever: 234.5 s against 300 s stock, validated in both directions (an un-annotated file compiles zero times; one annotated 116 KB client page compiles exactly once, 3,409 ms).

The persistent cache inverts, and that is a docs problem

The original post mentioned this in passing; here is the controlled version, because it changed our own plans. Same app, one machine, one day, 16.3.3.

cache state	wall	tasks re-executed
cold, no cache	281 s	8,609,739
fresh, same revision	44 s	0
2 days stale, window contains a dependency bump	301 s / 299 s	8,609,739
1 day stale, source churn only, 290 files	252 s	135,030 (1.6%)
2 days stale, source churn only, 319 files	377 s	1,602,768 (18.6%)

1. In these runs, a cache that crossed a dependency change was slower than no cache. It re-executed every task, the same 8,609,739 as a cold build to the digit, then paid restore, dependency tracking and rewrite on top of what was effectively a cold build. A framework bump makes it certain: the cache directory is keyed by release (turbopack/v16.3.3-<hash>/), so a bumped binary starts a fresh keyspace and the restored gigabytes buy nothing.
2. With manifests matching, file count predicts nothing; import reach does. Two windows of nearly identical size landed at 1.6% and 18.6% invalidation. The first beat cold by 29 s. The second lost to cold by 96 s. The difference is which files changed: the slow window's set includes shared modules whose importers all invalidate with them.
3. Synthetic churn will tell you the cache is always fine. Appending a comment line to 600 and then 1,200 real .tsx files re-executed only ~2 tasks per file and stayed far under cold, because the compiler re-parses each changed file, finds its meaning unchanged, and stops. Any benchmark built on synthetic edits misses the inversion entirely.

Practically this means "turn the filesystem cache on in CI" is not safe advice, which is not what we expected to conclude. A cache is reliably a win when it is near (same revision, or one change away). Past that it is a function of the invalidated fraction, and we could not find a cheap way to read that fraction before committing to a restore. The only pre-build signal we found that reliably predicts total loss is a dependency-manifest mismatch (package.json plus lockfile, byte-compared), so we now wipe when they differ. Necessary and clearly not sufficient: the 18.6% window had matching manifests and still lost.

Three questions:

• Is there a supported way to ask a persisted cache how much of it a given source state invalidates, short of restoring it and building?
• Could the engine short-circuit the "restore, discover it is all invalid, pay for the rewrite anyway" path once invalidation passes some fraction?
• Is the release-keyed cache directory intended to make cross-version reuse impossible, or is a migration planned? Given how often a framework bump lands in a normal week, that case alone may deserve an explicit skip-the-restore rule.

Independent of the answers, we think the docs should say plainly that a sufficiently stale cache is slower than no cache, because the failure is silent: the build still succeeds, it is just slower than the build you were trying to speed up.

One app, one machine, one release behind

One app, one machine, one week, and now one release behind. Our confidence is uneven on purpose. The mechanisms we are confident about because they are direct measurements with controls: the counter breakdown, the two failing priority tests, the invalidation fractions, the Rust-compiler comparison. The magnitudes we are much less confident about transferring, and the scheduler magnitude specifically is now known to be measured against superseded code. If any of this contradicts your own benchmarks, we would rather hear it than have it adopted quietly.

[ejc3]

We rebuilt everything on canary rather than keep hedging about 16.3.3, and it changed three conclusions, including one of ours that does not survive. Same app and host as above, plus two smaller apps; every arm below is stock-vs-patched on the same box, interleaved, with the loaded engine verified by matching the sha256 of the .node in /proc/<pid>/maps against the arm's binary. Runs that could not prove which binary they loaded were discarded rather than reported, which cost us ten rows and one earlier claim.

#96808 flips sign with core count

This is the measurement we offered, and it is the one we would most like a second opinion on. Parent is the commit immediately before #96808, child is #96808 itself, same app, same host, nothing else different:

cores	parent	child (with #96808)	delta	ranges
16	528 / 531 / 537	518 / 519	child 14 s faster	separate
30	395 / 400 / 403	380 / 396	child 11 s faster	overlapping
166	281 / 301 / 306 / 307 / 312	319 / 322 / 324 / 326	child 21 s slower	separate

task_miss is identical across arms at every point, so nothing about the work changed; this is purely scheduling. At 16 and 30 cores the change does what it says. At 166 threads every child run is slower than every parent run, about 7%. We are not claiming to know why, and we would rather be told we are holding it wrong than have this taken as a regression report. If you want a specific arm re-run, or the inline_execution_stats counters against it, say which and we will.

Two corrections we owe you about the queue sharding

In our last comment we said the sharding patch deliberately fails two of your priority_runner tests, and that "the win largely is the relaxation" of global priority order. Having ported it to canary's restructured Queue, both statements are wrong.

The port aims at global-max ordering by caching each shard's top priority in an atomic, and it passes both ordering tests:

cargo test --offline -p turbo-tasks --lib priority_runner
test priority_runner::tests::test_cpu_bound_tasks ... ok
test priority_runner::tests::test_cpu_bound_with_yield_tasks ... ok
test priority_runner::tests::test_mixed_cpu_bound_and_waiting_tasks ... ok
test result: ok. 12 passed; 0 failed; 0 ignored

Each ordering test also passes run individually with --exact, and the whole crate matches a stock control. Correcting one thing here: passing those tests is not the same as preserving the ordering, and a code review of our port found that it does not. pop picks a shard from the cached tops and then pops it without revalidating, so two poppers that select the same shard, or a stale-high top left behind by claim, can return a lower-priority item while a higher one sits in another shard. The existing tests do not cover it because they cannot reliably produce that interleaving. We are fixing pop to revalidate after taking the shard lock and will re-measure, since that costs an extra peek and an occasional rescan. So the contract question we put to you last time is still open rather than answered.

The sharding win is real and it is a wide-machine effect only

cores	canary stock	canary + sharding	delta	noise	ranges
166	317 / 319 / 320 / 321 / 325	274 / 277	44.9 s faster (14.0%)	8 s	separate
30	375 / 377 / 385 / 388 / 390 / 393	376 / 382 / 392	1.3 s	13 s	overlapping

task_miss and page count are identical in every run, so again this is scheduling only. The 16-core cell is still running and we will post it whatever it says.

The mechanism, with both engines instrumented identically on one host at 166 threads:

arm	queue-lock wait	acquisitions	contended	rate
stock	24,898 thread-s	67,840,156	36,515,476	53.8%
sharded	5,436 thread-s	56,007,229	11,324,956	20.2%

4.58x less blocked time. That also explains the 30-core row rather than contradicting it: at 166 threads more than half of the acquisitions are contended and the lock is genuinely the constraint, and at 30 cores there is not enough concurrency for it to bind. So the honest description is not "makes builds 14% faster", it is "removes a bottleneck that only binds above some core count, somewhere between 30 and 166". Taken with the #96808 table above, the two changes have opposite signs at the two ends, which suggests the scheduler wants different behaviour at 30 cores and at 166 rather than one setting for both.

Our own patch did not survive the move to canary

The task-fusion change in #98091 is a wall-clock null on canary. Nine cells, three apps at 16, 30 and 166 cores: eight are inside that configuration's own noise, and the only mover is the smallest app at 30 cores. The mechanism is exactly as described, and reproducible: task_miss drops 2.6% to 3.7% in every cell, which is the ~223k task executions. It just does not show up in wall time.

Two things about that we should say plainly. The 279 vs 298 we quoted earlier was measured with the filesystem cache off, while these are default-config cold builds, so it was never comparable to what we are reporting now. And we are retracting one number from our last comment: a 4.5 s improvement on the mid-size app at 166 cores came from rows that had not proved which binary they loaded; hash-verified, the same cell re-measures at 1.0 s, inside noise. The PR stays in draft while we measure whether removing those tasks removes their memory, since the original post in this thread flagged allocator pressure and 28.6 GB of live bytes as a suspect. If that is flat too, we will close it and say so here.

[ejc3]

Corrections to my last two comments, both from measuring rather than from argument.

The sharding number was wrong, and so was the ordering claim

A code review of our port found that pop selected a shard from the cached tops and then popped it without revalidating, so a tombstone left by claim, or two poppers picking the same shard, could return a lower-priority item while a higher one waited elsewhere. That is now fixed by re-reading the shard's real top after locking and rescanning when the hint was stale, with a test that fails without it. The suite passed before the fix because its keys happen to land in ten distinct shards, so no existing test could construct the case.

Fixing it costs most of the win:

	166 cores
sharding as first reported (ordering defect)	44.6 s, 13.9%
sharding with ordering actually preserved	18.1 s, 5.7%

Non-overlapping ranges, n=7 stock and n=5 patched, task_miss identical at 8,635,707 in every run. Still nothing at 16 or 30 cores. 44.9 s is withdrawn.

The lock table we posted is withdrawn too, because it was instrumented on the defective engine. Measured on the engine that matches the 18.1 s:

metric	stock	sharded (fixed)
queue-lock wait	24,997 thread-s	10,909 thread-s
acquisitions	67,346,090	79,100,695
contention rate	56.6%	15.3%

2.29x less blocked time, not the 4.58x we reported, and acquisitions rise 17.5% because a revalidating pop that finds a stale hint drops the lock and comes back. The cost of correctness is large and worth stating: 49.7% of pops find their hint stale and rescan, and 14.3% fall through to a blocking all-shards pass. At that rate it is fair to ask whether the per-shard hint earns its place; we do not have a better answer than the measurement.

Correcting the paragraph that was here: I wrote that roughly 60% of the win disappears when ordering is genuinely preserved, and put the strict-versus-relaxed policy question back to you. That was wrong, and measuring it rather than reasoning about it is what showed the error. We built a deliberately relaxed variant on the fixed tree, identical except that pop trusts the cached hint instead of revalidating, and ran all three arms interleaved:

arm	runs	mean
stock	319 / 315 / 318	317.3 s
strict (revalidating)	304 / 291 / 308	301.0 s
relaxed (hint trusted)	298 / 304 / 304	302.0 s

Relaxing the ordering buys nothing: 1.0 s in favour of strict, with the ranges overlapping. The relaxed arm fails only test_claim_tombstone_does_not_break_global_order and passes the other 111, so it is a clean policy variant that surrenders one named property for no speed. There is no trade-off here to weigh, so please ignore the question we asked. Revalidation costs 49.7% rescans and 17.5% more lock acquisitions and still comes out level, because the extra acquisitions are uncontended re-takes of an already-hot lock.

What we cannot yet explain is the rest of the gap. The defective engine measured 44.6 s faster than stock and the corrected one measures 16 to 18 s, and since revalidation is not the cause, something else in the correctness fixes is. We are bisecting that now and will report it, including if the answer is that our own fix is simply leaving performance on the table.

Updated PR: #98093.

We closed our own other PR

#98091 (task fusion) is closed. On canary it is a null on wall clock (8 of 9 app-and-core-count cells inside noise), on peak RSS, and on TurboMalloc peak live bytes, which was the tighter instrument at a 0.10 GB stock spread and showed no reading lower than stock. It reliably removes ~223k task executions per build and that converts into nothing measurable, which is itself the useful finding: task count on its own is not a good optimization target here.