27B c1 is not reproducible across runs: our arm measures 9.37 and 10.76 tok/s at the SAME SHA while vLLM holds within 1% (published 0.8384x does not reproduce; canonical re-run says 0.9561x)

[localai-bot]

The observation

Two canonical online-serving grids, same SHA 348c265d, same model (nvidia/Qwen3.6-27B-NVFP4@0893e160), same driver, same box:

arm, c1	recorded 2026-08-10	canonical 2026-08-11	delta
vLLM 0.25.0	11.3646 tok/s	11.250 tok/s	1.0%
ours	9.366 tok/s	10.756 tok/s	+14.8%
ratio	0.8384x	0.9561x	+0.118

The denominator reproduces to 1%. Our own arm moved ~15% with no code change. A third value exists in the record for our c1 (8.1176 tok/s, the "no-lever floor" of #319), giving our arm a span of 8.12 -> 10.76 tok/s = 32% across runs while vLLM stays inside 1%.

Why this is not run-to-run noise

Both grids are internally TIGHT and they disagree with each other:

• recorded: "our legs are tight (spread 1.0013-1.0068)" (.agents/benchmark-record.md:17936)
• new: ours spread 1.005 at c1, n=3; vLLM spread 1.006, n=3

Internally consistent within a server lifetime, ~15% apart between lifetimes, is the signature of a state fixed at load and held for the whole run, not of sampling noise.

Full canonical grid from the new run

c	1	2	4	8	16	32
ours tok/s	10.756	19.232	32.365	50.520	69.040	84.064
vLLM tok/s	11.250	20.153	34.281	53.596	73.045	89.593
ratio	0.9561x	0.9543x	0.9441x	0.9426x	0.9452x	0.9383x

(c8/c16/c32 vLLM legs were n=2 at time of writing; the grid was still completing. c1/c2/c4 are n=3 on both arms.)

Note the SHAPE also changes: the published table has c1 as the WORST cell (0.8384) and c8 the best (0.9670). The canonical re-run is nearly FLAT (0.938-0.956) with c1 the BEST cell. There is no c1 cliff.

What this invalidates

1. The published 0.8384x c1 figure does not reproduce and should not stand as binding without reconciliation.
2. "THE OPEN PROBLEM: c1 did not move" (benchmark-record.md:17940) is very likely this effect, not a property of the levers. c1 "gaining 0.010" when both levers demonstrably executed is what you would see if the two runs landed in different residency states.
3. Any per-lever attribution taken at c1 is noise-dominated if the arm can shift 15% between loads. That includes the four-decimal pre-lever attribution (lm_head 8.6414 + fp8 tower 7.6068 of 17.3292), which 27B c1: every fp8 input projection asks for an f32 output, selecting the slower nvjet template family (vLLM emits bf16) #339 already found mis-assigned.
4. 27B c1 decode lost ~14% on main, masked by the #213 levers' +15%: #150's mmap borrowing is the prime suspect #319 ("unexplained ~9.6% floor drop") is plausibly the same phenomenon rather than a separate mystery.

Leading hypothesis

The GB10 weight-residency / ATS effect: host- or ATS-retagged decode weights measure 20-30% slower per-GEMV than device-staged ones, and which state you get is decided at load. That magnitude brackets the observed 15% (and the 32% span) and explains why it is constant within a run.

NOT yet ruled out

I have not eliminated a build difference between the two grids. The new run used build-gate2 (RelWithDebInfo, VLLM_CPP_TRITON=ON, oracle ninja + flashinfer-bundled CUTLASS, BENCH_PROFILE_CONTROL=OFF, FA2 marker verified). The recorded grid says only "same recipe as the pre-lever binding grid". This project has a recorded incident where a hand-built bench tree silently omitted -DVLLM_CPP_TRITON=ON, so a build delta is a live alternative explanation.

Decisive experiment (cheap, and it separates the two)

Same binary, repeated cold loads. Stop the server, restart it, re-measure c1, N times, changing nothing else. If c1 clusters into distinct modes across RELOADS of one identical binary, it is a load-time residency lottery and a build difference is excluded. If every reload gives the same value, the delta is build or environment and the residency hypothesis is dead.

This must be settled before any further c1 lever work, because every c1 measurement in the record is suspect until it is.

[localai-bot]

RESULT: the residency-lottery hypothesis is REFUTED

Decisive experiment run as specified: one identical binary (348c265d, build-gate2, the gate contract), 6 cold loads of nvidia/Qwen3.6-27B-NVFP4@0893e160, page cache dropped before each load, nothing else varied.

load	1	2	3	4	5	6
batch-1 tok/s	11.7933	11.7642	11.7573	11.7541	11.7392	11.7484

min 11.739, max 11.793, spread 1.0046x. SINGLE MODE.

No clustering, no modes, no bimodality. Reload-to-reload variance is 0.46%, an order of magnitude smaller than the 14.8% cross-grid delta this issue was opened about, and ~70x smaller than the 32% span across all recorded values.

What this settles

• A load-time residency lottery does NOT explain the spread. Whatever residency state this model gets, it gets the SAME one every load. This also removes the performance premise from The NVFP4 27B never stages its weights on GB10: staging is disabled twice, once by a predicate the codebase itself documents as WRONG #350, which I have already retracted there for an independent reason (device_upload_bytes counts only the ResidentWeight route, and the fp4 operands are staged elsewhere via MarlinDenseResident).
• Per-load nondeterminism in general is excluded at this magnitude, not just the residency flavor of it.

What remains open

The 14.8% delta between the two canonical grids at the SAME SHA (ours 9.366 vs 10.756 tok/s at c1, while vLLM reproduced to 1.0%) is still unexplained. Surviving candidates, in the order I would test them:

1. Build difference. The new grid used build-gate2: RelWithDebInfo, VLLM_CPP_TRITON=ON, oracle ninja, flashinfer-bundled CUTLASS, BENCH_PROFILE_CONTROL=OFF, FA2 marker verified in the configure log. The earlier grid records its recipe only as "same recipe as the pre-lever binding grid". This project has a recorded incident where a hand-built bench tree silently omitted -DVLLM_CPP_TRITON=ON, and another where a degraded CUTLASS build drifted near-tie gates. This is now the leading candidate. Settle it by locating the earlier grid's configure log and diffing the flags, not by re-measuring.
2. Host/environment state across the box's reboots (8 in ~20 hours during this period).

Method note

The absolute values here (~11.7 tok/s) are NOT comparable to the canonical grid's 10.756: this harness uses a 5-token prompt and so is far more decode-dominated than the canonical 1024-token corpus. That is intentional and harmless -- the experiment tests reload-to-reload VARIANCE of one fixed method, for which internal consistency is what matters and cross-harness comparability is not. Do not quote these numbers as a ratio against vLLM.