v0.3.3: the October 12 vLLM nightly pin, four-card decode step 15.0 -> 14.3 ms, BetterBench 177 -> 188

The image pin moves to the 2026-10-12 vLLM ROCm 10 nightly: Flash-Next on four cards decodes a step in 14.3 ms
(v0.3.2: 15.0), 225 tok/s single-stream, with the same kernels, the same quality gate and the same KV cache.
The code of v0.3.2 plus the pin, the October compatibility fixes it rides on (notes/vllm-upgrade-2026-10.md), and
round 4 of notes/decode-step.md.

What changed for someone running it

• docker/Dockerfile builds on vllm/vllm-openai-rocm:nightly-rocm100-fe4df3df... (vLLM 0.31.1rc1.dev296,
  torch 2.12+rocm10.0) instead of the September nightly. Rebuild the image; on a host that needs the
  emulated-switch overlay, regenerate its patched pair for this image (overlay/emulated-switch/patch-hostcall.sh)
  and point ~/p2p-patched-nightly at it.
• The upstream DFlash draft-model exclusion bug (vllm#61003)
  is still present in this nightly; the plugin's alias workaround covers it (the 27B's drafter runs at full
  acceptance).
• R9K_MOE_ROUTE=r9k / R9K_MOE_FOLD=r9k (off since v0.3.0) do not start on this vLLM: the runner's forward
  changed shape. They stay off and were not re-ported (worth 0.13 ms a step on the old one).
• New in the tree, off: the fused MoE decode path (R9K_MOE_FUSE=1: silu(gate) * up in the gate_up GEMM's
  epilogue, the fp8 row quant inside the down GEMM's A-load; bit-identical outputs, tests/test_moe_fused.py).
  It measured the same step time as the three-launch path, so it ships switched off.
• A cold torch.compile cache on a new image makes the first launch's 8-stream numbers read 20% low (compile
  transients during the first requests of that batch shape); the second launch is representative. The validation
  below warms the cache first.

Numbers (test box, 4x R9700, 210 W and -42 mV)

Flash-Next TP4, warm cache	v0.3.2 (Sept nightly)	v0.3.3 (Oct 12 nightly)
decode step, 512-token essay	15.0 ms	14.3
single-stream decode (probe)	219 tok/s	225
BetterBench combined decode / 1 / 2 / 4 / 8 streams	176.8 / 172 / 261 / 403 / 514	188.0 / 175 / 277 / 410 / 530
8 / 16 streams (concurrent bursts, draft counters)	617-623 / 841	608-622 / 868
8k prefill	6,729-6,753	6,723-6,780
KV cache	515,577 tokens	513,088

Validation record

Every configuration on this code, on the pinned image, 2026-10-13: Flash-Next TP4 full BetterBench (combined 188.0, concurrency 175 / 277 / 410 / 530, KV 518,481 tokens), 25-min mixed soaks on TP4, the 27B and Flash-Next TP2 (690 / 290 / 244 requests, 0 errors), strict sanity under overload, 4-rank all-reduce / all-gather and 2-rank race tests, the single-GPU kernel gates; no MES timeouts on the pinned image. The full record is in notes/release-v0.3.3.md.

v0.3.2: KV cache +14% on four cards and +33% on two, 8 streams +6%, same quality

The hyper-connection weights run from their fp8 copies at every width and the bf16 copies are gone: KV cache
+14% on four cards (452k -> 516k tokens) and +33% on two (167k -> 221k), 8 streams +5.6% and 16 streams +3.4% at
TP4, same quality, single-stream and prefill unchanged within 1%. The code of v0.3.1 plus round 3 of
notes/decode-step.md.

What changed for someone running it

• serve/flashnext.sh at TP2 and TP4: the fp8 hyper-connection kernels (kernels/r9k_hc_f8.hip) now take any
  row count and are used to 255 rows (R9K_HC_FP8_DECODE_MAXM=255); above that the tiled fp8 GEMMs run both the
  up mix (as before) and the down GEMM (R9K_HC_FP8_DOWN=1, -0.8% at 8k prefill for its activation quant); with
  every width on fp8, the bf16 up / down weights are freed after quantisation (R9K_HC_FREE_BF16=1: 1.3 GB a
  card). The fp8 decode path is on at TP2 as well now: the freed pair outweighs the down copy that kept it off in
  v0.3.1. R9K_HC_FREE_BF16=0 restores the v0.3.1 memory layout, R9K_HC_FP8_DECODE_MAXM=16 its dispatch.
• The 27B is unchanged (no hyper-connections).

Numbers (test box, 210 W and -42 mV, probe.py, warm compile cache)

Flash-Next TP4	v0.3.1	v0.3.2
KV cache	452,115 tokens	515,577
single-stream decode	217-220 tok/s	219
8 / 16 streams	550-560 / 904-907	582 / 935
8k prefill	6,785-6,883	6,729
390-token prefill	1,660-1,760	1,691

Flash-Next TP2, experts in host RAM	v0.3.1	v0.3.2
KV cache	166,818 tokens	221,184
single-stream decode	120.1 tok/s	122.5
8 / 16 streams	185 / 153 (+-6% run to run)	174 / 163
8k prefill	2,589-2,603	2,594

Quality gate (2026-10-10, conc 1, both configs on this tree)

	GSM8K 1,319	HumanEval	strict sanity
v0.3.1 defaults	95.83% (1,264)	159 / 164	0 bad of 1,040
v0.3.2 defaults	95.75% (1,263)	162 / 164	0 bad of 1,040

Paired per question: 11 vs 10 discordant, McNemar p = 1.00; outputs identical on 18.5% of questions. No detectable
difference.

Validation record

Every configuration on this code, 2026-10-10: Flash-Next TP4 full BetterBench (combined 176.8, concurrency 172 / 261 / 403 / 514, KV 515,577 tokens), 25-min mixed soaks on TP4, the 27B and Flash-Next TP2 (792 / 292 / 244 requests, 0 errors), strict sanity under overload, 4-rank all-reduce / all-gather and 2-rank race tests, the single-GPU kernel gates -- the full record is in notes/release-v0.3.2.md.

v0.3.1: decode step 16.4 -> 15.0 ms on four cards, same quality, same KV, same prefill

Flash-Next on four cards: the decode step 16.4 -> 15.0 ms, single-stream decode 199 -> 217 tok/s, same
quality, same KV cache, same prefill. The code of v0.3.0 plus the decode-step round of 2026-10-09
(notes/decode-step.md) and the 2026-10 vLLM nightly compatibility fixes (notes/vllm-upgrade-2026-10.md).

What changed for someone running it

• serve/flashnext.sh turns on two new paths by default (the first at TP4 only):
  • fp8 hyper-connection decode GEMMs (R9K_HC_FP8_DECODE=1, kernels/r9k_hc_f8.hip): the down+hc_silu and
    up+gated-mean GEMMs at decode widths (up to 16 tokens) read the fp8 fragment-order copies the prefill mix path
    already holds, half the bytes of the bf16 kernels that were the step's largest stream (1.29 GB a step), with
    bf16 WMMA against the unquantized activations. 16.42 -> 15.33 ms a step. =0 restores the bf16 kernels. At
    TP2 (experts in host RAM) it stays off: the fp8 down copy costs 30k KV tokens there (133,306 -> 103,517) for
    +10% single-stream decode and nothing at 8 streams; R9K_HC_FP8_DECODE=1 opts in.
  • fused activation quant in the dense fp8 block GEMM (R9K_FP8_QA=1, r9k_gemm_fp8_block_qa): the
    per-token-group-128 quant runs inside the GEMM's A-load with bit-identical operands; 96 launches fewer a step.
    15.33 -> 15.01 ms. =0 splits it again.
• The plugin also runs on the 2026-10-09 vLLM ROCm 10 nightly (nightly-rocm100-8cbd5d03, vLLM 0.31.1rc1) with
  the same kernels; the image pin in docker/Dockerfile and the overlay stay on the validated September nightly.
  That nightly has an upstream bug in the DFlash draft model's fp8 exclusions (notes/upstream-issue-dflash-exclusions.md)
  which the plugin works around.
• Measured and left off: a 32 KB one-shot all-reduce cap (+0.35 ms a step), split-K across blocks for the hc
  down kernel, the fused routing + fold (-0.13 ms, two vLLM-runner patches). The decode all-reduce itself is at
  its protocol floor on this PCIe topology (fence and block-count sweeps in notes/decode-step.md).
• serve/serve.sh: PROF=1 PROFACT=CUDA records a GPU-only torch profile. tests/test_hc_f8.py,
  tests/test_fp8_qa.py (exact against their references, graph-timed benches); tests/test_ar_nrank.py takes NBS=.

Numbers (test box, 4x R9700, 210 W and -42 mV, probe.py unless noted)

Flash-Next TP4	v0.3.0	v0.3.1
decode step, 512-token essay (~/tp4tune.sh)	16.42-16.44 ms (five runs)	15.01 ms
single-stream decode	199 tok/s	217 tok/s
8 / 16 concurrent	547-561 / 871-889	567 / 898
8k prefill	6,750-6,870	6,743
KV cache	442-475k tokens	unchanged

Quality gate (2026-10-10, conc 1, no-think, both configs on this tree)

	GSM8K 1,319	HumanEval	strict sanity
v0.3.0 defaults	95.75% (1,263)	160 / 164	0 bad of 1,040
v0.3.1 defaults	95.83% (1,264)	161 / 164	0 bad of 1,040

Paired per question: 7 vs 8 discordant, McNemar p = 1.00; outputs identical on 25.5% of questions (the fp8 hc
decode rounds the hyper-connection GEMMs' bf16 bits differently in every layer). No detectable difference.

Validation record

Every configuration on this code, 2026-10-10: Flash-Next TP4 full BetterBench (combined 177.0, concurrency 168 / 267 / 392 / 495), 25-min mixed soaks (0 errors), strict sanity under overload, 27B TP2 and Flash-Next TP2 soaks + sanity, 4-rank all-reduce / all-gather and 2-rank race tests, the single-GPU kernel gates -- the full record with numbers is in notes/release-v0.3.1.md.

v0.3.0: Flash-Next on four cards, 70% more KV cache, 4% more prefill, same decode

r9700-stack v0.3.0

Flash-Next on four cards: 70% more KV cache, 4% more prefill, the same decode, on new measured defaults.
The code of v0.2.5 plus the decode-step fusions of 2026-10-07 (the P2P all-gather on by default) and the
prefill round of the same day (notes/prefill-fp8-pipe.md, notes/decode-nodes.md).

What changed for someone running it

• serve/flashnext.sh at TP4: the fp8 hyper-connection up GEMM with the gate mix fused (R9K_HC_FP8=r9k),
  memory utilization 0.98, cudagraphs only for prefill chunks up to 256 tokens. KV cache 279,564 -> 442-475k
  tokens (KVMEM=7.0 pins 475k), 8k prefill 6,769 -> 6,852-7,069 tok/s, decode step 16.4 ms and 199 tok/s
  unchanged, 888-905 tok/s at 16 streams. UTIL=0.94 CGSIZES=1,...,2048 R9K_HC_FP8=stock restores v0.2.5.
• The one-shot P2P all-gather for decode-sized gathers at TP > 2 (R9K_AG, default on): step 16.70 -> 16.40 ms.
• New kernels in the tree, off or opt-in: fp8 prefill GEMMs on the tiled WMMA kernel (163-182 TFLOPS), an exact
  block-scaled fp8 variant (slower than Triton here), one-launch MoE routing and the fused top-k sum with the
  shared expert, prefill all-reduce pipelining by row parts (correct, a loss on this topology), R9K_MEMSNAP.

Why the KV cache moved

vLLM sizes the KV cache from its profiling run's peak memory, and on this stack that peak was torch.compile and
inductor transients, not activations: 3.83 GiB against a measured 0.6 GiB model peak for a 4096-token chunk
(compat/memsnap.py). The graphs for 384-2048-token chunks cost another 1.4 GiB a card and 1.4 GiB of the
estimate for no soak-measured throughput. Utilization 0.98 then leaves about 1 GB at the soaked peak.

Checked (on this code, 2026-10-07/08, 210 W and -42 mV, prefix caching off)

Flash-Next TP4 on the new defaults, 2026-10-07: GSM8K full set at concurrency 1, paired per question against
the v0.2.6-era record: 95.45% vs 95.60%, 14 vs 16 discordant, McNemar p = 0.86, no detectable difference;
HumanEval 160 / 164 (the record). Mixed-length soak, 16 clients, 25 minutes: 684 requests, 0 errors, 6,666
prompt tok/s, VRAM peak 31,766 of 32,624 MiB a card. Strict sanity: 0 bad of 320 after long prompts, 0 of 1,700
at concurrency 17, 0 of 1,920 at 32. The 2048-chunk, 7.5 GiB and pipelined variants ran the same checks and are
documented as rejected.

The release validation chain on the final tree (~/validate-030.sh -> ~/val030.log): fn4 full BetterBench +
soak + sanity, 27B two cards soak + sanity, Flash-Next two cards soak + sanity, the 4-rank all-reduce /
all-gather / 2-rank race tests, the unit gates. A note on the record's provenance: the chain's first run
(23:24-00:28 UTC) and the 2026-10-07 "v0.2.6" validation launched the serve scripts without REPO=, so
serve/serve.sh mounted the production copy (~/r9700-build/repo, the v0.2.5 plugin) under the new serve
defaults -- the fp8 mix never ran there and that BetterBench (156.8 combined, 2k prefill 5,023) is v0.2.5 code
with the trimmed graphs. The run below mounts the release tree (checked on the container). Record:

configuration	probe (dec tok/s / c8 / c16 / 8k prefill)	soak, 16 clients, 25 min	strict sanity	VRAM
Flash-Next TP4, new defaults (KV 442,575 tokens)	BetterBench: combined 160.6, step p50 16.4 ms, conc 1 / 8 = 154.9 / 486.7 tok/s, prefill 2k / 8k / 32k = 5,178 / 7,182 / 7,246	628 ok, 0 errors, 6,722 prompt tok/s	0 bad of 320 (long prompts), 1,700 (conc 17), 1,440 (24), 1,920 (32)	31,301 MiB of 32,624, flat
27B two cards (KV 367,494)	212.8 / 578 / 672 / 3,951	314 ok, 0 errors, 3,139 prompt tok/s	0 bad of 320, 900, 720, 960	29.3 GB, flat
Flash-Next two cards, experts in host RAM (KV 133,306)	118.7 / 183 / 150 / 2,608	225 ok, 0 errors, 2,339 prompt tok/s	0 bad of 320, 900, 720, 960	flat

Against the previous record (v0.2.5 code, 2026-10-07 morning): the four-card BetterBench 162.8 -> 160.6 is within
the band (160.9 / 162.8 on identical code), 8k / 32k prefill +3.9% / +4.6%, 2k prefill -9% (the graph trim:
CGSIZES=1,...,2048 restores 5,684 at the cost of ~1.4 GiB of KV cache a card); the 27B is unchanged; the
two-card Flash-Next probe sits in its link-bound band.

Tests on the release tree: the 4-rank all-gather (test_ag_nrank, incl. graph replay) PASS, the 2-rank race
(test_ar_race, 1,500 replays, exact and compressed) ALL OK, the 4-rank all-reduce (test_ar_nrank) PASS after
a fix to the test itself (its burst and graph messages were sized at 80 KB against a 16 KiB one-shot cap and had
been failing with the kernel's -7 since the cap changed; the chain's grep hid it behind the race test's ALL OK --
b6e1c18). Single-GPU gates, all ALL OK / PASS: atiled_4bit, fold_mxfp4, prefill_4bit, tuned_cfgs, moe_mxfp4,
nvfp4, cache_moe, gemm_fp8, gdn_merge, moe_route_r9k, moe_sum_r9k, fp8_prefill_r9k. 0 MES timeouts on the boot.

Two BetterBench-only runs after the chain (03:10-03:50 UTC), on the release tree:

	defaults, run 2	defaults + the full cudagraph list (CGSIZES=1,...,2048)
KV cache	475,343 tokens	348,834
combined decode (median)	162.3	163.9
step p50 / update p99	16.4 / 17.1 ms	16.4 / 17.1
concurrency 1 / 8 (aggregate tok/s)	151.6 / 507.7	152.9 / 505.2
TTFT p99 at 8 streams	340 ms	275 ms
prefill 2k / 8k / 32k (tok/s)	5,101 / 7,129 / 7,245	5,833 / 7,220 / 7,221

So the band on the new defaults is 160.6-162.3 combined (v0.2.5 code: 160.9-162.8), the fp8 mix is +4.5% at
8k / 32k in every run, and the graph trim is a clean trade: +94-126k KV tokens against -12% on 2k-token prompts
and a higher TTFT tail under concurrency. Both are one knob in serve/flashnext.sh.

Also in this tag

• notes/mes-timeouts.md and the guest kernel parameters that settle the RDNA4 MES timeouts under KVM.
• notes/radiance-test-run.md: the radiance engine (1.0.4 / 1.0.8 / 1.1.1 / 1.2.0) through our checklist.
• README cut to highlights and a stats table.

v0.2.5: v0.2.4 with its validation record (no behaviour change)

r9700-stack v0.2.5

The v0.2.4 code, re-tagged with its validation record and the version string fixed. No behaviour change: against
v0.2.4 the plugin differs by the __version__ string and three lines of off-by-default diagnostics
(R9K_GDN_TRACE=1 now also covers the draft-less decode path).

Why a tag. v0.2.4 went out, by decision, on the unit gates and strict sanity under overload; the rest of the
release checklist ran the same day and was green on every configuration, and the tagged build announced itself as
0.2.3 in its startup log. This tag puts the record and the right version on the artifact people download.

What was checked (on this code, 2026-10-03, 210 W and -42 mV, prefix caching off)

See notes/release-v0.2.4.md, section "Checked", for the tables. In one line each:

• GSM8K, full 1,319-question set at concurrency 1: 27B two cards 94.69%, Flash-Next four cards 95.68%, two cards
  95.60%. HumanEval 164: 97.56% / 96.34% / 98.17%.
• Paired thinking-mode GSM8K on the 27B (first 800, concurrency 1): one-page and stock pages both 96.88%, 0
  discordant; no detectable difference against the September baseline (p = 0.36) or the reference stack (p = 0.21).
• BetterBench: decode level with or above the 225 W references on all three configurations, concurrency up 6-12% on
  the 27B, prefill 0-11% lower (the lower cap and the larger attention block).
• Mixed-length soaks, 16 clients, 25 minutes each: 1,283 requests, 0 errors, VRAM flat or plateaued.
• Strict sanity under overload: 0 bad answers of 8,340.
• One-page is deterministic run to run where the slot design was not (PROGRESS.md 2026-10-03, evening).

Also in this tag

• notes/bug-mes-invalidate-tlbs.md: the gfx1201 MES INVALIDATE_TLBS timeouts, the one escalation that reset
  the host, the per-card history, and the finding that linux-firmware's 2026-09-11 MES 0x93 does not reduce them --
  written as a comment for drm/amd issue 5759.

Upgrading

Nothing to do beyond v0.2.4. libr9k.so is unchanged.

v0.2.4: one GDN state page per request (64% more KV cache on the 27B)

r9700-stack v0.2.4

One change: the Gated DeltaNet layers keep one state page per request under speculative decoding instead of
vLLM's one page per candidate token. KV cache on the 27B on two cards goes from 223,329 to 367,494 tokens (+64%);
Flash-Next gains 18% on two cards and 9% on four; a running request pins a quarter to two thirds less of the pool;
and four concurrent 27B requests fit on one card, where stock pages admit two. Decode and concurrency are equal or
better, long prefill is 1-6% slower (a larger attention block). Output is bit-identical to the slot design.

Checked, the same day as the tag (2026-10-03, 210 W and -42 mV on every card, prefix caching off so one-page
is active on the 27B): the unit gates; strict sanity under overload on all three configurations (0 bad of 8,340
across the day); GSM8K on the full 1,319-question test set at concurrency 1 and HumanEval on all 164 problems on
all three configurations; the paired thinking-mode GSM8K against the September baselines on the 27B; a full
BetterBench on all three configurations; a 25-minute mixed-length soak (16 clients, 200-30k-token prompts) on all
three with VRAM sampled. Numbers in the tables below. The tag itself went out a few hours before these finished,
by decision; nothing in them changed the code.

What it does

vLLM verifies the speculative candidates of a step from a checkpoint and must be able to resume from whichever
candidate turns out to be the last accepted one, so it keeps a state page per candidate: 1 + 7 pages per request
per GDN layer group on the 27B. The plugin's verify kernel (r9k_gdn_spec_verify) carries, inside the single
page, a record of the previous step's candidate rows (their conv'd q/k/v and the a/b gates, two slots so a step can
read one and write the other, and a flag word for the live slot, its parity and a done counter). At the start of a
step it replays the accepted rows from the record into the state, stores that as the new checkpoint, then runs the
new candidates. The replay recomputes exactly what the slot design stored, so the result is bit-identical to it
(tests/test_gdn_onepage_r9k.py: multi-step simulation with random acceptance, both state dtypes, the sigmoid
gate, row maps, several layers, and identical requests in lockstep against the same request alone).

Stage 1 covers mamba_cache_mode "none", i.e. prefix caching off: PREFIX_CACHE=0, which is already the default
for Flash-Next and a knob for the 27B (serve/27b.sh ... PREFIX_CACHE=0). With prefix caching on, vLLM's pages are
used unchanged. R9K_GDN_STATE=stock forces the old layout.

210 W, -42 mV, prefix caching off	stock pages	one page
27B, two cards: KV cache	223,329	367,494	+64%
27B, two cards: pool pinned by 8 running requests	59.4%	11.5%
27B, two cards: decode / c8 / c16 / 8k prefill, tok/s	206 / 534 / 620 / 4,055	212 / 579 / 673 / 3,940
27B, one card: KV cache	37,732	62,295	+65%
Flash-Next, two cards (restarted launches): KV cache	154,641	183,202	+18%
Flash-Next, two cards: pinned per running request	10,386	6,736	-35%
Flash-Next, four cards: KV cache	256,682	279,564	+9%
Flash-Next, four cards: pinned per running request / 8 running	6,152 / 19.2%	3,739 / 10.7%
Flash-Next, four cards: decode / c16 / 8k prefill	194 / 893 / 7,230	195 / 898 / 6,790	prefill -6%

A defect found and fixed before the tag

The stock conv-window update takes max_query_len, the number of candidate columns the window rolls by, and the
plugin passed the width of the speculative state-index tensor. With one page per request that KV group's block
table has a single column, so on every step vLLM builds eagerly -- mixed prefill-plus-decode steps, non-uniform
batches -- the width was 1 and the window rolled as if each step had one candidate. Full-cudagraph decode steps
were right only because vLLM fills its graph buffer with a broadcasting copy. The symptom was subtle: a lone
request matched stock exactly, identical copies in a batch agreed with each other but not with the lone run, and
under --enforce-eager the output was garbage. A per-step checksum trace of the first GDN layer found it in the
second step after a prefill. The width now comes from the record. PROGRESS.md 2026-10-03 has the whole trail,
including what remains of batch dependence afterwards and why none of it is in the GDN path (the compressed
all-reduce on two cards, and the drafter's choices at three or more identical prompts in one step).

Checked

210 W, -42 mV, one-page	27B two cards	Flash-Next four cards	Flash-Next two cards (offload)
GSM8K full 1,319, no thinking, conc 1	94.69%	95.68%	95.60%
HumanEval 164	97.56% (160)	96.34% (158)	98.17% (161)
BetterBench combined decode, tok/s (README 225 W ref.)	200.3 (197.5)	160.9 (159.2)	97.3 (97.4)
BetterBench concurrency 1 / 2 / 4 / 8, tok/s	185 / 314 / 450 / 558	151 / 236 / 353 / 505	88 / 109 / 117 / 114
BetterBench prefill 2k / 8k / 16k / 32k, tok/s	3,860 / 4,076 / 4,035 / 3,825	5,689 / 6,930 / 7,169 / 6,929	2,149 / 3,511 / 3,834 / 3,784
time to first token p50	71 ms	77 ms	463 ms
mixed soak, 16 clients, 25 min	383 served, 0 errors, VRAM flat	661 served, 0 errors, VRAM plateau	239 served, 0 errors, 14 over-length prompts rejected by design
strict sanity under overload	0 bad of 900 and 960	0 bad of 1,700 and 1,920	0 bad of 900 and 960

Against the README's 225 W references: decode and step time level or slightly better (the undervolt), concurrency
up 6-12% on the 27B and 6% at 8 clients on four cards (the KV room), first token keeps the v0.2.3 gain, prefill
4-11% lower on four cards and 0-8% on the 27B at the short depths (the lower cap plus the larger attention block).

Paired GSM8K, 27B two cards, thinking on, first 800 questions, concurrency 1 (the comparable form of the
September baselines): one-page 96.88% and stock pages 96.88% on the same code, 0 discordant questions, 782 of 800
outputs byte-identical; against our v0.2.0-era baseline (97.50%) 7 vs 12 discordant, p = 0.36; against the reference
stack (97.62%) 5 vs 11, p = 0.21: no detectable difference either way. The 99% output divergence from the September
runs is the v0.2.3 short-prefill core taking a more exact numeric path.

Determinism. Replaying the 18 differing thinking answers alone, twice per server: one-page gave identical
text and identical step counts every time, on both card pairs; stock pages did not (one question changed length
between two runs on the same server), and stock's step counts moved with the attention block size too. The slot
design has a rare run-to-run wobble that one-page does not; accuracy is unaffected. Low-priority follow-up now that
one-page replaces it.

• Reviewer passes by the 27B itself on the three diffs: no confirmed defect.
• The deliberate five-card reproduction of the night's host crash (three servers launched at once): clean.

Not checked

• Nothing from the v0.2.2 checklist is outstanding for this release.

Operational notes from the same day (PROGRESS.md 2026-10-03)

• The test host crashed once when a chronic MES firmware timeout on one card escalated into a GPU reset; under
  passthrough the reset took the card off the bus and the host sync-flooded. The timeout has occurred ~130 times
  since September on every card and self-recovers; amdgpu.gpu_recovery=0 in the guest is the proposed
  containment. Not caused by this code or by load.
• All cards now run at 210 W with a -42 mV offset (LACT); the guest's power-cap service had a bug that left cards at
  225 W after a reboot, fixed.

Upgrading

Rebuild libr9k.so (kernels/build.sh; serve/serve.sh does it when the kernels are newer than the library).
To get the memory, run with prefix caching off (PREFIX_CACHE=0 for the 27B); nothing else changes. First launch
of a configuration compiles and keeps ~0.45 GiB; restart once, as before.

Files

• kernels/r9k_gdn.hip: r9k_gdn_spec_verify, r9k_gdn_spec_commit, flags in r9k_gdn_seq; kernels/build.sh.
• r9700_vllm/models/gdn.py: spec_verify, _spec_width, the one-page KV spec and binding, the mixed-step and
  non-spec-decode paths, R9K_GDN_STATE, diagnosis knobs R9K_GDN_TRACE / R9K_GDN_PAGE_PAD.
• r9700_vllm/models/qwen3_5.py (new), r9700_vllm/models/qwen4_exp.py, r9700_vllm/__init__.py: model classes
  that size the mamba page for the record.
• tests/test_gdn_onepage_r9k.py (new).
• PROGRESS.md 2026-10-03: design, measurements, the defect and its trace, the host crash, the undervolt.

r9700-stack v0.2.3: first token in half the time, 28% more KV cache on two cards

r9700-stack v0.2.3

Two changes. A short prompt's first token arrives in about half the time -- the 27B on two cards is now level with
the reference stack -- and Flash-Next on two cards gets 28% more KV cache. Decode, long-prompt prefill and memory
use are otherwise unchanged.

What this release was checked with, and what it was not. v0.2.2 set a checklist for every release (unit gates,
all-reduce suites, strict sanity under overload, mixed-length soaks, a full BetterBench, on every configuration).
v0.2.3 is tagged on part of it, by decision: the unit gates, strict sanity under overload on all three
configurations (0 bad answers of 3,850), and the two long soaks for the memory change. Not run: a quality
eval, a full BetterBench, the mixed soak on a restarted two-card launch. Short prompts now go through a numerically
different (more exact) path, so an eval is owed with the next release.

1. Time to first token: the GDN prefill core

first token, ms (client median, one request at a time)	45 tokens	83	159	236	320	404	656
27B, two cards: v0.2.2	97	96	147	~147	148	148	172
27B, two cards: v0.2.3	47	60	81	92	128	142	172
reference stack, same cards, same day	46	68	93		125		158
Flash-Next, four cards: v0.2.2	87	87	88	88	92	107	131
Flash-Next, four cards: v0.2.3	44	58	73	87	91	107	132

The 27B answered a 45-token prompt in 97 ms and the reference in 46, and the 2026-09-21 investigation had closed
that as upstream behaviour. The server's own counters said otherwise: zero queue time, one engine step per request,
and that step ~90 ms where a decode step is 23. A profile of the step with Python stacks found it: a prefill step
runs vLLM's chunked GDN core eagerly in every GDN layer -- six Triton kernels plus glue, about 30 launches and 1.2
ms of Python per layer whatever the prompt length, 48 layers on the 27B. That is the fixed ~60 ms, and it is also
what every running request waits through when a new one joins the batch.

Now, for a step with few prefill tokens, the recurrence runs token by token in one launch per layer
(r9k_gdn_seq: the arithmetic decode already uses, the state in registers, any number of tokens) after the causal
conv in one launch (r9k_gdn_conv, bit-identical to vLLM's Triton kernel -- which, it turned out, truncates its
bf16 tap products on this backend; found by enumerating the rounding choices against its output). Both kernels
address the batch's own rows, so a step that mixes prefills with running spec-decode sequences needs no gather.

Token by token is slower on the GPU than the chunked form (~1.8 us per token and layer), so it only pays while the
step is bound by launches rather than compute. R9K_GDN_PREFILL_MAX (prefill tokens per step) defaults to 256,
where Flash-Next on four cards breaks even; serve/27b.sh sets 400, the 27B breaking even near 440. Above the
threshold nothing changes. R9K_GDN_PREFILL_MAX=0 restores vLLM's core; R9K_GDN_PREFILL_CONV=stock keeps its
conv. An older libr9k.so without the two new symbols keeps the chunked core (and the old first-token time).

Numerics: against an fp64 token-by-token reference the new core differs only by bf16 rounding flips (relative
error 1-6e-5); vLLM's chunked form, which does its solves on bf16 q/k, is at 5e-3 from the same reference. So the
outputs for short prompts are not bit-identical to v0.2.2 -- they are closer to the exact recurrence, and now and
then a token differs. tests/test_gdn_prefill_r9k.py holds all of this, including spec-decode sequences through
the new kernel being bit-identical to the decode kernel.

The investigation also measured and rejected two other candidates: prefill graphs up to 2,048 tokens (moved only the
158-319-token range on the 27B and cost 0.8 GiB of VRAM) and prefix caching off (no change in first-token time;
2.1 GiB less VRAM and 6% more KV on the 27B, noted for later). Flash-Next on two cards with offloaded experts:
52 ms at 44 tokens; above its graph limit it runs eagerly as before (177 ms at 82 tokens).

2. Flash-Next on two cards: --gpu-memory-utilization 0.94 -> 0.96

	0.94	0.96
KV cache, first launch of a configuration	95,429 tokens	128,772
KV cache, restarted launch (the normal way to run it)	121,299	155,791 (+28%)

v0.2.2 rejected 0.96 because it died with 69 MiB free when the request queue drained. That allocation was the
short-conv prefill batch fixed in the same release, and it had not been retested since. At 0.96 on this code: the
soak that killed it (8 clients, prompts of 18k-30k tokens, 16 minutes) served 86 with none failed; the mixed soak
(16 clients, 200-30k tokens, 26 minutes) served 227 with none failed; VRAM flat at 30,396 of 32,624 MiB; strict
sanity after 24k-token prefills 0 bad of 180. Those ran on a first launch, which has the smaller pool. On a
restarted launch the first 5.7 minutes of the mixed soak peaked at 31,350 MiB with no error, and the run was
stopped there, so that case has had a burst, not a soak. serve/serve.sh picks 0.96 only for TP2 with offloaded
experts; four cards and the 27B keep 0.94; UTIL=0.94 restores the old value.

Checked

• Unit gates: test_gdn_prefill_r9k.py (new), test_gdn_decode_r9k.py, test_gdn_merge.py, test_moe_mxfp4.py,
  test_cache_moe.py, test_gemm_fp8.py, test_ple_conv.py.
• Strict sanity with more requests than max_num_seqs, after short and after long prefills: 27B two cards 0 bad
  of 990; Flash-Next four cards 0 of 1,870; Flash-Next two cards 0 of 990. (Run at a 512-token threshold, which
  sends more steps down the new path than the shipped 256 / 400.)
• Concurrency probes, three per launch, Flash-Next four cards with and without the change: 884-887 against
  866-880 tok/s at 16 concurrent, i.e. unchanged.
• The two soaks and the sanity checks of section 2.
• The smoke test caught one bug before the tag: with speculative decoding the conv state is wider than the conv's
  three columns, the wrapper asserted exactly three, and the server died at warm-up. Fixed; the unit test has the
  case.

Not checked

• No quality eval (GSM8K or HumanEval) on this code.
• No full BetterBench; the first-token numbers above are per-request client medians (bench/ttft_breakdown.py),
  not BetterBench's streaming p50.
• No full-length soak of a restarted two-card launch at 0.96.

Upgrading

Rebuild libr9k.so (kernels/build.sh; serve/serve.sh does it when the kernels are newer than the library).
Nothing else changes: same models, same launch scripts, same knobs plus the two new ones above. On a first launch
of a configuration the plugin source hash changes the compile cache key, so the first launch compiles and keeps
~0.45 GiB; restart once, as before.

Files

• kernels/r9k_gdn.hip: r9k_gdn_seq, r9k_gdn_conv; r9700_vllm/models/gdn.py: seq_core, conv_prefill,
  _prefill, the knobs.
• tests/test_gdn_prefill_r9k.py, bench/ttft_breakdown.py.
• serve/serve.sh (UTIL), serve/27b.sh (threshold 400), serve/flashnext.sh (notes).
• PROGRESS.md 2026-10-02: the investigation, the kernel, the threshold measurements, the 0.96 soaks.

r9700-stack v0.2.2: three fixes for real traffic

r9700-stack v0.2.2

A fix release. Upgrade, whatever you run. Every release so far (v0.1.0, v0.2.0, v0.2.1) has three bugs that
no fixed-depth benchmark and no evaluation at concurrency 1 can see; each was found by loading the server the way
real traffic does.

	affects	symptom
1	two cards: Flash-Next with offloaded experts, and the 27B	wrong output: when a request joins a batch that is already decoding, one sequence can turn to garbage (the right beginning, then one token repeated)
2	Flash-Next with offloaded experts	memory leak: out of memory after minutes to hours of mixed-length prompts
3	Flash-Next, any configuration; four cards at 16 concurrent first	out of memory when one step holds prompts of very different lengths

Speed and correct outputs are unchanged from v0.2.1.

If you cannot upgrade: R9K_R4D_AR=0 (all-reduce on RCCL) avoids the wrong output on two cards. The other two
have no setting that avoids them; fewer concurrent sequences (NSEQ) makes the third less likely, and restarting
the server returns leaked memory.

1. Garbled answers: a race in the 2-rank all-reduce

The two cards exchange and sum their halves of every layer's output through a double-buffered scratch, so that a
card already on the next message cannot overwrite what the other is still reading. Which half to use was decided
per block of the kernel, while the number of blocks followed the message size. After a small message the next
larger one could therefore land in the same half as the message before and overwrite its last rows. The two cards
then disagreed about the last sequence of the batch, and it decoded garbage from that step on.

It needs two back-to-back messages of different sizes with the cards slightly out of step, which is what happens
when a prompt joins a batch of running decodes. Measured with a strict version of the sanity check
(bench/sanity_stress.py), sending more requests than max_num_seqs:

	v0.2.1	v0.2.2
Flash-Next TP2, 9 / 12 / 16 concurrent (limit 8): bad answers	74 of 900 / 57 of 1,200 / 64 of 1,600	0 of 4,600
Flash-Next TP2, sparse graph sizes: rounds with a bad answer	99 of 100	0 of 100
27B TP2, 9 concurrent (limit 8): bad answers	8 of 900	0 of 2,700
Flash-Next TP4, 17 / 24 / 32 concurrent (limit 16): bad answers	0 of 6,020	0 of 6,020
the same server on RCCL or libr4d all-reduce	0 of 1,620 each

The fix: every call advances every block's counter (a fixed launch grid; blocks without data do nothing else), so
all blocks always agree on the half. The 4-rank kernels were built the same way and are fixed the same way.
tests/test_ar_race.py reproduces the fault with the old grid (6,008 wrong outputs in 1,500 graph replays) and
passes with the new one.

2. Memory leak in the expert cache

The cache kept one set of alignment buffers per batch shape, per layer, for good. A prefill chunk shares each
step's token budget with whatever is decoding, so under real traffic nearly every step has a new shape, and each
left up to 28 MiB per card behind.

With a new soak test (bench/soak.py: 8 clients for 15 minutes, each sending an 18k-30k-token slice of real text):

	v0.2.1	v0.2.2
requests served / failed	77 / 369	111 / 0
VRAM used per card	30.5 -> 32.6 GiB in 13 minutes, then out of memory, engine dead	31.0 GiB flat, one step to 31.6

Now there is one buffer set per layer, reused by every step. tests/test_cache_shapes.py.

3. Out of memory when a step holds prompts of very different lengths

Flash-Next's short convolution packs all the prompts being prefilled in one step into a batch of prompts x
longest, and holds about six tensors of that size. One long chunk next to many short prompts is many times the
step's real tokens: 3,000 + 15 x 70 tokens peak at 4.8 GiB in that one function, where vLLM's memory profile (one
sequence) budgeted for 84 MiB per tensor. This is vLLM's own algorithm; our kernel override mirrored it.

On four cards, a soak of 24 clients with prompts of 200 to 30k tokens took the cards from 29.4 to 32.2 GiB in 40
seconds and killed the engine. On two cards it showed as a one-time 640 MiB step that happened to fit.

The fix packs the step's prompts by length, in groups of at most 1.25x the step's own token count. Each sequence's
convolution reads only its own tokens and state, so the results are bit-identical (tests/test_ple_conv.py
compares with stock); the peak for the cases above is 0.4-0.5 GiB.

Also in this release

• Two-card prefill depends on the prompt. v0.2.1 reported prefill of 2,208-3,839 tok/s, 5-14% above v0.2.0.
  Those are BetterBench figures on its filler (one paragraph's words, shuffled), and the gain exists only for such
  narrow prompts. Real text prefills at about 2,300-2,600 tok/s on every version (bench/prefill_kinds.py).
  The four-card numbers do not have this dependence.
• UTIL=0.96 is rejected for two cards: it ran at 69 MiB free and died when the request queue drained (bug
  3, as it turned out). The default stays 0.94.
• A same-day baseline for the 27B against the reference stack on the same two cards: decode 196.6 vs 197.5
  tok/s, prefill 81-87%, time to first token 116 vs 65 ms, concurrency 94-97%.
• Release checklist. A release is now stressed at more requests than max_num_seqs and soaked with
  mixed-length prompts, on every configuration, before it is tagged; the sanity check is strict.

Validation

Run on the release code, on every configuration: 26 single-GPU unit gates, the all-reduce suites on two and four
ranks, a full BetterBench, the strict stress at more requests than max_num_seqs, and a mixed-length soak with
VRAM sampled every second.

	Flash-Next, 2 cards + offload	Flash-Next, 4 cards	27B, 2 cards
single-stream decode (tok/s)	97.4	157.3	197.5
decode step p50 / TTFT p50	25.6 ms / 483 ms	16.8 ms / 97 ms	23.5 ms / 114 ms
concurrency 1 / 2 / 4 / 8 / 16	87 / 109 / 116 / 116 / 116	151 / 239 / 355 / 473 / 624	174 / 280 / 413 / 519
bad answers over the sequence limit	0 of 2,500	0 of 4,900	0 of 2,100
mixed-length soak: served / failed	230 / 0 (16 clients, 26 min)	450 / 0 (24 clients, 15 min)	232 / 0 (12 clients, 15 min)
VRAM peak in the soak, of 32.6 GiB	30.9	30.2	32.5, once

Speed matches the previous release on all three within run-to-run noise.

Upgrading

No configuration changes. libr9k.so must be rebuilt (kernels/build.sh): the plugin refuses an older one.

Read next

PROGRESS.md (2026-10-01 and 2026-10-02 sections) for the measurements and the bisect; CHANGELOG.md for the list.

r9700-stack v0.2.1: Flash-Next on 2x R9700, tuned and measured

r9700-stack v0.2.1

A tuning-and-corrections release for Qwen3.8-Flash-Next on two R9700s (TP2, experts in host RAM), plus a
correction to v0.2.0's quality statement. No new kernels. The four-card numbers and all numerics are those of
v0.2.0; stock vLLM (0.30 nightly) and stock ROCm 10, as before.

Performance: Flash-Next on 2× R9700 with offloaded experts

Full BetterBench (20 passes), one card per PLX switch, 225 W cap, 34 GB of experts per rank in host RAM, 270
expert slots per layer in VRAM, MTP-3.

	v0.2.1 defaults	v0.2.0 defaults
single-stream decode	97.4 tok/s	93.8	+4%
decode step p50	25.6 ms	25.7 ms	=
time to first token p50	484 ms	555 ms	-13%
prefill 2k / 8k / 16k / 32k	2,208 / 3,558 / 3,839 / 3,793	2,099 / 3,163 / 3,434 / 3,329	+5% / +12% / +12% / +14%
concurrency 1 / 2 / 4 / 8 / 16	87 / 109 / 116 / 118 / 113	84 / 104 / 108 / 111 / 95	+4% / +5% / +7% / +6% / +19%
time to first token at 8 concurrent	0.95 s	7.1 s	7× faster
8 copies of one prompt (code / json)	434 / 577 tok/s	283 / 347	+53% / +66%
sanity set	8 / 8	8 / 8

What changed

Three defaults, each found by measurement and confirmed by the BetterBench above:

• NSEQ=8 in serve/flashnext.sh. Every request holds 18 KV blocks whatever its length (vLLM: four
  recurrent-state groups × (1 + 3 MTP blocks) + 2 attention blocks). With graphs and activations sized for 16
  sequences there was KV room for four to six; the rest queued. Pass NSEQ=16 on four cards.
• R9K_LRU_THRESH=0.99 (was 0.5). Eight different requests route to about 150 distinct experts per layer.
  Above half the slots the cache manager inserted nothing -- on 72% of steps -- so the cache stopped following the
  traffic and every step read 31 experts per layer from host RAM. With only NSEQ=8, conc-8 fell to 77 tok/s;
  with the threshold raised it is 118.
• R9K_LRU_GATHER=64,16 (was 8,16): the insert copy is ~15% faster at 8 or more inserts, equal below.

None of them can change outputs: the expert cache is bit-identical to reading every expert from the backing store
(tests/test_cache_moe.py).

What bounds this configuration

The PCIe link, not the GPUs. Every routed expert that is not resident is 1.245 MiB per card, and the copy kernel
already runs at the link rate (11.4-13.5 GB/s on PCIe 3). Flash-Next routes diffusely -- 90% of a layer's routing
needs 216-365 of its 512 experts -- so real, mixed traffic misses 12-16% of its routed experts per step and total
throughput levels off near 115 tok/s from four requests up. More users share that total; they do not add to it
(per-request decode 96 / 64 / 34 / 17 tok/s at 1 / 2 / 4 / 8).

The cards agree: a kernel is always running, but they draw 206 W with one request and 170 W with eight, against
219 W in prefill. Under concurrent load most of the compute is waiting for experts.

Near-identical requests are the best case, because they share their experts: eight copies of one prompt run at
434-577 tok/s. Four cards with everything in VRAM are 1.6× faster single-stream and 3-6× at concurrency, and a
PCIe 4 or 5 host would lift the two-card numbers with no code change.

Quality: a correction to v0.2.0

v0.2.0's notes said the round-3 decode fusions together sit about half a point below the earlier numerics. They
do not. On the full GSM8K test set (1,319 questions, chain-of-thought, concurrency 1, paired):

fusions on	accuracy	reference-only right	leg-only right	p
none (reference)	96.82%
none, six hours later	96.82%	0	0	1.00
router GEMM	96.82%	6	6	1.00
indexer norm + rope glue	96.97%	0	2	0.50
hyper-connection mix	97.04%	5	8	0.58
GDN speculative-decode core	96.97%	5	7	0.77
shared expert	96.74%	6	5	1.00
all five (the default)	97.04%	5	8	0.58

The half point came from evaluating on the first 800 questions, where the reference happens to score 98.0% (95.0%
on the other 519): anything that reshuffles marginal questions looks like a loss on that slice. Evals for a change
of default now use the full set. Nothing needs switching off.

New tools

• R9K_EXPERT_CACHE_STATS=1: per-rank counters from the expert cache, logged every
  R9K_EXPERT_CACHE_STATS_SEC seconds (distinct routed experts, inserts, experts read through from host, steps
  over the insert threshold). Opt-in; it adds a few small launches per layer.
• bench/mix.py: conc-8 throughput by workload mix (one prompt type against several).
• tuning/lru_gather_bench.py: host → VRAM copy rate of the insert kernel by insert count and launch grid.

Operational notes

• Restart once after the first launch of a new configuration. The launch that compiles leaves ~0.45 GiB less
  for the KV pool (94k against 121k tokens at NSEQ=8).
• On a box with two PLX switches put one card on each for offloaded experts (GPUS=0,2), and both on one
  switch for plain tensor parallel.

Tried and rejected

240 expert slots (more KV room; -28% on a mixed batch, -6% single stream), NBT=2048 as a default (-24%
prefill), re-splitting the slots across layers by routing mass (13.3% → 12.8% misses). UTIL=0.96 ran clean with
28% more KV room but has not held a 32k context under load, so it is not a default.

Read next

PROGRESS.md (2026-09-29 sections) for every number above and the method behind it; CHANGELOG.md for the list.

r9700-stack v0.2.0: Flash-Next on 4x R9700, ahead on every metric

r9700-stack v0.2.0

Tuned gfx1201 kernels and a vLLM plugin that run Qwen3.8-Flash-Next on 4× and Qwen3.8-27B-NVFP4 on 2×
AMD Radeon AI PRO R9700, using stock vLLM (0.30 nightly) and stock ROCm 10 -- no forks, no patched source,
nothing to re-port when either moves.

The headline: on four cards, Flash-Next is now ahead of the fastest known alternative stack on every metric
we measure. v0.1.0 trailed it by 5-20%. This is on a PCIe 3 host (cards behind two PLX switches on Gen3
uplinks, ~13.7 GB/s between them); a PCIe 5 box would lift the multi-card numbers further with no code change.

Performance

Qwen3.8-Flash-Next (MXFP4 / fp8 GPTQ), TP4, 4× R9700 at a 225 W cap, full BetterBench (20 passes), MTP-3
speculative decoding on both stacks. The reference column is the fastest known alternative stack, measured on
the same box, checkpoint and cap.

	v0.2.0	reference
single-stream decode	159.2 tok/s	134.1	+19%
decode step p50	16.8 ms	20.4 ms	-18%
time to first token p50	94 ms	145 ms	1.5× faster
prefill 2k / 8k / 16k / 32k	6,408 / 7,365 / 7,451 / 7,182	5,279 / 5,711 / 5,977 / 6,106	+18-29%
concurrency 1 / 2 / 4 / 8 / 16	151 / 231 / 350 / 478 / 635	126 / 197 / 303 / 427 / 542	+12-20%
sanity set	8 / 8

Qwen3.8-27B-NVFP4 on 2× R9700 (TP2) is unchanged from v0.1.0: 94% of the reference stack on decode, 80-86% on
prefill (the PCIe 3 all-reduce ceiling on this host).

What moved it

• Prefill (+20-31% over the reference at every depth): a WMMA scorer for the sparse-attention indexer that
  touches only the visible columns; per-row scales and staged 16 B stores in the MoE prefill epilogues; the
  compressed hierarchical 4-rank all-reduce; switch-local P2P on both PLX switches (2× peer bandwidth); prefill
  chunks up to 2k tokens captured as HIP graphs (TTFT 137 → 88 ms).
• Decode (+19% single-stream, +12-20% at concurrency): on this ROCm each HIP-graph node costs ~1.5 µs of
  dispatch, and a Flash-Next step had ~3,250 of them against the reference stack's ~1,640. Five fusions replaced
  the runs of tiny launches with one kernel each: router GEMM, indexer norm + rope, hyper-connection mix,
  the shared expert (8 → 4 launches), and the Gated DeltaNet speculative-decode core (11 graph nodes → 1 per
  layer). The last two alone took the step from 19.0 to 16.8 ms.

What's in it (new since v0.1.0)

• Four-card host work (host/): 32 GB BARs for two cards per PLX switch, guest-side BAR placement so every
  card sits at its host address, switch-local P2P on both switches. Documented, scripted, DKMS-packaged.
• N-rank all-reduce (one-shot / two-shot) and the compressed 4-rank hierarchical variant.
• Sparse attention kernels for Qwen4Exp and the indexer scorer.
• Decode fusions listed above, each with a R9K_*=stock knob and a bit-level or paired-eval gate.
• vLLM 0.30 base; TP=4 tuning table; hybrid block-fp8 dispatch; MXFP4 LM heads.
• A CHANGELOG.md, and notes/independence.md lists every runtime hook beyond vLLM's extension points.

Quality

Every change to a shipped default runs an 800-question chain-of-thought paired eval at concurrency 1 (paired
McNemar) plus the 8-prompt sanity set before it ships. v0.2.0's default scores 97.50% on that eval, no
detectable difference from the previous default on its own (5 / 1 discordant, p = 0.22), sanity 8 / 8. Across all
runs the three round-3 decode fusions together sit about half a point below the round-2 build (97.9-98.0%); that
is a knob, not a bug -- R9K_ROUTER=stock R9K_QSA_GLUE=stock R9K_HC_MIX=stock restores round-2 numerics for ~10%
of decode. The full bisect is in PROGRESS.md.

Licence

Apache-2.0, with one carve-out: serve/templates/qwen-fixed-v22.3.jinja is GGZ14's, used with permission
and credit. kernels/third_party/davetha/ is Apache-2.0 upstream and passes on normally. See NOTICE.

Known limits

• Two P2P chains only. Switch-local P2P covers the pairs on each PLX switch; traffic between the switches
  still crosses the host's Gen3 uplinks, which is why the 4-rank all-reduce is compressed.
• The stack is not reproducible at concurrency > 1 (dynamic batching changes reduction order); evaluate at
  concurrency 1, where it is near bit-reproducible (bench/eval.py selftest).
• Numerics move a little with every fusion. Each is gated, but the accumulated drift from a series of them is
  only visible in the paired evals, which is why they are run against the previous default rather than the
  original.

Read next

notes/picking-up.md for orientation; PROGRESS.md for every number and every rejected idea; CHANGELOG.md
for the list.