v0.3.1: decode step 16.4 -> 15.0 ms on four cards, same quality, same KV, same prefill

Flash-Next on four cards: the decode step 16.4 -> 15.0 ms, single-stream decode 199 -> 217 tok/s, same
quality, same KV cache, same prefill. The code of v0.3.0 plus the decode-step round of 2026-10-09
(notes/decode-step.md) and the 2026-10 vLLM nightly compatibility fixes (notes/vllm-upgrade-2026-10.md).

What changed for someone running it

• serve/flashnext.sh turns on two new paths by default (the first at TP4 only):
  • fp8 hyper-connection decode GEMMs (R9K_HC_FP8_DECODE=1, kernels/r9k_hc_f8.hip): the down+hc_silu and
    up+gated-mean GEMMs at decode widths (up to 16 tokens) read the fp8 fragment-order copies the prefill mix path
    already holds, half the bytes of the bf16 kernels that were the step's largest stream (1.29 GB a step), with
    bf16 WMMA against the unquantized activations. 16.42 -> 15.33 ms a step. =0 restores the bf16 kernels. At
    TP2 (experts in host RAM) it stays off: the fp8 down copy costs 30k KV tokens there (133,306 -> 103,517) for
    +10% single-stream decode and nothing at 8 streams; R9K_HC_FP8_DECODE=1 opts in.
  • fused activation quant in the dense fp8 block GEMM (R9K_FP8_QA=1, r9k_gemm_fp8_block_qa): the
    per-token-group-128 quant runs inside the GEMM's A-load with bit-identical operands; 96 launches fewer a step.
    15.33 -> 15.01 ms. =0 splits it again.
• The plugin also runs on the 2026-10-09 vLLM ROCm 10 nightly (nightly-rocm100-8cbd5d03, vLLM 0.31.1rc1) with
  the same kernels; the image pin in docker/Dockerfile and the overlay stay on the validated September nightly.
  That nightly has an upstream bug in the DFlash draft model's fp8 exclusions (notes/upstream-issue-dflash-exclusions.md)
  which the plugin works around.
• Measured and left off: a 32 KB one-shot all-reduce cap (+0.35 ms a step), split-K across blocks for the hc
  down kernel, the fused routing + fold (-0.13 ms, two vLLM-runner patches). The decode all-reduce itself is at
  its protocol floor on this PCIe topology (fence and block-count sweeps in notes/decode-step.md).
• serve/serve.sh: PROF=1 PROFACT=CUDA records a GPU-only torch profile. tests/test_hc_f8.py,
  tests/test_fp8_qa.py (exact against their references, graph-timed benches); tests/test_ar_nrank.py takes NBS=.

Numbers (test box, 4x R9700, 210 W and -42 mV, probe.py unless noted)

Flash-Next TP4	v0.3.0	v0.3.1
decode step, 512-token essay (~/tp4tune.sh)	16.42-16.44 ms (five runs)	15.01 ms
single-stream decode	199 tok/s	217 tok/s
8 / 16 concurrent	547-561 / 871-889	567 / 898
8k prefill	6,750-6,870	6,743
KV cache	442-475k tokens	unchanged

Quality gate (2026-10-10, conc 1, no-think, both configs on this tree)

	GSM8K 1,319	HumanEval	strict sanity
v0.3.0 defaults	95.75% (1,263)	160 / 164	0 bad of 1,040
v0.3.1 defaults	95.83% (1,264)	161 / 164	0 bad of 1,040

Paired per question: 7 vs 8 discordant, McNemar p = 1.00; outputs identical on 25.5% of questions (the fp8 hc
decode rounds the hyper-connection GEMMs' bf16 bits differently in every layer). No detectable difference.

Validation record

Every configuration on this code, 2026-10-10: Flash-Next TP4 full BetterBench (combined 177.0, concurrency 168 / 267 / 392 / 495), 25-min mixed soaks (0 errors), strict sanity under overload, 27B TP2 and Flash-Next TP2 soaks + sanity, 4-rank all-reduce / all-gather and 2-rank race tests, the single-GPU kernel gates -- the full record with numbers is in notes/release-v0.3.1.md.