mudler · localai-bot · Aug 7, 2026
diff --git a/.agents/NOW.md b/.agents/NOW.md
@@ -18,7 +18,7 @@ Working head: `row/backend-rocm-w0` (#41). Prior: benchmark checkpoint
 | f32-out GEMV audit | Only laguna + ds4 bf16 tower affected; gate models unaffected | Re-verify ds4 tower same-tool |
 | Invocation-parity prevention | CI guard + checklist landing | Merge; build-verify `kGemvHeuristicAlgos` on dgx |
 | MiniMax-H3 lane | **bf16 shards STREAM both towers (DiT + encoder); Q4_K_M enc cond cos 0.9975, 3.5° med, DIFFUSE** | render A/B on saved embeds |
-| Kimi-Linear-48B | **Paged-incremental decode LANDS (#113): 18.9 tok/s, 4.5× over recompute, 0.90× vLLM ~21** (gap 0.20×→0.90×). Gate A token-identical to recompute (122/128); NOT STRICT (p7 near-tie); decode 90% cuBLAS-GEMV-parity | Next: bf16 residual stream (p7+cast) + paged-FA2 MLA |
+| Kimi-Linear-48B | 122/128 @ 18.9 tok/s (0.90× vLLM) = best. **bf16 stream REFUTED (§20/#118: 122→4/128 KDA repeat-loop, no speed win); STRICT unreachable (p7 near-tie).** SERVER fold scoped (ARCH-ONE-SURFACE req4; runner aborts on Kimi KV `runner.cpp:489`) | ACTIVE: SERVER fold + last 0.10× |
 | 35B fresh grid | **BOUND** @`1ea26427`: 0.93-1.03x, c16 0.93x. INTAKE + Option A both NEGATIVE | Lever left: prefill glue (#61) |
 | Qwen3.5-4B revalidation | 0.9971x @`59674cf1` (#35); TTFT/PSS pass, TPOT/ITL open | `docs/bench-evidence/` |
 | MXFP4 parity | c1 1.020, c2-c8 0.962-0.969. **#82 CLOSED: ptxas-lineage REFUTED (A/B ties our+vLLM PTX all ptxas/JIT; +10us=engine context, not codegen)** | TERMINAL: at parity |

diff --git a/.agents/benchmark-record.md b/.agents/benchmark-record.md
@@ -19,6 +19,50 @@ from relative link targets repointed for this file's location.
 
 # Benchmarks
 
+## KIMI-BF16-STREAM — bf16 residual stream end-to-end REFUTED (122→4/128, KDA repeat-loop destabilization, no speed win); STRICT is NOT reachable by residual-precision (§14-§20 all closed); 122/128 @ 18.9 tok/s (0.90× vLLM) is the coherent best; SERVER runner fold scoped (runner aborts on Kimi's KV today) (2026-08-07, `row/KIMI-BF16-STREAM-CLOSE`, base `origin/main` `2f029a10`, GB10 sm_121a, PR #118)
+
+The #113 follow-on tested the §19-named residual #1 — the bf16 residual stream END-TO-END, framed as
+"the ONE lever both verdicts point at" (STRICT via vLLM bf16 rounding on p7 + speed via killing the 3%
+CastBf16). Implemented STRUCTURALLY (bf16 `DBuf`s for hidden/residual/normed-hidden/block-outputs via
+`vt::FusedChain(kFusedAddRmsNormStd)`, mirroring `deepseek_v2.cpp:479-615`; supersedes the partial §14
+RoundDevBf16 knob), gated behind `VT_KIMI_BF16_STREAM` (default OFF). Clean-from-`origin/main` `2f029a10`
+CUDA build (`-DVLLM_CPP_CUDA=ON -DVLLM_CPP_TRITON=ON -DVLLM_CPP_CUDA_ARCHITECTURES=121a
+-DVLLM_CPP_CUTLASS_DIR=…cutlass-4.5.0`, nvcc 13.0.88, Release), built in `/dev/shm`. CPU gate
+`test_kimi_linear_forward` 15/15·875 with the knob OFF (byte-identical); knob ON keeps the greedy-TOKEN
+state-carry check (incremental==recompute) but trips case-(l)'s 1e-5 logit tol (too tight for bf16).
+
+**GB10 full 48.9B 128-gate (single-load/config, `flock $HOME/gpu.lock`, `drop_caches`, memory-monitored,
+min-avail 18G, NO reboot; §12 golden md5 `bfa5bdbf…`; CONTROL reproduces §19's 122 EXACTLY, 3×):**
+
+| config | env (all `DEVICE_COMPUTE=1 DEVICE_KDA=1 DEVICE_KDA_CHUNK=1`, `--incremental`) | /128 | tok/s (steady) |
+|---|---|---|---|
+| CONTROL (f32 stream) | — | **122** | 18.9-19.0 |
+| **+bf16 stream** | `VT_KIMI_BF16_STREAM=1` | **4** | 19.8 |
+| bf16 stream, recompute+f64-island (diagnostic) | `VT_KIMI_BF16_STREAM=1` (no device-KDA, non-incremental) | **5** | 1.47 |
+
+- **REFUTED on BOTH axes.** bf16 residual REGRESSES 122→4/128: the KDA recurrence DESTABILIZES into
+  degenerate REPEAT LOOPS (p1 `15383,387,15383,387…`, p2 `220,16,25,…`, p4 `220,2466,25,…`) — the
+  §14/§15 "bf16 destabilizes KDA" pathology, confirmed STRUCTURALLY. **No speed win** (18.9→19.8, within
+  noise; the removed CastBf16 is a memory-bound decode's ~3% that overlaps the GEMVs, offset by the added
+  `ToStream` + bf16-norm-weight casts).
+- **Diagnostic (5/128 vs §14's f32-variance BF16_RESIDUAL=106):** the STRUCTURAL stream computes the
+  RMSNorm variance over the bf16-ROUNDED residual (vLLM's ACTUAL `fused_add_rms_norm` order,
+  `cpu_ops.cpp:326-332`) — MORE vLLM-faithful yet EVEN LESS stable than §14's f32-variance approximation.
+  Both bf16 variants sit far below the f32 control's 122 ⇒ the direction is dead in ALL variance
+  treatments.
+- **STRICT verdict, definitive:** with §14 (host-precision plateau 120), §15 (device-KDA 122), §16
+  (device-MLA 109), §18 (chunk-every-step 102), and now bf16-stream (4-5) — **p7 is an INTRINSIC near-tie;
+  122/128 @ 18.9 tok/s (0.90× vLLM) is Kimi-Linear's coherent best; STRICT is NOT reachable by residual-
+  precision or device-island approximation.** The only remaining STRICT/speed path is vLLM's ACTUAL fused
+  kernels via the full runner fold.
+- **SERVER runner fold (ARCH-ONE-SURFACE req 4): scoped, enabling-blocked.** The runner ABORTS on Kimi's
+  KV today — `VT_CHECK(mamba_spec->shapes == …)` at `runner.cpp:489-493` fails because Kimi lacks the
+  qwen3_5 `linear_*` config fields + `layer_types` (its KDA split lives in `linear_attn_config`). The fold
+  = synthesize layer_types + source GDN geometry from `linear_attn_config` + a Kimi KDA-paged block
+  (`KdaChunkPrefill`/`KdaGatedDeltaRule` over `gdn_state`) + a NoPE-MLA-paged block (`ForwardMlaAttentionBlock`
+  identity-RoPE) + bind in `ForwardDevice`; a multi-brick runner-touching integration (gate-model regression
+  risk), NOT landed this campaign. `VT_KIMI_BF16_STREAM` kept default-OFF as a documented-measured-negative.
+
 ## KIMI-PAGED-INCREMENTAL — paged-incremental decode LANDS the 5× speed win (4.23→18.9 tok/s, 0.20×→0.90× vLLM); Gate A token-identical to recompute; STRICT NOT reached (p7 intrinsic near-tie, 122/128); decode is 90% cuBLAS-GEMV-parity (2026-08-07, `row/KIMI-PAGED-INCREMENTAL`, base `origin/main` `68b394bc`, commit `f9ba4a9c`, GB10 sm_121a, PR #113)
 
 Full 48.9B GB10 gate (single-load/config, `flock $HOME/gpu.lock`, `drop_caches`, memory-monitored, min-avail 18-21 GiB, NO reboot) vs the §12 STRICT `greedy_ids.npy` (md5 `bfa5bdbf…`). vLLM ~21 stands from #111 (re-run only ours). The §18 lever (e) built: prefill-once (KDA recurrent+conv state carried, NoPE-MLA latent-KV cached) + recurrent decode-step; mirrors vLLM `kimi_gdn_linear_attn._forward` (prefill=`chunk_kda_with_fused_gate` / decode=`fused_recurrent_kda`) at `vllm-src` `a4e3cb4`.

diff --git a/.agents/specs/kimi-linear.md b/.agents/specs/kimi-linear.md
@@ -1359,6 +1359,109 @@ the last ~10% speed land.
 
 ---
 
+## 20. bf16-STREAM CLOSE + PRODUCTION RUNNER FOLD (2026-08-07, `row/KIMI-BF16-STREAM-CLOSE`, #113 follow-on)
+<!-- state: 2026-08-07 -->
+
+The #113 follow-on double-close: (1) the §19-named **bf16 residual stream end-to-end** (the ONE
+lever both verdicts point at — STRICT via matching vLLM's bf16 rounding on p7, speed via killing
+the 3% CastBf16 + f32↔bf16 round-trips) and (2) the **paged-FA2 MLA decode** (§16 residual d) —
+PLUS a coordinator-directed **production runner fold** (the born-on-runner MUST-route seam): fold
+Kimi's decode onto `ModelRegistry::Forward` so `/v1/completions` serves it at the fast rate, not
+just the `examples/kimi_linear_gen` CLI.
+
+### 20.1 bf16 residual stream (residual #1) — MEASURED-NEGATIVE / REFUTED (`VT_KIMI_BF16_STREAM`, default OFF)
+The partial §14 `VT_KIMI_BF16_RESIDUAL` knob (RoundDevBf16 rounding f32 STORAGE in place) was
+SUPERSEDED by a STRUCTURAL bf16 stream, mirroring `deepseek_v2.cpp:479-615`
+(`DeepseekV2Model::ForwardBody`/`RunLayer`, byte-exact vs the vLLM oracle): `hidden`/`res`/normed-
+`dhn`/block-outputs are bf16 `DBuf`s; `vt::FusedChain(kFusedAddRmsNormStd)` carries the bf16
+add+RMSNorm (the CPU/CUDA kernel rounds the residual store to bf16 and computes the RMSNorm variance
+over that bf16-rounded sum — vLLM's ACTUAL `fused_add_rms_norm` order, `cpu_ops.cpp:326-332`, NOT
+§14's f32-pre-store-sum). Impl (`kimi_linear_device.cpp`): `StreamDType()`/`Bf16Stream()`; `GemmBf16`
+elides the per-GEMM `CastBf16` when the act is already bf16; `AddRmsNormS` builds a LOSSLESS bf16 norm
+weight (CUDA `RmsNorm`/`FusedChain` require `weight.dtype==x.dtype`, cuda_ops.cu:452,3480); `ToStream`
+rounds each block output; the MoE per-expert gather strides in the stream dtype. Applied IDENTICALLY
+to `DeviceForwardBodyBf16` (recompute) + `…Incremental` (paged decode). CPU tiny gate: 15/15·875 with
+the knob OFF (byte-identical); with it ON, case-(l)'s 1e-5 logit tolerance trips (19 tiny assertions)
+but the greedy-TOKEN check (incremental==recompute) STILL PASSES — the state-carry wiring is correct,
+the 1e-5 tol is just too tight for bf16.
+
+**GB10 FULL 48.9B 128-gate — MEASURED NEGATIVE (single-load/config, `flock $HOME/gpu.lock`,
+`drop_caches`, min-avail 18G, NO reboot; §12 golden md5 `bfa5bdbf…`; CONTROL reproduces §19 EXACTLY
+3×):**
+
+| config | env (all `DEVICE_COMPUTE=1 DEVICE_KDA=1 DEVICE_KDA_CHUNK=1`, `--incremental`) | /128 | tok/s |
+|---|---|---|---|
+| CONTROL (f32 stream) | — | **122** | 18.9-19.0 |
+| **+bf16 stream** | `BF16_STREAM=1` | **4** | 19.8 |
+| bf16 stream, recompute+f64-island (diagnostic, no device-KDA, non-incremental) | `BF16_STREAM=1` | **5** | 1.47 |
+
+**REFUTED on BOTH axes.** The bf16 residual stream REGRESSES 122→4/128 — the KDA recurrence
+DESTABILIZES into degenerate REPEAT LOOPS (p1 `15383,387,15383,387…`, p2 `220,16,25,220,16,25…`,
+p4 `220,2466,25…`), the §14/§15 "bf16 destabilizes KDA" pathology, now confirmed STRUCTURALLY. The
+diagnostic (recompute+f64-island, the closest analog to §14's BF16_RESIDUAL=106) collapses to 5/128
+because the STRUCTURAL stream computes the RMSNorm variance over the bf16-ROUNDED residual (vLLM-
+faithful) — EVEN LESS stable than §14's f32-variance approximation; both bf16 variants sit far below
+the f32 control's 122. **No speed win:** 18.9→19.8 is within noise (the removed CastBf16 is a memory-
+bound decode's ~3% that overlaps the GEMVs; the added `ToStream` + bf16-norm-weight casts offset it).
+This REFUTES the §19 hypothesis that "a bf16 residual stream closes p7 AND wins speed." Combined with
+§14 (host-precision plateau 120), §15 (device-KDA 122), §16 (device-MLA 109), §18 (chunk-every-step
+102): **p7 is an INTRINSIC near-tie; 122/128 @ 18.9 tok/s (0.90× vLLM) is Kimi-Linear's coherent
+best; STRICT is NOT reachable by residual-precision OR device-island approximation — only by vLLM's
+ACTUAL fused kernels via the full runner fold (§20.3).** The knob STAYS as a documented-MEASURED-
+NEGATIVE A/B (default OFF), per the §14/§16 precedent.
+
+### 20.2 paged-FA2 MLA decode (residual #2) — the MLA HALF of the runner fold (§20.3c)
+The incremental `MlaSoftmaxIslandInc` host f64 softmax (D2H `dq` + H2D `out` per NoPE-MLA layer per
+step) is a per-step host round-trip (part of the ~15% decode host-idle, §19). The principled fix is
+the paged device decode attention over the runner's MLA `attn_kv` group — i.e. `ForwardMlaAttentionBlock`
+run IN the runner (§20.3c), NOT a CLI-only device MLA (the coordinator's "don't build more CLI-only
+machinery"). Grounded in DeepSeek-V2's MLA decode (`mla::ForwardMlaAttentionBlock` / `vt::MlaDecodeAttention`,
+the geometry cousin; Kimi dims §16: nah=32, qk=192, v=128, NoPE ⇒ identity RoPE, no q-lora, scale
+qk**-0.5). NOTE §16 already MEASURED the WHOLE-sequence `vt::Attention` approximation NEGATIVE
+(122→109), so the decode must use vLLM's ACTUAL FA2 tiling (the paged MLA op), not an approximation.
+
+### 20.3 production runner fold (coordinator directive; ARCH-ONE-SURFACE req 4) — SCOPED, ENABLING-BLOCKED
+`KimiLinearModel::ForwardDevice` (the runner forward, bound at `kimi_linear_registry.cpp:87`)
+`KimiLinearModel::ForwardDevice` (the runner forward, bound at `kimi_linear_registry.cpp:87`)
+today routes the full model to `ForwardDeviceCompute` — the O(n²) recompute (4.24 tok/s) — and
+`(void)attn_meta;(void)attn_kv;` (`kimi_linear_device.cpp`), so the SERVER serves at the slow
+rate; the fast paged-incremental path is CLI-only. The fold routes Kimi's decode through the
+runner's prefill/decode phase (`attn_meta`) + the het-KV groups the runner already declares
+(`MakeKimiLinearKVCache`: MLA latent `attn_kv` + KDA MambaSpec `gdn_state`). ENABLING PREREQUISITE
+(measured this campaign): Kimi's `config.json` has **NO `layer_types`** (the KDA/full-attn split
+lives in `linear_attn_config.{kda_layers,full_attn_layers}`), AND Kimi lacks the qwen3_5 `linear_num_
+key_heads`/`linear_key_head_dim`/… fields the runner derives GDN geometry from. So the runner **ABORTS
+on Kimi's KV setup TODAY**: with the KDA MambaSpec group declared, `gdn_group_id_>=0`, and the
+`VT_CHECK(mamba_spec->shapes == {conv_dim,conv_state_len},{Hv,Dv,Dk})` at **`runner.cpp:489-493`**
+compares Kimi's `{12288,3},{32,128,128}` against the config-derived `{0,0},{0,0,0}` → HARD FAIL.
+MEASURED-by-reading, not run (a server smoke would need the 91.5 GiB load). The fold's landing points
+(file:line): (a) synthesize `layer_types` + source the GDN geometry from `linear_attn_config`
+(`runner.cpp:464-493` + config parse) so the runner allocates the two groups without aborting;
+(b) a Kimi KDA-paged block (`vt::KdaChunkPrefill` prefill / `vt::KdaGatedDeltaRule` decode + conv
+gather/scatter over `gdn_state` slots keyed by `non_spec_state_indices` — NOT `GdnBlockPaged`,
+which is per-HEAD-scalar `vt::GdnDecode`); (c) a NoPE-MLA-paged block via `ForwardMlaAttentionBlock`
+(identity-RoPE, NoPE scale) over the paged MLA `attn_kv` (= residual #2 runner-side); (d) bind the
+paged forward in `ForwardDevice`. Gates: engine token-identity (paged-engine == CLI-incremental,
+then vs the §12 STRICT golden) + a `/v1/completions` server smoke (streamed, coherent, rate
+consistent with the CLI 18.9 tok/s). This is the `ARCH-ONE-SURFACE` req 4 (a capability is DONE only
+when `include/vllm.h` exposes it; the CLI is a thin client). It is a substantial multi-brick, runner-
+touching integration (the shared qwen3_5 GDN path — regression risk to the gate models) — NOT one-
+campaign-completable to production quality; SCOPED here as the named born-on-runner residual.
+
+### 20.4 Status — bf16-stream REFUTED; runner fold SCOPED (ARCH-ONE-SURFACE req 4)
+- **bf16 residual stream (20.1): MEASURED-NEGATIVE / REFUTED** on GB10 (122→4/128, repeat-loop
+  destabilization; no speed win). The §19 "bf16 stream closes p7 + wins speed" hypothesis is
+  refuted. Knob kept default-OFF, documented-measured-negative (§14/§16 precedent).
+- **STRICT (128/128): NOT reachable** by any residual-precision or device-island lever tried
+  (§14-§20). p7 is an INTRINSIC near-tie; **122/128 @ 18.9 tok/s (0.90× vLLM) is the coherent best.**
+- **SERVER runner fold (20.3): SCOPED, enabling-blocked** — the runner aborts on Kimi's KV today
+  (`runner.cpp:489-493`); the fold is the named born-on-runner residual with the file:line landing
+  points above. NOT landed this campaign (multi-brick, runner-touching, gate-model regression risk).
+- Row STAYS `ACTIVE` on the SERVER fold + the last 0.10× speed (both need vLLM's ACTUAL kernels via
+  the paged fold — the same lever, coupled). The STRICT thread is CLOSED as a definitive near-tie.
+
+---
+
 ## Structured contract (machine-readable — mirrors deepseek-v4-flash.md)
 
 ## Scope

diff --git a/.agents/state.md b/.agents/state.md
@@ -41465,3 +41465,35 @@ both named residuals). Harness `--incremental`; commit `f9ba4a9c` off `origin/ma
 - **Default:** `--incremental` opt-in; `VT_KIMI_DEVICE_KDA`/`_CHUNK` STAY OFF (≠ STRICT). Row STAYS
   ACTIVE. Next: bf16 residual stream (closes 3% CastBf16 + island round-trips AND the p7 near-tie) +
   paged-FA2 MLA decode. `local-ai-worker` parked during GPU, RESTORED at end. PR #113.
+
+## Kimi-Linear: bf16 residual stream REFUTED; STRICT is intrinsically unreachable; SERVER fold scoped (spec §20, PR #118)
+<!-- state: 2026-08-07T20:30 -->
+
+`row/KIMI-BF16-STREAM-CLOSE` (helper worktree, DRAFT PR #118), the #113 follow-on. Base `origin/main`
+`2f029a10`. Tested the §19-named residual #1 (bf16 residual stream END-TO-END) as the STRICT+speed lever.
+
+- **bf16 residual stream (residual #1) — MEASURED-NEGATIVE / REFUTED.** STRUCTURAL bf16 stream
+  (`hidden`/`res`/`dhn`/block-outputs bf16, `vt::FusedChain(kFusedAddRmsNormStd)`, mirroring
+  `deepseek_v2.cpp:479-615`; supersedes the partial §14 RoundDevBf16), gated `VT_KIMI_BF16_STREAM`
+  default OFF. Clean GB10 CUDA build (/dev/shm, nvcc 13.0.88), CPU gate 15/15·875 knob-OFF. **GB10 full
+  48.9B 128-gate (flock, drop_caches, min-avail 18G, no reboot; golden `bfa5bdbf…`):** CONTROL (f32)
+  **122/128 @ 18.9 tok/s** (reproduces §19 3×); **+bf16 stream 4/128 @ 19.8** — KDA recurrence
+  DESTABILIZES into degenerate repeat loops (`15383,387,15383,387…`), the §14/§15 pathology confirmed
+  structurally; NO speed win. Diagnostic (bf16 recompute+f64-island) **5/128** vs §14's f32-variance
+  BF16_RESIDUAL=106 — the structural stream computes the RMSNorm variance over the bf16-rounded residual
+  (vLLM-faithful, `cpu_ops.cpp:326-332`), even less stable. Both bf16 variants ≪ the f32 control's 122.
+  REFUTES the §19 "bf16 stream closes p7 + wins speed" hypothesis. Knob kept default-OFF, documented-
+  measured-negative (§14/§16 precedent).
+- **STRICT (128/128) — DEFINITIVELY NOT reachable** by residual-precision or device-island levers
+  (§14 plateau 120, §15 device-KDA 122, §16 device-MLA 109, §18 chunk-every-step 102, §20 bf16 4-5).
+  **p7 is an INTRINSIC near-tie; 122/128 @ 18.9 tok/s (0.90× vLLM) is Kimi's coherent best.** The STRICT
+  thread is CLOSED as a near-tie; only vLLM's ACTUAL fused kernels (the paged fold) could move it.
+- **SERVER runner fold (ARCH-ONE-SURFACE req 4) — SCOPED, enabling-blocked.** The runner ABORTS on
+  Kimi's KV today: `VT_CHECK(mamba_spec->shapes==…)` at `runner.cpp:489-493` fails (Kimi lacks the
+  qwen3_5 `linear_*` fields + `layer_types`; its split is in `linear_attn_config`). Fold = synthesize
+  layer_types + source GDN geometry from `linear_attn_config` + a Kimi KDA-paged block
+  (`KdaChunkPrefill`/`KdaGatedDeltaRule` over `gdn_state`, NOT per-head `GdnBlockPaged`) + a NoPE-MLA-paged
+  block (`ForwardMlaAttentionBlock` identity-RoPE) + bind in `ForwardDevice`. Multi-brick, runner-touching
+  (gate-model regression risk) — NOT landed this campaign; named born-on-runner residual, spec §20.3.
+- Row STAYS `ACTIVE` on the SERVER fold + the last 0.10× speed. Worker was PARKED during GPU work;
+  RESTORE at campaign end.