diff --git a/.agents/NOW.md b/.agents/NOW.md index 8e1a47f4..7b07420b 100644 --- a/.agents/NOW.md +++ b/.agents/NOW.md @@ -18,7 +18,7 @@ Working head: `row/backend-rocm-w0` (#41). Prior: benchmark checkpoint | f32-out GEMV audit | Only laguna + ds4 bf16 tower affected; gate models unaffected | Re-verify ds4 tower same-tool | | Invocation-parity prevention | CI guard + checklist landing | Merge; build-verify `kGemvHeuristicAlgos` on dgx | | MiniMax-H3 lane | **bf16 shards STREAM both towers (DiT + encoder); Q4_K_M enc cond cos 0.9975, 3.5° med, DIFFUSE** | render A/B on saved embeds | -| Kimi-Linear-48B | **Paged-incremental decode LANDS (#113): 18.9 tok/s, 4.5× over recompute, 0.90× vLLM ~21** (gap 0.20×→0.90×). Gate A token-identical to recompute (122/128); NOT STRICT (p7 near-tie); decode 90% cuBLAS-GEMV-parity | Next: bf16 residual stream (p7+cast) + paged-FA2 MLA | +| Kimi-Linear-48B | 122/128 @ 18.9 tok/s (0.90× vLLM) = best. **bf16 stream REFUTED (§20/#118: 122→4/128 KDA repeat-loop, no speed win); STRICT unreachable (p7 near-tie).** SERVER fold scoped (ARCH-ONE-SURFACE req4; runner aborts on Kimi KV `runner.cpp:489`) | ACTIVE: SERVER fold + last 0.10× | | 35B fresh grid | **BOUND** @`1ea26427`: 0.93-1.03x, c16 0.93x. INTAKE + Option A both NEGATIVE | Lever left: prefill glue (#61) | | Qwen3.5-4B revalidation | 0.9971x @`59674cf1` (#35); TTFT/PSS pass, TPOT/ITL open | `docs/bench-evidence/` | | MXFP4 parity | c1 1.020, c2-c8 0.962-0.969. **#82 CLOSED: ptxas-lineage REFUTED (A/B ties our+vLLM PTX all ptxas/JIT; +10us=engine context, not codegen)** | TERMINAL: at parity | diff --git a/.agents/benchmark-record.md b/.agents/benchmark-record.md index 7d93c97d..c6adb339 100644 --- a/.agents/benchmark-record.md +++ b/.agents/benchmark-record.md @@ -19,6 +19,50 @@ from relative link targets repointed for this file's location. # Benchmarks +## KIMI-BF16-STREAM — bf16 residual stream end-to-end REFUTED (122→4/128, KDA repeat-loop destabilization, no speed win); STRICT is NOT reachable by residual-precision (§14-§20 all closed); 122/128 @ 18.9 tok/s (0.90× vLLM) is the coherent best; SERVER runner fold scoped (runner aborts on Kimi's KV today) (2026-08-07, `row/KIMI-BF16-STREAM-CLOSE`, base `origin/main` `2f029a10`, GB10 sm_121a, PR #118) + +The #113 follow-on tested the §19-named residual #1 — the bf16 residual stream END-TO-END, framed as +"the ONE lever both verdicts point at" (STRICT via vLLM bf16 rounding on p7 + speed via killing the 3% +CastBf16). Implemented STRUCTURALLY (bf16 `DBuf`s for hidden/residual/normed-hidden/block-outputs via +`vt::FusedChain(kFusedAddRmsNormStd)`, mirroring `deepseek_v2.cpp:479-615`; supersedes the partial §14 +RoundDevBf16 knob), gated behind `VT_KIMI_BF16_STREAM` (default OFF). Clean-from-`origin/main` `2f029a10` +CUDA build (`-DVLLM_CPP_CUDA=ON -DVLLM_CPP_TRITON=ON -DVLLM_CPP_CUDA_ARCHITECTURES=121a +-DVLLM_CPP_CUTLASS_DIR=…cutlass-4.5.0`, nvcc 13.0.88, Release), built in `/dev/shm`. CPU gate +`test_kimi_linear_forward` 15/15·875 with the knob OFF (byte-identical); knob ON keeps the greedy-TOKEN +state-carry check (incremental==recompute) but trips case-(l)'s 1e-5 logit tol (too tight for bf16). + +**GB10 full 48.9B 128-gate (single-load/config, `flock $HOME/gpu.lock`, `drop_caches`, memory-monitored, +min-avail 18G, NO reboot; §12 golden md5 `bfa5bdbf…`; CONTROL reproduces §19's 122 EXACTLY, 3×):** + +| config | env (all `DEVICE_COMPUTE=1 DEVICE_KDA=1 DEVICE_KDA_CHUNK=1`, `--incremental`) | /128 | tok/s (steady) | +|---|---|---|---| +| CONTROL (f32 stream) | — | **122** | 18.9-19.0 | +| **+bf16 stream** | `VT_KIMI_BF16_STREAM=1` | **4** | 19.8 | +| bf16 stream, recompute+f64-island (diagnostic) | `VT_KIMI_BF16_STREAM=1` (no device-KDA, non-incremental) | **5** | 1.47 | + +- **REFUTED on BOTH axes.** bf16 residual REGRESSES 122→4/128: the KDA recurrence DESTABILIZES into + degenerate REPEAT LOOPS (p1 `15383,387,15383,387…`, p2 `220,16,25,…`, p4 `220,2466,25,…`) — the + §14/§15 "bf16 destabilizes KDA" pathology, confirmed STRUCTURALLY. **No speed win** (18.9→19.8, within + noise; the removed CastBf16 is a memory-bound decode's ~3% that overlaps the GEMVs, offset by the added + `ToStream` + bf16-norm-weight casts). +- **Diagnostic (5/128 vs §14's f32-variance BF16_RESIDUAL=106):** the STRUCTURAL stream computes the + RMSNorm variance over the bf16-ROUNDED residual (vLLM's ACTUAL `fused_add_rms_norm` order, + `cpu_ops.cpp:326-332`) — MORE vLLM-faithful yet EVEN LESS stable than §14's f32-variance approximation. + Both bf16 variants sit far below the f32 control's 122 ⇒ the direction is dead in ALL variance + treatments. +- **STRICT verdict, definitive:** with §14 (host-precision plateau 120), §15 (device-KDA 122), §16 + (device-MLA 109), §18 (chunk-every-step 102), and now bf16-stream (4-5) — **p7 is an INTRINSIC near-tie; + 122/128 @ 18.9 tok/s (0.90× vLLM) is Kimi-Linear's coherent best; STRICT is NOT reachable by residual- + precision or device-island approximation.** The only remaining STRICT/speed path is vLLM's ACTUAL fused + kernels via the full runner fold. +- **SERVER runner fold (ARCH-ONE-SURFACE req 4): scoped, enabling-blocked.** The runner ABORTS on Kimi's + KV today — `VT_CHECK(mamba_spec->shapes == …)` at `runner.cpp:489-493` fails because Kimi lacks the + qwen3_5 `linear_*` config fields + `layer_types` (its KDA split lives in `linear_attn_config`). The fold + = synthesize layer_types + source GDN geometry from `linear_attn_config` + a Kimi KDA-paged block + (`KdaChunkPrefill`/`KdaGatedDeltaRule` over `gdn_state`) + a NoPE-MLA-paged block (`ForwardMlaAttentionBlock` + identity-RoPE) + bind in `ForwardDevice`; a multi-brick runner-touching integration (gate-model regression + risk), NOT landed this campaign. `VT_KIMI_BF16_STREAM` kept default-OFF as a documented-measured-negative. + ## KIMI-PAGED-INCREMENTAL — paged-incremental decode LANDS the 5× speed win (4.23→18.9 tok/s, 0.20×→0.90× vLLM); Gate A token-identical to recompute; STRICT NOT reached (p7 intrinsic near-tie, 122/128); decode is 90% cuBLAS-GEMV-parity (2026-08-07, `row/KIMI-PAGED-INCREMENTAL`, base `origin/main` `68b394bc`, commit `f9ba4a9c`, GB10 sm_121a, PR #113) Full 48.9B GB10 gate (single-load/config, `flock $HOME/gpu.lock`, `drop_caches`, memory-monitored, min-avail 18-21 GiB, NO reboot) vs the §12 STRICT `greedy_ids.npy` (md5 `bfa5bdbf…`). vLLM ~21 stands from #111 (re-run only ours). The §18 lever (e) built: prefill-once (KDA recurrent+conv state carried, NoPE-MLA latent-KV cached) + recurrent decode-step; mirrors vLLM `kimi_gdn_linear_attn._forward` (prefill=`chunk_kda_with_fused_gate` / decode=`fused_recurrent_kda`) at `vllm-src` `a4e3cb4`. diff --git a/.agents/specs/kimi-linear.md b/.agents/specs/kimi-linear.md index 29f6ae03..d61a6d72 100644 --- a/.agents/specs/kimi-linear.md +++ b/.agents/specs/kimi-linear.md @@ -1359,6 +1359,109 @@ the last ~10% speed land. --- +## 20. bf16-STREAM CLOSE + PRODUCTION RUNNER FOLD (2026-08-07, `row/KIMI-BF16-STREAM-CLOSE`, #113 follow-on) + + +The #113 follow-on double-close: (1) the §19-named **bf16 residual stream end-to-end** (the ONE +lever both verdicts point at — STRICT via matching vLLM's bf16 rounding on p7, speed via killing +the 3% CastBf16 + f32↔bf16 round-trips) and (2) the **paged-FA2 MLA decode** (§16 residual d) — +PLUS a coordinator-directed **production runner fold** (the born-on-runner MUST-route seam): fold +Kimi's decode onto `ModelRegistry::Forward` so `/v1/completions` serves it at the fast rate, not +just the `examples/kimi_linear_gen` CLI. + +### 20.1 bf16 residual stream (residual #1) — MEASURED-NEGATIVE / REFUTED (`VT_KIMI_BF16_STREAM`, default OFF) +The partial §14 `VT_KIMI_BF16_RESIDUAL` knob (RoundDevBf16 rounding f32 STORAGE in place) was +SUPERSEDED by a STRUCTURAL bf16 stream, mirroring `deepseek_v2.cpp:479-615` +(`DeepseekV2Model::ForwardBody`/`RunLayer`, byte-exact vs the vLLM oracle): `hidden`/`res`/normed- +`dhn`/block-outputs are bf16 `DBuf`s; `vt::FusedChain(kFusedAddRmsNormStd)` carries the bf16 +add+RMSNorm (the CPU/CUDA kernel rounds the residual store to bf16 and computes the RMSNorm variance +over that bf16-rounded sum — vLLM's ACTUAL `fused_add_rms_norm` order, `cpu_ops.cpp:326-332`, NOT +§14's f32-pre-store-sum). Impl (`kimi_linear_device.cpp`): `StreamDType()`/`Bf16Stream()`; `GemmBf16` +elides the per-GEMM `CastBf16` when the act is already bf16; `AddRmsNormS` builds a LOSSLESS bf16 norm +weight (CUDA `RmsNorm`/`FusedChain` require `weight.dtype==x.dtype`, cuda_ops.cu:452,3480); `ToStream` +rounds each block output; the MoE per-expert gather strides in the stream dtype. Applied IDENTICALLY +to `DeviceForwardBodyBf16` (recompute) + `…Incremental` (paged decode). CPU tiny gate: 15/15·875 with +the knob OFF (byte-identical); with it ON, case-(l)'s 1e-5 logit tolerance trips (19 tiny assertions) +but the greedy-TOKEN check (incremental==recompute) STILL PASSES — the state-carry wiring is correct, +the 1e-5 tol is just too tight for bf16. + +**GB10 FULL 48.9B 128-gate — MEASURED NEGATIVE (single-load/config, `flock $HOME/gpu.lock`, +`drop_caches`, min-avail 18G, NO reboot; §12 golden md5 `bfa5bdbf…`; CONTROL reproduces §19 EXACTLY +3×):** + +| config | env (all `DEVICE_COMPUTE=1 DEVICE_KDA=1 DEVICE_KDA_CHUNK=1`, `--incremental`) | /128 | tok/s | +|---|---|---|---| +| CONTROL (f32 stream) | — | **122** | 18.9-19.0 | +| **+bf16 stream** | `BF16_STREAM=1` | **4** | 19.8 | +| bf16 stream, recompute+f64-island (diagnostic, no device-KDA, non-incremental) | `BF16_STREAM=1` | **5** | 1.47 | + +**REFUTED on BOTH axes.** The bf16 residual stream REGRESSES 122→4/128 — the KDA recurrence +DESTABILIZES into degenerate REPEAT LOOPS (p1 `15383,387,15383,387…`, p2 `220,16,25,220,16,25…`, +p4 `220,2466,25…`), the §14/§15 "bf16 destabilizes KDA" pathology, now confirmed STRUCTURALLY. The +diagnostic (recompute+f64-island, the closest analog to §14's BF16_RESIDUAL=106) collapses to 5/128 +because the STRUCTURAL stream computes the RMSNorm variance over the bf16-ROUNDED residual (vLLM- +faithful) — EVEN LESS stable than §14's f32-variance approximation; both bf16 variants sit far below +the f32 control's 122. **No speed win:** 18.9→19.8 is within noise (the removed CastBf16 is a memory- +bound decode's ~3% that overlaps the GEMVs; the added `ToStream` + bf16-norm-weight casts offset it). +This REFUTES the §19 hypothesis that "a bf16 residual stream closes p7 AND wins speed." Combined with +§14 (host-precision plateau 120), §15 (device-KDA 122), §16 (device-MLA 109), §18 (chunk-every-step +102): **p7 is an INTRINSIC near-tie; 122/128 @ 18.9 tok/s (0.90× vLLM) is Kimi-Linear's coherent +best; STRICT is NOT reachable by residual-precision OR device-island approximation — only by vLLM's +ACTUAL fused kernels via the full runner fold (§20.3).** The knob STAYS as a documented-MEASURED- +NEGATIVE A/B (default OFF), per the §14/§16 precedent. + +### 20.2 paged-FA2 MLA decode (residual #2) — the MLA HALF of the runner fold (§20.3c) +The incremental `MlaSoftmaxIslandInc` host f64 softmax (D2H `dq` + H2D `out` per NoPE-MLA layer per +step) is a per-step host round-trip (part of the ~15% decode host-idle, §19). The principled fix is +the paged device decode attention over the runner's MLA `attn_kv` group — i.e. `ForwardMlaAttentionBlock` +run IN the runner (§20.3c), NOT a CLI-only device MLA (the coordinator's "don't build more CLI-only +machinery"). Grounded in DeepSeek-V2's MLA decode (`mla::ForwardMlaAttentionBlock` / `vt::MlaDecodeAttention`, +the geometry cousin; Kimi dims §16: nah=32, qk=192, v=128, NoPE ⇒ identity RoPE, no q-lora, scale +qk**-0.5). NOTE §16 already MEASURED the WHOLE-sequence `vt::Attention` approximation NEGATIVE +(122→109), so the decode must use vLLM's ACTUAL FA2 tiling (the paged MLA op), not an approximation. + +### 20.3 production runner fold (coordinator directive; ARCH-ONE-SURFACE req 4) — SCOPED, ENABLING-BLOCKED +`KimiLinearModel::ForwardDevice` (the runner forward, bound at `kimi_linear_registry.cpp:87`) +`KimiLinearModel::ForwardDevice` (the runner forward, bound at `kimi_linear_registry.cpp:87`) +today routes the full model to `ForwardDeviceCompute` — the O(n²) recompute (4.24 tok/s) — and +`(void)attn_meta;(void)attn_kv;` (`kimi_linear_device.cpp`), so the SERVER serves at the slow +rate; the fast paged-incremental path is CLI-only. The fold routes Kimi's decode through the +runner's prefill/decode phase (`attn_meta`) + the het-KV groups the runner already declares +(`MakeKimiLinearKVCache`: MLA latent `attn_kv` + KDA MambaSpec `gdn_state`). ENABLING PREREQUISITE +(measured this campaign): Kimi's `config.json` has **NO `layer_types`** (the KDA/full-attn split +lives in `linear_attn_config.{kda_layers,full_attn_layers}`), AND Kimi lacks the qwen3_5 `linear_num_ +key_heads`/`linear_key_head_dim`/… fields the runner derives GDN geometry from. So the runner **ABORTS +on Kimi's KV setup TODAY**: with the KDA MambaSpec group declared, `gdn_group_id_>=0`, and the +`VT_CHECK(mamba_spec->shapes == {conv_dim,conv_state_len},{Hv,Dv,Dk})` at **`runner.cpp:489-493`** +compares Kimi's `{12288,3},{32,128,128}` against the config-derived `{0,0},{0,0,0}` → HARD FAIL. +MEASURED-by-reading, not run (a server smoke would need the 91.5 GiB load). The fold's landing points +(file:line): (a) synthesize `layer_types` + source the GDN geometry from `linear_attn_config` +(`runner.cpp:464-493` + config parse) so the runner allocates the two groups without aborting; +(b) a Kimi KDA-paged block (`vt::KdaChunkPrefill` prefill / `vt::KdaGatedDeltaRule` decode + conv +gather/scatter over `gdn_state` slots keyed by `non_spec_state_indices` — NOT `GdnBlockPaged`, +which is per-HEAD-scalar `vt::GdnDecode`); (c) a NoPE-MLA-paged block via `ForwardMlaAttentionBlock` +(identity-RoPE, NoPE scale) over the paged MLA `attn_kv` (= residual #2 runner-side); (d) bind the +paged forward in `ForwardDevice`. Gates: engine token-identity (paged-engine == CLI-incremental, +then vs the §12 STRICT golden) + a `/v1/completions` server smoke (streamed, coherent, rate +consistent with the CLI 18.9 tok/s). This is the `ARCH-ONE-SURFACE` req 4 (a capability is DONE only +when `include/vllm.h` exposes it; the CLI is a thin client). It is a substantial multi-brick, runner- +touching integration (the shared qwen3_5 GDN path — regression risk to the gate models) — NOT one- +campaign-completable to production quality; SCOPED here as the named born-on-runner residual. + +### 20.4 Status — bf16-stream REFUTED; runner fold SCOPED (ARCH-ONE-SURFACE req 4) +- **bf16 residual stream (20.1): MEASURED-NEGATIVE / REFUTED** on GB10 (122→4/128, repeat-loop + destabilization; no speed win). The §19 "bf16 stream closes p7 + wins speed" hypothesis is + refuted. Knob kept default-OFF, documented-measured-negative (§14/§16 precedent). +- **STRICT (128/128): NOT reachable** by any residual-precision or device-island lever tried + (§14-§20). p7 is an INTRINSIC near-tie; **122/128 @ 18.9 tok/s (0.90× vLLM) is the coherent best.** +- **SERVER runner fold (20.3): SCOPED, enabling-blocked** — the runner aborts on Kimi's KV today + (`runner.cpp:489-493`); the fold is the named born-on-runner residual with the file:line landing + points above. NOT landed this campaign (multi-brick, runner-touching, gate-model regression risk). +- Row STAYS `ACTIVE` on the SERVER fold + the last 0.10× speed (both need vLLM's ACTUAL kernels via + the paged fold — the same lever, coupled). The STRICT thread is CLOSED as a definitive near-tie. + +--- + ## Structured contract (machine-readable — mirrors deepseek-v4-flash.md) ## Scope diff --git a/.agents/state.md b/.agents/state.md index c95e0d1c..e8792801 100644 --- a/.agents/state.md +++ b/.agents/state.md @@ -41465,3 +41465,35 @@ both named residuals). Harness `--incremental`; commit `f9ba4a9c` off `origin/ma - **Default:** `--incremental` opt-in; `VT_KIMI_DEVICE_KDA`/`_CHUNK` STAY OFF (≠ STRICT). Row STAYS ACTIVE. Next: bf16 residual stream (closes 3% CastBf16 + island round-trips AND the p7 near-tie) + paged-FA2 MLA decode. `local-ai-worker` parked during GPU, RESTORED at end. PR #113. + +## Kimi-Linear: bf16 residual stream REFUTED; STRICT is intrinsically unreachable; SERVER fold scoped (spec §20, PR #118) + + +`row/KIMI-BF16-STREAM-CLOSE` (helper worktree, DRAFT PR #118), the #113 follow-on. Base `origin/main` +`2f029a10`. Tested the §19-named residual #1 (bf16 residual stream END-TO-END) as the STRICT+speed lever. + +- **bf16 residual stream (residual #1) — MEASURED-NEGATIVE / REFUTED.** STRUCTURAL bf16 stream + (`hidden`/`res`/`dhn`/block-outputs bf16, `vt::FusedChain(kFusedAddRmsNormStd)`, mirroring + `deepseek_v2.cpp:479-615`; supersedes the partial §14 RoundDevBf16), gated `VT_KIMI_BF16_STREAM` + default OFF. Clean GB10 CUDA build (/dev/shm, nvcc 13.0.88), CPU gate 15/15·875 knob-OFF. **GB10 full + 48.9B 128-gate (flock, drop_caches, min-avail 18G, no reboot; golden `bfa5bdbf…`):** CONTROL (f32) + **122/128 @ 18.9 tok/s** (reproduces §19 3×); **+bf16 stream 4/128 @ 19.8** — KDA recurrence + DESTABILIZES into degenerate repeat loops (`15383,387,15383,387…`), the §14/§15 pathology confirmed + structurally; NO speed win. Diagnostic (bf16 recompute+f64-island) **5/128** vs §14's f32-variance + BF16_RESIDUAL=106 — the structural stream computes the RMSNorm variance over the bf16-rounded residual + (vLLM-faithful, `cpu_ops.cpp:326-332`), even less stable. Both bf16 variants ≪ the f32 control's 122. + REFUTES the §19 "bf16 stream closes p7 + wins speed" hypothesis. Knob kept default-OFF, documented- + measured-negative (§14/§16 precedent). +- **STRICT (128/128) — DEFINITIVELY NOT reachable** by residual-precision or device-island levers + (§14 plateau 120, §15 device-KDA 122, §16 device-MLA 109, §18 chunk-every-step 102, §20 bf16 4-5). + **p7 is an INTRINSIC near-tie; 122/128 @ 18.9 tok/s (0.90× vLLM) is Kimi's coherent best.** The STRICT + thread is CLOSED as a near-tie; only vLLM's ACTUAL fused kernels (the paged fold) could move it. +- **SERVER runner fold (ARCH-ONE-SURFACE req 4) — SCOPED, enabling-blocked.** The runner ABORTS on + Kimi's KV today: `VT_CHECK(mamba_spec->shapes==…)` at `runner.cpp:489-493` fails (Kimi lacks the + qwen3_5 `linear_*` fields + `layer_types`; its split is in `linear_attn_config`). Fold = synthesize + layer_types + source GDN geometry from `linear_attn_config` + a Kimi KDA-paged block + (`KdaChunkPrefill`/`KdaGatedDeltaRule` over `gdn_state`, NOT per-head `GdnBlockPaged`) + a NoPE-MLA-paged + block (`ForwardMlaAttentionBlock` identity-RoPE) + bind in `ForwardDevice`. Multi-brick, runner-touching + (gate-model regression risk) — NOT landed this campaign; named born-on-runner residual, spec §20.3. +- Row STAYS `ACTIVE` on the SERVER fold + the last 0.10× speed. Worker was PARKED during GPU work; + RESTORE at campaign end. diff --git a/docs/BENCHMARKS.md b/docs/BENCHMARKS.md index 5ddb2007..450735df 100644 --- a/docs/BENCHMARKS.md +++ b/docs/BENCHMARKS.md @@ -314,7 +314,7 @@ built on it rather than keeping the flattering one. | Multimodal image, audio, video | Correctness gated, speed unmeasured | Per-modality speed grids | | `/v1/videos` OpenAI (Sora) shape | **No number owed**: a CPU serving-surface change (request aliases, the MP4 content route, and reference conditioning wiring), unit-gated only, no kernel or generation path touched | Video generation speed stays the MiniMax-H3 FP4 row below | | Qwen3-dense decode CUDA-graph | Token-exact pass, ~4.3% e2e directional | Steady-state per-step tok/s | -| Kimi-Linear-48B-A3B (KDA+MLA+MoE) | **Paged-incremental decode (§19) MEASURED GB10: 18.9 tok/s** (4.5× over recompute 4.23), Gate A token-identical to recompute (122/128), NOT STRICT (p7 near-tie), OFF | vLLM ~21 tok/s (16-tok aggregate floor); ours-incremental steady **0.90× vLLM**, the MEASURED 5× decode gap (0.20×, #111) closed to ~1.1×; residual = per-step projection GEMVs + host orchestration | +| Kimi-Linear-48B-A3B (KDA+MLA+MoE) | **Paged-incremental (§19) GB10: 18.9 tok/s @ 122/128 = coherent best.** bf16 stream REFUTED (§20/#118: 122→4/128 KDA repeat-loop, no speed win); STRICT unreachable (§14-§20 levers closed), p7 near-tie | vLLM ~21 (16-tok aggregate floor); ours **0.90× vLLM**. Last 0.10× + STRICT need vLLM's real kernels via the SERVER fold (scoped) | | vLLM 0.26 re-benchmark | Pending | Re-run the binding grids on the advanced pin | | MiniMax-H3 FP4 speed (W-FP4a) | **Measured GB10 (`row/H3-FP4-GPU-E2E`).** Marlin W4A16 byte-exact vs bf16; fp4 a memory win, 0.8x bf16/forward. Real-ckpt fp4-resident e2e RUNS (mp4/wav) | fp4 speed CLOSED. bf16-vs-quant A/B: ENCODER half MEASURED (§8.15), DiT half NOT (no bf16 render exists). Detail: benchmark-record + spec §8 | | MiniMax-H3 render coherence (`row/H3-RENDER-CLOSE` #77) | **CLOSED: a COHERENT scene on GB10.** #70/#74 white was wrong-PARTITION usage (t2va on the ref2va ckpt); t2va on the FL2VA GGUF renders a prompt-matched orange cat (adj-cos 0.95 vs 0.06, no patch-grid) | Verified first: t2va inputs byte-exact vs upstream; CUDA device==host at seq 1920. Follow-up `H3-TASK-PARTITION-GUARD`: the task/partition mismatch now RAISES 1:1 with `_resolve_task` (spec §8.6-8.7) | diff --git a/docs/ENVIRONMENT.md b/docs/ENVIRONMENT.md index a64b0723..28478e45 100644 --- a/docs/ENVIRONMENT.md +++ b/docs/ENVIRONMENT.md @@ -65,6 +65,7 @@ portable/reference path. In normal operation leave them unset. | `VT_MM_DECODE_EAGER` | off (graph on) | Set to `1` to force the eager per-step multimodal (Qwen3.6-27B image/video) decode instead of routing it through the captured dense decode graph. Rollback / A-B knob; the graphed path is token-exact with the eager path | | `VT_KIMI_DEVICE_COMPUTE` | off (opt-in) | `=1` routes the Kimi-Linear-48B-A3B runner path (`KimiLinearModel::ForwardDevice`) through the W7 DBuf-resident device COMPUTE (`ForwardDeviceCompute`, the whole KDA/NoPE-MLA + MoE hybrid over pooled DBufs via the shared vt:: ops) instead of the default W6 host-reference compose. Default OFF keeps the CPU-verified host-ref-compose seam as production until the device compute is GPU-verified against the SACRED oracle; the device compute is CPU-gated (`test_kimi_linear_forward`, device==W2 reference within f32-accumulation tolerance, greedy-token-identical) but its GPU numerics are a NAMED pending. The flag exists so the device path CAN be exercised as the runner path for that verification | | `VT_KIMI_BF16_RESIDUAL` | off (opt-in) | `=1` carries the Kimi-Linear device-compute residual stream in bf16 like vLLM's `fused_add_rms_norm` (residual/block-outputs bf16, RMSNorm variance over the f32 pre-store sum), via in-place f32→bf16→f32 rounds. W7-speed STRICT-lever A/B (spec §14). Default OFF → byte-identical. MEASURED: alone net-zero; WITH `VT_KIMI_BF16_ISLANDS` → 120/128 (best, still a near-tie, NOT STRICT) | +| `VT_KIMI_BF16_STREAM` | off (opt-in) | `=1` carries the Kimi-Linear device-compute inter-layer residual stream in bf16 END-TO-END (STRUCTURAL bf16 `DBuf`s for hidden/residual/normed-hidden/block-outputs via `vt::FusedChain(kFusedAddRmsNormStd)`, mirroring `deepseek_v2.cpp:479-615`), SUPERSEDING the partial `VT_KIMI_BF16_RESIDUAL` in-place-rounding knob; elides the per-GEMM `CastBf16` on residual-fed projections. **MEASURED NEGATIVE / REFUTED** (spec §20, GB10 full 48.9B 128-gate): REGRESSES 122→4/128 — the bf16 residual rounding DESTABILIZES the KDA recurrence into degenerate repeat loops (the §14/§15 pathology, confirmed structurally), with NO speed win (18.9→19.8, within noise). The p7-STRICT close is NOT reachable by this or any residual-precision lever; kept as a documented-measured-negative A/B (default OFF → byte-identical to the f32 stream), per the `VT_KIMI_DEVICE_MLA`/`VT_KIMI_ISLAND_F32ACC` precedent | | `VT_KIMI_BF16_ISLANDS` | off (opt-in) | `=1` rounds the Kimi-Linear host-fallback island INPUTS (KDA q/k/v/g1/beta, NoPE-MLA q/kv/kpe) to bf16 (RNE) before the recurrence/softmax, toward vLLM's GDN-Triton/FA2 kernel precision. W7-speed STRICT-lever A/B (spec §14). Default OFF → byte-identical. MEASURED best config paired with `VT_KIMI_BF16_RESIDUAL` (106→120/128) | | `VT_KIMI_ISLAND_F32ACC` | off (opt-in) | `=1` computes the Kimi-Linear island recurrence/softmax in f32 accumulation (not f64). W7-speed A/B knob, **MEASURED NEGATIVE** (91–106/128; kept as a documented-negative A/B, spec §14). Default OFF → byte-identical | | `VT_KIMI_DEVICE_KDA` | off (opt-in) | `=1` runs the Kimi-Linear KDA per-k-channel gated-delta RECURRENCE through the net-new device op `vt::KdaGatedDeltaRule` (`cuda_gdn.cu` `KdaScanKernel`, the per-channel-decay `g[T,H,D]` variant of GDN's per-head-scalar decay; FLA `fused_recurrent_gated_delta_rule_fwd_kernel` IS_KDA=True) instead of the f64 host recompute. The principled path to STRICT AND the speed lever (spec §14/§15): the recurrence runs vLLM's actual f32-on-bf16 arithmetic on device rather than a host f64 recompute that is more precise than vLLM and coin-flips near-ties. Requires `VT_KIMI_DEVICE_COMPUTE=1`; the elementwise decay gate + beta stay host. Unit-gated (`test_ops_kda_recurrence`: broadcast-g == `vt::GdnPrefill` bit-identical, distinct-per-channel vs f64 ref, CPU↔CUDA parity). Default OFF (parity-enabler: flip only with the token gate green) → byte-identical | diff --git a/docs/FEATURES.md b/docs/FEATURES.md index 2dbc808e..506a154b 100644 --- a/docs/FEATURES.md +++ b/docs/FEATURES.md @@ -119,7 +119,7 @@ speed-pending, which [BENCHMARKS.md](BENCHMARKS.md) tracks. | `Glm4ForCausalLM` | GLM-4-9B-0414 | near-tie 16/16 vs vLLM 0.25.0 | pending | | `Glm4MoeLiteForCausalLM` | zai-org/GLM-4.7-Flash (31.2B, MLA MoE) | near-tie 8/8 vs vLLM 0.25.0 | pending | | `LagunaForCausalLM` | poolside/Laguna-S-2.1-NVFP4, GGUF-Q4_K, Laguna-XS | byte-exact near-tie (distributional vs vLLM) | vLLM parity+ 1.03x, default on | -| `KimiLinearForCausalLM` | Kimi-Linear-48B-A3B (KDA + NoPE-MLA + MoE) | **Paged-incremental decode (§19) MEASURED GB10: 18.9 tok/s (4.5× over recompute, 0.90× vLLM ~21)**; Gate A token-identical to recompute (122/128); not STRICT (p7 near-tie) | default off (opt-in `--incremental`); 5× decode gap (0.20×) closed to ~1.1× | +| `KimiLinearForCausalLM` | Kimi-Linear-48B-A3B (KDA + NoPE-MLA + MoE) | **Paged-incremental decode (§19) GB10: 18.9 tok/s (0.90× vLLM) @ 122/128 = coherent best**; STRICT unreachable (bf16 stream REFUTED §20/#118 122→4/128; p7 intrinsic near-tie) | CLI opt-in `--incremental`; SERVER fold scoped (ARCH-ONE-SURFACE req 4; runner aborts on Kimi KV) | | `KimiK3ForConditionalGeneration` | Kimi-K3 (2.8T MoE) | scaffold: registry+config+enumeration gated, forward refuses | HW-infeasible (~1.56 TB); no run | | `CohereForCausalLM` | Command-R / Cohere (and Cohere2) | scaffold: W0 tiny-random oracle run-verified; real-checkpoint gate blocked | no run | @@ -252,7 +252,7 @@ CPU elementwise GEMM (f32/f16/bf16) runs AVX2 and AVX-512 tiers on x86 where the | Gap | State | Detail | |---|---|---| -| Kimi-Linear-48B-A3B (KDA + NoPE-MLA + MoE hybrid) | **Paged-incremental decode (§19) MEASURED GB10: 18.9 tok/s = 4.5× over recompute 4.23, 0.90× vLLM ~21**; Gate A token-identical to recompute (122/128); NOT STRICT (p7 intrinsic near-tie), default OFF | speed lever LANDS (5× gap 0.20×→0.90×); STRICT owed (p7); residual = per-step GEMVs + host orchestration | +| Kimi-Linear-48B-A3B (KDA + NoPE-MLA + MoE hybrid) | **Paged-incremental (§19) GB10: 18.9 tok/s (0.90× vLLM) @ 122/128 = coherent best**; STRICT unreachable: bf16 stream REFUTED (§20/#118: 122→4/128), §14-§20 all levers closed, p7 near-tie | speed lever LANDS (0.90×); STRICT closed as near-tie; SERVER fold scoped (ARCH-ONE-SURFACE req 4) | | Multi-GPU execution | Hardware-blocked | TP proven equal to tp=1 on CPU; no 2-GPU box to run it | | LoRA end to end | CPU brick landed | Unwired standalone; not usable through the server | | Multimodal over HTTP | Architecturally blocked | Vision tower lives outside the registered engine forward | diff --git a/docs/STATUS.md b/docs/STATUS.md index d1256b40..38ef235c 100644 --- a/docs/STATUS.md +++ b/docs/STATUS.md @@ -75,7 +75,7 @@ token-for-token correctness against the pinned oracle. | DeepSeek-V2 MLA | Correctness-complete, speed-pending | Token-exact 8/8 (DeepSeek-V2-Lite); 0.86-0.95x output rate, TTFT faster at c4/c8. A2+A5 MLA norm-rope fold default-ON (`VT_MLA_FUSED_NORM_ROPE`, bit-exact rollback, SACRED 8/8 unchanged; forensics in benchmark-record) — kimi_k3/kimi-linear inherit it | | GLM-4 dense (sandwich norms, partial rope) | Correctness-complete, speed-pending | Token-exact 16/16 (GLM-4-9B-0414); first GLM-family model; partial interleaved RoPE + Gemma2 sandwich norms + biased qkv | | GLM-4.7-Flash (MLA + GLM MoE) | Correctness-complete, speed-pending | Token-exact 8/8 (GLM-4.7-Flash, 31.2B); reuses the DeepSeek-V2 MLA stack; first e2e coverage of the q_lora query branch + noaux_tc sigmoid router with routed-scaling | -| Kimi-Linear-48B-A3B (KDA + NoPE-MLA + MoE hybrid) | **Paged-incremental decode MEASURED GB10 (§19): 18.9 tok/s, 4.5× over recompute, 0.90× vLLM ~21** (gap 0.20×→0.90×); Gate A == recompute (122/128), NOT STRICT (p7 near-tie), OFF | `test_kimi_linear_forward` **15/15·875** (CPU byte-exact); GB10 incr-chunk 122/128 == recompute, 18.87/19.03 tok/s; STRICT owed (p7) | +| Kimi-Linear-48B-A3B (KDA + NoPE-MLA + MoE hybrid) | **Paged-incremental (§19) GB10: 18.9 tok/s (0.90× vLLM) @ 122/128 = coherent best.** STRICT unreachable — bf16 stream REFUTED (§20/#118: 122→4/128), §14-§20 levers closed, p7 near-tie | `test_kimi_linear_forward` **15/15·875** (knob OFF); GB10 control 122 (3×) vs +bf16 4. SERVER fold scoped (ARCH-ONE-SURFACE req 4; `runner.cpp:489`) | | Gemma-3 dense (GeGLU, dual rope, sandwich norms) | Correctness-complete, speed-pending | STRICT token-exact 48/48 greedy (gemma-3-1b-it); first Gemma-family model; GeGLU (gelu_pytorch_tanh) + dual per-layer RoPE theta + Gemma-RMSNorm sandwich norms + sqrt(hidden) embed-scale + query_pre_attn_scalar scaling | | Gemma-2 dense (attn + final logit soft-cap) | Correctness-complete, speed-pending | Near-tie-band 48/48 (gemma-2-2b-it): 44/48 strict on vLLM's greedy + 4/48 at 0.0-nat ties in vLLM's own logits; proves the attention + final logit soft-cap primitives (attn_logit_softcapping 50 + final 30); the inverse of Gemma-3 (both soft-caps, no QK-norm) | | Gemma-1 dense (the original Gemma) | Correctness-complete, speed-pending | STRICT token-exact 48/48 greedy (gemma-2b); two fused norms/layer, head_dim scale, GeGLU + sqrt(hidden) embed-scale, tied lm_head; no soft-cap/QK-norm/sliding. **D1 (2026-07-31): the whole Gemma family (1/2/3/4) folded to the default-ON bf16 merged-QKV descriptor (`MergedQkvEnabled`); re-gated Gemma-2 SACRED 48/48 (global+sliding) + Gemma-4 STRICT 32/32 — its existing gate held** | diff --git a/src/vllm/model_executor/models/kimi_linear_device.cpp b/src/vllm/model_executor/models/kimi_linear_device.cpp index cd10211d..ede348fc 100644 --- a/src/vllm/model_executor/models/kimi_linear_device.cpp +++ b/src/vllm/model_executor/models/kimi_linear_device.cpp @@ -244,6 +244,41 @@ void RoundDevBf16(const Dev& d, DBuf& x) { vt::CastF32(d.q, out, b.t()); } +// (6) VT_KIMI_BF16_STREAM — carry the inter-layer residual stream in bf16 END-TO-END, +// exactly as vLLM (DeepseekV2Model::ForwardBody / deepseek_v2.cpp:479-615: hidden / +// residual / normed-hidden / block-outputs all bf16; fused_add_rms_norm rounds the +// residual store to bf16). This SUPERSEDES the partial VT_KIMI_BF16_RESIDUAL knob (spec +// §14, which rounded f32 STORAGE in place): the STRUCTURAL bf16 storage IS vLLM's +// rounding AND removes the per-GEMM CastBf16 on every residual-fed projection (the +// decode-decomposition's 3% CastBf16, spec §19) plus the f32<->bf16 round-trips. The +// projection OUTPUTS stay f32 (the KDA/MLA islands read f32 unchanged), so the ONLY +// numeric change vs the f32 stream is the residual add/norm rounding to bf16 at each +// layer boundary — the isolated p7-STRICT lever (spec §13/§14 root cause). Default OFF +// (parity-enabler: flip only with the token gate green at the flipped default); when ON, +// VT_KIMI_BF16_RESIDUAL's manual RoundDevBf16 is redundant and skipped. +bool Bf16Stream() { + static const bool on = [] { + const char* e = std::getenv("VT_KIMI_BF16_STREAM"); + return e != nullptr && e[0] == '1'; + }(); + return on; +} +inline DType StreamDType() { return Bf16Stream() ? DType::kBF16 : DType::kF32; } + +// Cast a block output DBuf to the residual-stream dtype (identity when already `sdt`; +// f32 -> bf16 round for the bf16 stream — the o_proj/down/combine GEMM's bf16 store, +// done as a separate cast so the block internals stay dtype-agnostic). RNE-equal to a +// (bf16,bf16)->bf16 MatmulBT store, so numerically vLLM-faithful. +DBuf ToStream(const Dev& d, DBuf&& out, DType sdt) { + if (out.t().dtype == sdt) return std::move(out); + Tensor ot = out.t(); + std::vector shape(ot.shape, ot.shape + ot.rank); + DBuf s(d, sdt, shape); + Tensor st = s.t(); + vt::CastBf16(d.q, st, ot); // f32 -> bf16 (sdt is bf16 in the non-identity case) + return s; +} + // Device-resident weight view. On CPU this ALIASES the host f32 bytes exactly as // dense_attn::ResidentWeight does for a CPU device (host-pointer aliasing is a CPU // property); the CUDA staging over materialized OwnedTensors is the born-on-runner @@ -281,10 +316,17 @@ inline Tensor ResidentBf16W(const Dev& d, const OwnedTensor& w, // projection numerics (cuda_matmul.cu:3); the residual stream stays f32. void GemmBf16(const Dev& d, Tensor& out, const Tensor& act, const OwnedTensor& w, int64_t N, int64_t K) { + Tensor wt = ResidentBf16W(d, w, {N, K}); + // bf16 residual stream (VT_KIMI_BF16_STREAM): the activation is ALREADY bf16, so the + // per-GEMM CastBf16 (the decode-decomposition's 3%, spec §19) is elided — feed the + // bf16 activation straight into the (bf16,bf16)->out MatmulBT. + if (act.dtype == DType::kBF16) { + vt::MatmulBT(d.q, out, act, wt); + return; + } const int64_t T = act.shape[0]; DBuf ab(d, DType::kBF16, {T, K}); vt::CastBf16(d.q, ab.t(), act); - Tensor wt = ResidentBf16W(d, w, {N, K}); vt::MatmulBT(d.q, out, ab.t(), wt); } @@ -300,6 +342,25 @@ void AddRmsNorm(const Dev& d, DBuf& out, const Tensor& x, const Tensor& w, DBuf& } } +// Stream-dtype-aware residual add+RMSNorm for the bf16 device forward. The residual- +// stream norm weights are kept host f32 (ReadF32 from the bf16 checkpoint — the value +// IS already a bf16 value); the CUDA RmsNorm/FusedChain kernels REQUIRE +// weight.dtype == x.dtype (cuda_ops.cu:452,3480), so for the bf16 stream the f32-stored +// (= bf16-valued) weight is cast to bf16 (LOSSLESS) before the norm — matching vLLM's +// bf16 norm weight exactly. For the f32 stream this aliases the host f32 bytes (WF32), +// byte-identical to the plain AddRmsNorm above. +void AddRmsNormS(const Dev& d, DBuf& out, const Tensor& x, const std::vector& wv, + int64_t H, DBuf& res, float eps, DType sdt) { + if (sdt == DType::kF32) { + AddRmsNorm(d, out, x, WF32(d, wv, {H}), res, eps); + return; + } + DBuf wf(d, DType::kF32, {H}, wv.data()); + DBuf wb(d, DType::kBF16, {H}); + vt::CastBf16(d.q, wb.t(), wf.t()); + AddRmsNorm(d, out, x, wb.t(), res, eps); +} + // silu(gate@x) * (up@x) -> down@(...) — a gated SwiGLU MLP via the shared vt:: ops // on separate gate/up/down host weights (torch [out,in]). MoeSiluMul (not // SiluAndMul) is used so the merged-GEMM checker is not tripped: the fused @@ -1055,12 +1116,15 @@ DBuf MoeBlockDeviceBf16(const Dev& d, const MoeResidentWeights& w, const Tensor& for (int64_t t = 0; t < T; ++t) for (int64_t j = 0; j < k; ++j) lists[static_cast(ids[static_cast(t * k + j)])].push_back({t, j}); - const size_t row_bytes = static_cast(H) * sizeof(float); + // Gather per-expert rows in the STREAM dtype (bf16 stream: dh is bf16, so the row + // stride is the bf16 width and xg is bf16 — SwiGluDeviceBf16's GemmBf16 reads it + // straight; f32 stream: unchanged f32 gather). Grounding deepseek_v2.cpp MoeBlock. + const size_t row_bytes = static_cast(H) * vt::SizeOf(dh.dtype); for (int64_t e = 0; e < E; ++e) { const auto& list = lists[static_cast(e)]; if (list.empty()) continue; const int64_t n = static_cast(list.size()); - DBuf xg(d, DType::kF32, {n, H}); + DBuf xg(d, dh.dtype, {n, H}); for (int64_t r = 0; r < n; ++r) d.b.Copy(d.q, static_cast(xg.ptr()) + static_cast(r) * row_bytes, static_cast(dh.data) + @@ -1111,51 +1175,62 @@ DBuf DeviceForwardBodyBf16(const Dev& d, const KimiLinearWeights& weights, VT_CHECK(static_cast(rw.layers.size()) == L, "KimiLinear bf16 device compute: resident layer count != num_hidden_layers"); - // embed (bf16 table -> f32 out) -> residual stream. - DBuf hidden(d, DType::kF32, {T, H}); + // The residual-stream dtype: bf16 END-TO-END (VT_KIMI_BF16_STREAM, vLLM's regime, + // mirroring deepseek_v2.cpp:479-615) or f32 (default). The bf16 stream removes the + // per-GEMM CastBf16 (projections read the bf16 stream directly) AND rounds the + // residual add/norm to bf16 at every layer boundary (the p7-STRICT lever); the + // partial VT_KIMI_BF16_RESIDUAL RoundDevBf16 is then redundant and skipped. + const DType sdt = StreamDType(); + const bool round_res = Bf16Residual() && sdt == DType::kF32; + + // embed (bf16 table -> stream out) -> residual stream. + DBuf hidden(d, sdt, {T, H}); { DBuf dids(d, DType::kI32, {T}, token_ids.data()); Tensor htab = ResidentBf16W(d, rw.embed_tokens, {V, H}); Tensor hh = hidden.t(); vt::Embedding(d.q, hh, htab, dids.t()); } - const bool bf16_res = Bf16Residual(); - if (bf16_res) RoundDevBf16(d, hidden); // vLLM embed output is bf16 - DBuf res(d, DType::kF32, {T, H}); + if (round_res) RoundDevBf16(d, hidden); // vLLM embed output is bf16 + DBuf res(d, sdt, {T, H}); res.Zero(d); Tensor hcur = hidden.t(); std::shared_ptr hold; for (int64_t l = 0; l < L; ++l) { const KimiLinearLayerResidentWeights& lw = rw.layers[static_cast(l)]; - DBuf dhn(d, DType::kF32, {T, H}); - AddRmsNorm(d, dhn, hcur, WF32(d, lw.input_layernorm, {H}), res, eps); - if (bf16_res) RoundDevBf16(d, res); // vLLM stores residual bf16 (variance saw f32 sum) - DBuf attn = lw.is_kda ? KdaLayerDeviceBf16(d, lw.kda, dhn.t(), p, T) - : MlaLayerDeviceBf16(d, lw.mla, dhn.t(), p, T); - if (bf16_res) RoundDevBf16(d, attn); // vLLM attn_output is bf16 - DBuf dh2(d, DType::kF32, {T, H}); - AddRmsNorm(d, dh2, attn.t(), WF32(d, lw.post_attention_layernorm, {H}), res, eps); - if (bf16_res) RoundDevBf16(d, res); - DBuf mlp = lw.is_moe ? MoeBlockDeviceBf16(d, lw.moe, dh2.t(), p, T) - : DenseMlpDeviceBf16(d, lw.dense, dh2.t(), p, T); - if (bf16_res) RoundDevBf16(d, mlp); // vLLM mlp output is bf16 + DBuf dhn(d, sdt, {T, H}); + AddRmsNormS(d, dhn, hcur, lw.input_layernorm, H, res, eps, sdt); + if (round_res) RoundDevBf16(d, res); + DBuf attn = ToStream(d, + lw.is_kda ? KdaLayerDeviceBf16(d, lw.kda, dhn.t(), p, T) + : MlaLayerDeviceBf16(d, lw.mla, dhn.t(), p, T), + sdt); + if (round_res) RoundDevBf16(d, attn); + DBuf dh2(d, sdt, {T, H}); + AddRmsNormS(d, dh2, attn.t(), lw.post_attention_layernorm, H, res, eps, sdt); + if (round_res) RoundDevBf16(d, res); + DBuf mlp = ToStream(d, + lw.is_moe ? MoeBlockDeviceBf16(d, lw.moe, dh2.t(), p, T) + : DenseMlpDeviceBf16(d, lw.dense, dh2.t(), p, T), + sdt); + if (round_res) RoundDevBf16(d, mlp); auto* held = new DBuf(std::move(mlp)); hcur = held->t(); hold = std::shared_ptr(held, [](void* q) { delete static_cast(q); }); } - DBuf dnorm(d, DType::kF32, {T, H}); - AddRmsNorm(d, dnorm, hcur, WF32(d, rw.final_norm, {H}), res, eps); + DBuf dnorm(d, sdt, {T, H}); + AddRmsNormS(d, dnorm, hcur, rw.final_norm, H, res, eps, sdt); // logits_indices gather-before-lm_head, in REQUEST order. Tensor src = dnorm.t(); - DBuf dgather(d, DType::kF32, + DBuf dgather(d, sdt, logits_indices.empty() ? std::vector{1, 1} : std::vector{static_cast(logits_indices.size()), H}); if (!logits_indices.empty()) { - const size_t rb = static_cast(H) * sizeof(float); + const size_t rb = static_cast(H) * vt::SizeOf(sdt); char* dp = static_cast(dgather.ptr()); const char* sp = static_cast(dnorm.ptr()); for (size_t i = 0; i < logits_indices.size(); ++i) { @@ -1479,17 +1554,21 @@ DBuf DeviceForwardBodyBf16Incremental(const Dev& d, const KimiLinearWeights& wei VT_CHECK(static_cast(rw.layers.size()) == L, "KimiLinear incremental: resident layer count != num_hidden_layers"); const bool use_chunk = DeviceKdaChunk(); + // Residual-stream dtype (VT_KIMI_BF16_STREAM), byte-for-byte the same treatment as + // DeviceForwardBodyBf16 (recompute) so the paged-incremental path stays token- + // identical to recompute (the Gate A / case-(l) byte-exact state-carry proof). + const DType sdt = StreamDType(); + const bool round_res = Bf16Residual() && sdt == DType::kF32; - DBuf hidden(d, DType::kF32, {T, H}); + DBuf hidden(d, sdt, {T, H}); { DBuf dids(d, DType::kI32, {T}, token_ids.data()); Tensor htab = ResidentBf16W(d, rw.embed_tokens, {V, H}); Tensor hh = hidden.t(); vt::Embedding(d.q, hh, htab, dids.t()); } - const bool bf16_res = Bf16Residual(); - if (bf16_res) RoundDevBf16(d, hidden); - DBuf res(d, DType::kF32, {T, H}); + if (round_res) RoundDevBf16(d, hidden); + DBuf res(d, sdt, {T, H}); res.Zero(d); Tensor hcur = hidden.t(); std::shared_ptr hold; @@ -1497,37 +1576,41 @@ DBuf DeviceForwardBodyBf16Incremental(const Dev& d, const KimiLinearWeights& wei int64_t kda_idx = 0, mla_idx = 0; for (int64_t l = 0; l < L; ++l) { const KimiLinearLayerResidentWeights& lw = rw.layers[static_cast(l)]; - DBuf dhn(d, DType::kF32, {T, H}); - AddRmsNorm(d, dhn, hcur, WF32(d, lw.input_layernorm, {H}), res, eps); - if (bf16_res) RoundDevBf16(d, res); - DBuf attn = lw.is_kda - ? KdaLayerDeviceBf16Inc(d, lw.kda, dhn.t(), p, T, - cache.kda[static_cast(kda_idx++)], - is_prefill, use_chunk) - : MlaLayerDeviceBf16Inc(d, lw.mla, dhn.t(), p, T, base_pos, - cache.mla[static_cast(mla_idx++)]); - if (bf16_res) RoundDevBf16(d, attn); - DBuf dh2(d, DType::kF32, {T, H}); - AddRmsNorm(d, dh2, attn.t(), WF32(d, lw.post_attention_layernorm, {H}), res, eps); - if (bf16_res) RoundDevBf16(d, res); - DBuf mlp = lw.is_moe ? MoeBlockDeviceBf16(d, lw.moe, dh2.t(), p, T) - : DenseMlpDeviceBf16(d, lw.dense, dh2.t(), p, T); - if (bf16_res) RoundDevBf16(d, mlp); + DBuf dhn(d, sdt, {T, H}); + AddRmsNormS(d, dhn, hcur, lw.input_layernorm, H, res, eps, sdt); + if (round_res) RoundDevBf16(d, res); + DBuf attn = ToStream( + d, + lw.is_kda ? KdaLayerDeviceBf16Inc(d, lw.kda, dhn.t(), p, T, + cache.kda[static_cast(kda_idx++)], + is_prefill, use_chunk) + : MlaLayerDeviceBf16Inc(d, lw.mla, dhn.t(), p, T, base_pos, + cache.mla[static_cast(mla_idx++)]), + sdt); + if (round_res) RoundDevBf16(d, attn); + DBuf dh2(d, sdt, {T, H}); + AddRmsNormS(d, dh2, attn.t(), lw.post_attention_layernorm, H, res, eps, sdt); + if (round_res) RoundDevBf16(d, res); + DBuf mlp = ToStream(d, + lw.is_moe ? MoeBlockDeviceBf16(d, lw.moe, dh2.t(), p, T) + : DenseMlpDeviceBf16(d, lw.dense, dh2.t(), p, T), + sdt); + if (round_res) RoundDevBf16(d, mlp); auto* held = new DBuf(std::move(mlp)); hcur = held->t(); hold = std::shared_ptr(held, [](void* q) { delete static_cast(q); }); } - DBuf dnorm(d, DType::kF32, {T, H}); - AddRmsNorm(d, dnorm, hcur, WF32(d, rw.final_norm, {H}), res, eps); + DBuf dnorm(d, sdt, {T, H}); + AddRmsNormS(d, dnorm, hcur, rw.final_norm, H, res, eps, sdt); Tensor src = dnorm.t(); - DBuf dgather(d, DType::kF32, + DBuf dgather(d, sdt, logits_indices.empty() ? std::vector{1, 1} : std::vector{static_cast(logits_indices.size()), H}); if (!logits_indices.empty()) { - const size_t rb = static_cast(H) * sizeof(float); + const size_t rb = static_cast(H) * vt::SizeOf(sdt); char* dp = static_cast(dgather.ptr()); const char* sp = static_cast(dnorm.ptr()); for (size_t i = 0; i < logits_indices.size(); ++i) {