Kimi-Linear: paged-incremental decode (prefill-once + carried-state decode) — close the 5x vLLM gap + STRICT - #113
Closed
localai-bot wants to merge 2 commits into
Closed
Kimi-Linear: paged-incremental decode (prefill-once + carried-state decode) — close the 5x vLLM gap + STRICT#113localai-bot wants to merge 2 commits into
localai-bot wants to merge 2 commits into
Conversation
…state decode; CPU byte-exact gated (§19) (#113) Implements the §18 named lever (e): replace the O(n²) full-recompute vehicle (ForwardDeviceCompute re-runs [0..prompt+t] every step, the 4.24 tok/s rate) with vLLM's decode regime — PREFILL the prompt ONCE (capturing the KDA recurrent+conv state per KDA layer and the NoPE-MLA latent-KV per MLA layer into a persistent KimiDecodeCache) then advance ONE token per step from the CARRIED state (mirrors kimi_gdn_linear_attn.py prefill=chunk / decode=recurrent). - KimiDecodeCache + KimiLinearModel::ForwardPrefillIncremental / ForwardDecodeStepIncremental (kimi_linear.h, kimi_linear_device.cpp). State carry: vt::KdaGatedDeltaRule state [1,nh,hd,hd] in/out; vt::CausalConv1dFwd conv-tap carry (mamba decode); growing host NoPE-MLA latent-KV (kv|kpe). - kimi-linear-gen --incremental A/B vehicle (recompute vs incremental). - CPU byte-exact state-carry gate: test_kimi_linear_forward 15/15·875 (was 14/14·825) — NEW case (l): carried decode-step logits are byte-identical (1e-5) to a fresh full-sequence recompute at each step, greedy-identical. A pure wiring proof (Laguna W6 pattern); no regressions. - Record repairs (pre-existing reds at HEAD): document VT_KIMI_DEVICE_KDA_CHUNK (docs/ENVIRONMENT.md) + VT_KDA_CHUNK_TRITON (env-doc allowlist); fix the §17 dangling kda-chunk-aot/ link (moved to triton_kernels/ in §18/#111). GB10 Gate A (incremental==recompute) / Gate B (STRICT vs the §12 golden) / memory audit / speed re-measure vs vLLM ~21 tok/s are OWED next. Defaults unchanged (opt-in harness flag; VT_KIMI_DEVICE_KDA/_CHUNK stay OFF). Row: row/KIMI-PAGED-INCREMENTAL (#113) FOLLOWING_AGENTS_PROTOCOL Assisted-by: Claude Code:claude-opus-4-8 [ClaudeCode]
mudler
added a commit
that referenced
this pull request
Aug 7, 2026
…, 0.90× vLLM), Gate A token-identical, STRICT not reached; decode 90% GEMV-parity (#113) GB10 measured results for the paged-incremental decode (row/KIMI-PAGED-INCREMENTAL, PR #113), single-load per config, flock, drop_caches, min-avail 18-21 GiB, no reboot, §12 golden md5 bfa5bdbf: - recompute (O(n²)) 122/128 @ 4.23 tok/s (reproduces #111). - incremental+recurrence 120/128 @ 16.63 (p7 near-tie flips). - incremental+chunk-prefill 122/128 @ 18.87/19.03 tok/s (2 runs): token-IDENTICAL to recompute (Gate A PASS) and the fastest = 4.5× over recompute, 0.90× of vLLM ~21 (the #111 aggregate floor) — the measured 5× decode gap (0.20×) closed to ~1.1×. - Gate B STRICT NOT reached (122/128): chunk-prefill in the right vehicle does NOT close p7 — refutes the #111 hypothesis; p7 is an intrinsic near-tie (§13/§14). - Decode decomposition (nsys, ours): ~90% is the identical cuBLAS gemvx symbol vLLM uses (batch-1 GEMV-parity); KDA 2.3%, MoE glue 2.3%, CastBf16 3%. Killing O(n²) alone reaches parity-class; no lever load-bearing beyond it (residual = host idle + bf16-cast, the latter also the p7-STRICT lever). vLLM-live-nsys@0.82 NOT run: a box-safety violation (reservation + nsys below the 15 GiB LIFE-CRITICAL floor). - Design mirrors vLLM kimi_gdn_linear_attn._forward (prefill=chunk_kda_with_fused_gate / decode=fused_recurrent_kda) @ vllm-src a4e3cb4; divergences noted (§19). Spec §19, STATUS/BENCHMARKS/FEATURES rows, benchmark-record, state.md, NOW refreshed. Default unchanged (opt-in; VT_KIMI_DEVICE_KDA/_CHUNK stay OFF; 122/128 ≠ STRICT). Row: row/KIMI-PAGED-INCREMENTAL (#113) FOLLOWING_AGENTS_PROTOCOL Assisted-by: Claude Code:claude-opus-4-8 [ClaudeCode]
mudler
force-pushed
the
row/KIMI-PAGED-INCREMENTAL
branch
from
August 7, 2026 14:40
de5f00c to
631ad5f
Compare
…, 0.90× vLLM), Gate A token-identical, STRICT not reached; decode 90% GEMV-parity (#113) GB10 measured results for the paged-incremental decode (row/KIMI-PAGED-INCREMENTAL, PR #113), single-load per config, flock, drop_caches, min-avail 18-21 GiB, no reboot, §12 golden md5 bfa5bdbf: - recompute (O(n²)) 122/128 @ 4.23 tok/s (reproduces #111). - incremental+recurrence 120/128 @ 16.63 (p7 near-tie flips). - incremental+chunk-prefill 122/128 @ 18.87/19.03 tok/s (2 runs): token-IDENTICAL to recompute (Gate A PASS) and the fastest = 4.5× over recompute, 0.90× of vLLM ~21 (the #111 aggregate floor) — the measured 5× decode gap (0.20×) closed to ~1.1×. - Gate B STRICT NOT reached (122/128): chunk-prefill in the right vehicle does NOT close p7 — refutes the #111 hypothesis; p7 is an intrinsic near-tie (§13/§14). - Decode decomposition (nsys, ours): ~90% is the identical cuBLAS gemvx symbol vLLM uses (batch-1 GEMV-parity); KDA 2.3%, MoE glue 2.3%, CastBf16 3%. Killing O(n²) alone reaches parity-class; no lever load-bearing beyond it (residual = host idle + bf16-cast, the latter also the p7-STRICT lever). vLLM-live-nsys@0.82 NOT run: a box-safety violation (reservation + nsys below the 15 GiB LIFE-CRITICAL floor). - Design mirrors vLLM kimi_gdn_linear_attn._forward (prefill=chunk_kda_with_fused_gate / decode=fused_recurrent_kda) @ vllm-src a4e3cb4; divergences noted (§19). Spec §19, STATUS/BENCHMARKS/FEATURES rows, benchmark-record, state.md, NOW refreshed. Default unchanged (opt-in; VT_KIMI_DEVICE_KDA/_CHUNK stay OFF; 122/128 ≠ STRICT). Row: row/KIMI-PAGED-INCREMENTAL (#113) FOLLOWING_AGENTS_PROTOCOL Assisted-by: Claude Code:claude-opus-4-8 [ClaudeCode]
mudler
force-pushed
the
row/KIMI-PAGED-INCREMENTAL
branch
from
August 7, 2026 14:43
631ad5f to
61e6328
Compare
mudler
added a commit
that referenced
this pull request
Aug 7, 2026
…x), 0.90x of vLLM (#113) row/KIMI-PAGED-INCREMENTAL squash. The measured 5x decode gap CLOSES to ~1.1x: chunk-prefill ONCE (vt::KdaChunkPrefill) carrying the final recurrent state per KDA layer (KdaScanKernel verified to READ the passed state — byte-exact carry), causal-conv taps carried, NoPE-MLA KV appended per token; decode advances via the #104 recurrent kernel T=1. TOKEN-IDENTITY PASS: byte-identical to the recompute path across all 128 tokens. Steady 18.87/19.03 tok/s (2 runs) vs vLLM ~21 matched config. Mirrors vLLM's ACTUAL mechanism (kimi_gdn_linear_attn state carry, MambaSpec KDA state + MLA latent pages — cited; divergences deliberate + named). STRICT honestly refuted as a prefill-order artifact: chunk-prefill in the RIGHT vehicle reproduces recompute EXACTLY at 122/128 — p7 is an intrinsic near-tie. Decomposition: our decode is ~90% the IDENTICAL cuBLAS gemvx symbol vLLM calls at batch-1 — parity-class by construction. vLLM-live nsys at util 0.82 correctly NOT run (would breach the 15 GiB floor). Defaults stay OFF pending STRICT (--incremental opt-in). Residuals ranked: bf16 residual stream e2e (targets BOTH p7/STRICT and the CastBf16+island tail), paged-FA2 MLA, ~15% host idle. Also documents two env vars + fixes a dangling spec link; the state entry's heading/anchor adjacency fixed. FOLLOWING_AGENTS_PROTOCOL Assisted-by: Claude Code:claude-opus-4-8 [ClaudeCode]
Collaborator
Author
|
Landed on main as the squash (mudler authorship). 0.90x of vLLM measured; the bf16 residual-stream campaign (the coupled STRICT + last-10% lever) is next. 🤖 Generated with Claude Code |
mudler
added a commit
that referenced
this pull request
Aug 7, 2026
…losed as intrinsic near-tie; SERVER fold scoped (#118) The #113 follow-on campaign, an honest measured negative. Structural bf16 residual stream (VT_KIMI_BF16_STREAM, mirrors deepseek_v2.cpp FusedChain order) DESTABILIZES the KDA recurrence into degenerate repeat loops: GB10 full-48.9B 128-token gate scores 4/128 (diagnostic variant 5/128) vs the f32 control's 122/128 @ 18.9 tok/s (reproduced 3x, golden bfa5bdbf). No speed win (19.8 within noise). With Sec.14 (host plateau 120), Sec.15 (device-KDA 122), Sec.16 (device-MLA 109), Sec.18 (chunk-every-step 102) and now Sec.20: STRICT 128/128 is NOT reachable by residual-precision or device-island levers; p7 is an intrinsic near-tie and 122/128 @ 0.90x vLLM is Kimi's coherent best. Knob lands default-OFF as a documented-measured-negative A/B (the VT_KIMI_DEVICE_MLA precedent); CPU gate knob-OFF byte-exact 15/15/875 (re-run by the operator on this merged tree). SERVER runner fold (ARCH-ONE-SURFACE req 4) scoped, enabling-blocked: the runner aborts on Kimi KV at runner.cpp:489-493 (needs layer_types + GDN geometry synthesized from linear_attn_config, a KDA-paged block, a NoPE-MLA paged block, ForwardDevice binding) - spec Sec.20.3, tracked as the single remaining STRICT/speed/server lever. FOLLOWING_AGENTS_PROTOCOL Assisted-by: Claude Code:claude-fable-5 [ClaudeCode]
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Mission (row/KIMI-PAGED-INCREMENTAL)
Close the MEASURED 5× decode gap on Kimi-Linear-48B-A3B (#111: vLLM ~21 tok/s vs ours 4.24, 0.20×) AND reach STRICT, via the §18 named lever (e) paged-incremental decode: PREFILL the prompt ONCE (carry KDA recurrent+conv state + NoPE-MLA latent-KV) then RECURRENT-decode each token from the persistent state — killing the O(n²) full-recompute.
Landed here (CPU-gated)
KimiDecodeCache+ForwardPrefillIncremental/ForwardDecodeStepIncremental(kimi_linear.h,kimi_linear_device.cpp). State carry:vt::KdaGatedDeltaRulestate in/out,vt::CausalConv1dFwdconv-tap carry, growing host NoPE-MLA latent-KV.kimi-linear-gen --incrementalA/B vehicle.test_kimi_linear_forward15/15·875 — the carried decode-step is byte-identical (1e-5) to a fresh full-recompute at each step, greedy-identical (Laguna W6 wiring proof).VT_KIMI_DEVICE_KDA_CHUNK+VT_KDA_CHUNK_TRITON; fixed the §17 dangling link.OWED (this session, GB10)
--incrementalvsForwardDeviceComputeatVT_KIMI_DEVICE_KDA=1— token identity.VT_KIMI_DEVICE_KDA_CHUNK=1) — the p7 suspect in the right vehicle.Defaults unchanged (opt-in). Spec §19. Draft until GB10 Gate B (STRICT) + the speed win land.
🤖 Generated with Claude Code
https://claude.ai/code/session_014fZAcg1WcU8V629k6HWKys