Kimi-Linear STRICT close: device NoPE-MLA + chunk_kda/incremental (row/KIMI-STRICT-CLOSE) - #107
Closed
localai-bot wants to merge 1 commit into
Closed
Kimi-Linear STRICT close: device NoPE-MLA + chunk_kda/incremental (row/KIMI-STRICT-CLOSE)#107localai-bot wants to merge 1 commit into
localai-bot wants to merge 1 commit into
Conversation
…A) — MEASURED-NEGATIVE on GB10, STRICT owes the actual FA2/chunk_kda kernels (#107) The spec §15 residual (d) — the 7 NoPE-MLA layers still run a host f64 softmax island that is MORE precise than vLLM's FA2 and coin-flips near-ties — was attempted in its device-COMPUTE form (the §15 device-KDA pattern applied to the MLA half) and MEASURED NEGATIVE on the full 48.9B GB10 gate. Implementation (additive, default OFF): `VT_KIMI_DEVICE_MLA` + `MlaAttnCoreDevice` (kimi_linear_device.cpp) route the NoPE causal softmax over per-head [k_nope|k_pe(shared)]/v through the shared device op vt::Attention (f32 online max-subtracted softmax = vLLM's FA2 regime) instead of the f64 host MlaSoftmaxIsland. vt::Attention carries one head-dim; MLA is asymmetric (qk=qk_nope+qk_rope=192, v=128), so value is zero-PADDED to qk — the weighted sum over the 0 tail is 0, so out[:,:,:v] is byte-exact. Wired into the f32 and bf16 paths. MLA dims VERIFIED from the real 48.9B config.json (nah=32, qk_nope=128, qk_rope=64, v=128, kv_lora=512, q_lora=None; 7 full-attn/20 KDA) — not the K3 nums. Unit gate (RED-first, CPU) GREEN: test_kimi_linear_forward 14/14·825 (was 13/13·656) — NEW case (g2) KimiMlaAttnCoreDevice (pad-V + vt::Attention) == a from-first-principles f64 causal-softmax reference (rtol 3e-3); a perturbed scale fails 108 assertions. Env-gated whole-forward green (VT_KIMI_DEVICE_MLA=1 alone and with VT_KIMI_DEVICE_KDA=1). Same on the GB10 CUDA binary (210 GDN + 23 KDA syms linked, CUTLASS-NVFP4 + FA2 + Triton-AOT). Full 48.9B GB10 gate vs the §12 STRICT golden (single-load per config, flock gpu.lock, min-avail 21 GiB, no reboot): control device-KDA reproduces 122/128 + 4.24 tok/s EXACTLY; +device-MLA REGRESSES to 109/128 AND 3.89 tok/s. Why (the §14 razor, re-proven): device-KDA works because its recurrence is the SAME algorithm as vLLM's decode kernel (f32-on-bf16); but vLLM's MLA prefill uses FA2 (a specific reduction ORDER) and vt::Attention's plain online-softmax is the right MATH in a DIFFERENT order, so it coin-flips near-ties (breaks p3 16→3 into the §14 163586x repeat) while p7 stays diverged, and the per-(t,h) build + pad-V waste slow the O(n^2) recompute. An approximation of vLLM's kernel is not enough. VERDICT: VT_KIMI_DEVICE_MLA STAYS OFF, kept as a documented-MEASURED-NEGATIVE A/B knob (parity-lever precedent: §14 ISLAND_F32ACC). device-KDA (122/128, 4.24) remains the best config, itself default OFF. The one-brick STRICT-close did NOT land. STRICT residual, sharpened: needs vLLM's ACTUAL kernels — (c) chunk_kda prefill family (Triton-AOT regen for sm_121a) + (d) paged FA2 mla::ForwardMlaAttentionBlock (NOT this vt::Attention approximation) + (e) paged-incremental decode (needs a decode/paged-attn op, query_len != key_len). Records: spec §16, benchmark-record entry, state.md anchored entry, NOW.md + model-matrix row + docs/STATUS/BENCHMARKS/FEATURES one-liners. All doc/record gates green. Box left clean (build tree/markers removed, worker restored, golden preserved, no reboot). FOLLOWING_AGENTS_PROTOCOL Assisted-by: Claude Code:claude-opus-4-8 [ClaudeCode]
mudler
force-pushed
the
row/KIMI-STRICT-CLOSE
branch
from
August 7, 2026 10:40
8056c9b to
6a368df
Compare
mudler
added a commit
that referenced
this pull request
Aug 7, 2026
…ICT needs vLLM's actual kernels (#107) row/KIMI-STRICT-CLOSE squash. The tractable §15 brick — routing the 7 NoPE-MLA layers through shared vt::Attention (f32 online softmax, pad-V for the asymmetric qk=192/v=128 dims, MLA geometry VERIFIED from the real 48.9B config) — implemented, RED-first unit-gated (14/14-825), and MEASURED NEGATIVE on the full GB10 gate: 109/128 + 3.89 tok/s vs device-KDA's 122 + 4.24 — the right math in a DIFFERENT reduction order than vLLM's FA2 coin-flips near-ties. VT_KIMI_DEVICE_MLA stays OFF as a documented measured-negative A/B knob; no flips (122 != STRICT). Control reproduced #104 EXACTLY. Re-proves the razor: only vLLM's ACTUAL kernels close the last near-tie. The real STRICT path, scoped: (c) the FLA chunk_kda prefill family (4 Triton kernels + sm_121a AOT regen — prime suspect); (d) paged FA2 mla::ForwardMlaAttentionBlock; (e) paged-incremental decode (query_len != key_len paged attention; kills the O(n^2)). FOLLOWING_AGENTS_PROTOCOL Assisted-by: Claude Code:claude-opus-4-8 [ClaudeCode]
Collaborator
Author
|
Landed on main as the squash (mudler authorship). The three real STRICT bricks are the scoped next campaign; its GPU leg must RE-PARK local-ai-worker first (restored, 35GB resident — cannot co-tenant the 92GB Kimi load). 🤖 Generated with Claude Code |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Row
row/KIMI-STRICT-CLOSE— closing Kimi-Linear-48B-A3B toward the STRICT 128-token gate (spec §15 residuals c/d/e).Baseline (#104, spec §15): device-KDA (
VT_KIMI_DEVICE_KDA) = 122/128 + 4.24 tok/s, default OFF; sole divergence = p7 pos-6 comma near-tie.Result: device NoPE-MLA lever MEASURED-NEGATIVE — STRICT owes vLLM's ACTUAL kernels
The §15 residual (d) was attempted in its device-COMPUTE form and measured on the full 48.9B GB10 gate:
VT_KIMI_DEVICE_COMPUTE=1)DEVICE_KDA=1DEVICE_MLA=1VT_KIMI_DEVICE_MLAroutes the 7 NoPE-MLA layers' softmax core throughvt::Attention(f32 online softmax; value zero-padded qk=192 vs v=128,out[:,:,:v]byte-exact). It is the right MATH but a DIFFERENT reduction order than vLLM's FA2, so it coin-flips near-ties (breaks p3 16→3 into the §14163586×repeat) and the per-(t,h) build slows the O(n²) recompute — the §14 razor re-proven: an approximation of vLLM's kernel is not enough.Verdict:
VT_KIMI_DEVICE_MLASTAYS OFF, kept as a documented-MEASURED-NEGATIVE A/B knob (§14ISLAND_F32ACCprecedent). No default flip. Row STAYSACTIVE.Bricks
VT_KIMI_DEVICE_MLA) — implemented, CPU RED-first gate 14/14·825, GB10-measured NEGATIVE. Kept as A/B knob.mla::ForwardMlaAttentionBlock(residual d, the ACTUAL kernel) + paged-incremental decode (residual e, needs a decode/paged-attn op —query_len≠key_len, whichvt::Attentioncannot express). Scoped.Gates: CPU
test_kimi_linear_forward14/14·825; GB10 CUDA build clean (210 GDN + 23 KDA syms); all doc/record gates green. Box left clean.🤖 Generated with Claude Code
https://claude.ai/code/session_014fZAcg1WcU8V629k6HWKys