Kimi-Linear: bf16 residual stream REFUTED (measured-negative) + SERVER runner-fold scope (§20) - #118
Closed
localai-bot wants to merge 1 commit into
Closed
Kimi-Linear: bf16 residual stream REFUTED (measured-negative) + SERVER runner-fold scope (§20)#118localai-bot wants to merge 1 commit into
localai-bot wants to merge 1 commit into
Conversation
…UTED (§20, #118) Tested the §19-named residual #1 — a bf16 residual stream END-TO-END, framed as "the ONE lever both verdicts point at" (STRICT via matching vLLM's bf16 rounding on the p7 near-tie + speed via killing the 3% per-GEMM CastBf16). Implemented STRUCTURALLY (bf16 DBufs for hidden/residual/normed-hidden/block-outputs via vt::FusedChain(kFusedAddRmsNormStd), mirroring deepseek_v2.cpp:479-615; supersedes the partial §14 VT_KIMI_BF16_RESIDUAL RoundDevBf16 knob), gated behind VT_KIMI_BF16_STREAM (default OFF). GemmBf16 elides the per-GEMM CastBf16 when the act is already bf16; AddRmsNormS builds a lossless bf16 norm weight (CUDA RmsNorm/FusedChain require weight.dtype==x.dtype, cuda_ops.cu:452,3480); ToStream rounds block outputs; the MoE per-expert gather strides in the stream dtype. Applied identically to the recompute + paged-incremental bodies. GB10 full 48.9B 128-gate (flock, drop_caches, min-avail 18G, no reboot; golden bfa5bdbf…): CONTROL (f32 stream) reproduces §19's 122/128 @ 18.9 tok/s (3×); +bf16-stream REGRESSES to 4/128 @ 19.8 — the KDA recurrence destabilizes into degenerate repeat loops (the §14/§15 pathology, confirmed structurally); no speed win. Diagnostic (bf16 recompute+f64-island) 5/128 vs §14's f32-variance BF16_RESIDUAL=106 — the structural stream computes the RMSNorm variance over the bf16-rounded residual (vLLM's actual fused_add_rms_norm order), even less stable. Both bf16 variants far below the f32 control's 122. REFUTES the §19 "bf16 stream closes p7 + wins speed" hypothesis. With §14 (host- precision plateau 120), §15 (device-KDA 122), §16 (device-MLA 109), §18 (chunk-every- step 102): p7 is an INTRINSIC near-tie; 122/128 @ 18.9 tok/s (0.90× vLLM) is the coherent best; STRICT is NOT reachable by residual-precision or device-island levers. The knob stays default-OFF as a documented-measured-negative A/B (§14/§16 precedent). Also records the SERVER runner fold scope (ARCH-ONE-SURFACE req 4): the runner ABORTS on Kimi's KV today (VT_CHECK at runner.cpp:489-493 — Kimi lacks the qwen3_5 linear_* config fields + layer_types); the fold's file:line landing points are the named born-on-runner residual (spec §20.3). CPU gate test_kimi_linear_forward 15/15·875 (knob OFF, byte-exact). Records: spec §20, benchmark-record, STATUS/BENCHMARKS/ FEATURES/ENVIRONMENT, NOW, state. FOLLOWING_AGENTS_PROTOCOL Assisted-by: Claude Code:claude-opus-4-8 [ClaudeCode]
mudler
force-pushed
the
row/KIMI-BF16-STREAM-CLOSE
branch
from
August 7, 2026 18:04
404ba02 to
9ad9afb
Compare
localai-bot
changed the title
localai-bot
marked this pull request as ready for review
August 7, 2026 18:06
mudler
added a commit
that referenced
this pull request
Aug 7, 2026
…losed as intrinsic near-tie; SERVER fold scoped (#118) The #113 follow-on campaign, an honest measured negative. Structural bf16 residual stream (VT_KIMI_BF16_STREAM, mirrors deepseek_v2.cpp FusedChain order) DESTABILIZES the KDA recurrence into degenerate repeat loops: GB10 full-48.9B 128-token gate scores 4/128 (diagnostic variant 5/128) vs the f32 control's 122/128 @ 18.9 tok/s (reproduced 3x, golden bfa5bdbf). No speed win (19.8 within noise). With Sec.14 (host plateau 120), Sec.15 (device-KDA 122), Sec.16 (device-MLA 109), Sec.18 (chunk-every-step 102) and now Sec.20: STRICT 128/128 is NOT reachable by residual-precision or device-island levers; p7 is an intrinsic near-tie and 122/128 @ 0.90x vLLM is Kimi's coherent best. Knob lands default-OFF as a documented-measured-negative A/B (the VT_KIMI_DEVICE_MLA precedent); CPU gate knob-OFF byte-exact 15/15/875 (re-run by the operator on this merged tree). SERVER runner fold (ARCH-ONE-SURFACE req 4) scoped, enabling-blocked: the runner aborts on Kimi KV at runner.cpp:489-493 (needs layer_types + GDN geometry synthesized from linear_attn_config, a KDA-paged block, a NoPE-MLA paged block, ForwardDevice binding) - spec Sec.20.3, tracked as the single remaining STRICT/speed/server lever. FOLLOWING_AGENTS_PROTOCOL Assisted-by: Claude Code:claude-fable-5 [ClaudeCode]
Collaborator
Author
|
Landed on main as b8493d5 (mudler-authored squash via commit-tree). Operator-run gates on the merged tree: test_kimi_linear_forward 15/15 · 875 assertions knob-OFF byte-exact (fresh CPU build), plus the nine record/doc gates unpiped — 000000000. Conflict resolution: keyed records rebuilt from main wholesale + this branch's row edits re-applied; state entry appended with anchor adjacency verified. The campaign's verdict is recorded: STRICT closed as intrinsic near-tie (122/128 @ 18.9 tok/s = 0.90x vLLM is the coherent best), VT_KIMI_BF16_STREAM stays default-OFF as a documented measured negative, and the single remaining lever (paged runner fold, spec §20.3 / ONE-SURFACE ROW 7) is tracked as task #281. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The #113 follow-on tested the §19-named residual #1 — a bf16 residual stream end-to-end — as the STRICT+speed lever, and precisely scopes the coordinator-directed SERVER runner fold (ARCH-ONE-SURFACE req 4). Spec §20.
Result: bf16 residual stream REFUTED (measured-negative)
STRUCTURAL bf16 stream (bf16 DBufs for hidden/residual/normed-hidden/block-outputs via
vt::FusedChain(kFusedAddRmsNormStd), mirroringdeepseek_v2.cpp:479-615; supersedes the partial §14VT_KIMI_BF16_RESIDUALknob), gated behindVT_KIMI_BF16_STREAM(default OFF).GB10 full 48.9B 128-gate (flock, drop_caches, min-avail 18G, no reboot; golden
bfa5bdbf…):--incremental,DEVICE_KDA_CHUNK=1)The bf16 residual rounding DESTABILIZES the KDA recurrence into degenerate repeat loops (
15383,387,15383,387…) — the §14/§15 pathology confirmed structurally. No speed win (18.9→19.8, noise). The diagnostic (5/128 vs §14's f32-variance BF16_RESIDUAL=106) shows the structural stream computes the RMSNorm variance over the bf16-rounded residual (vLLM-faithful), even less stable.REFUTES the §19 "bf16 stream closes p7 + wins speed" hypothesis. With §14 (plateau 120), §15 (device-KDA 122), §16 (device-MLA 109), §18 (chunk 102): p7 is an intrinsic near-tie; 122/128 @ 18.9 tok/s (0.90× vLLM) is Kimi's coherent best; STRICT is NOT reachable by residual-precision or device-island levers. The knob stays default-OFF as a documented-measured-negative A/B (§14/§16 precedent).
SERVER runner fold (ARCH-ONE-SURFACE req 4): SCOPED, enabling-blocked
The runner ABORTS on Kimi's KV today:
VT_CHECK(mamba_spec->shapes==…)atrunner.cpp:489-493fails because Kimi lacks the qwen3_5linear_*config fields +layer_types(its KDA split lives inlinear_attn_config). The fold = synthesize layer_types + source GDN geometry fromlinear_attn_config+ a Kimi KDA-paged block (KdaChunkPrefill/KdaGatedDeltaRuleovergdn_state) + a NoPE-MLA-paged block (ForwardMlaAttentionBlockidentity-RoPE) + bind inForwardDevice. A multi-brick, runner-touching integration (gate-model regression risk), NOT landed this campaign — the named born-on-runner residual (spec §20.3).Gates
test_kimi_linear_forward15/15·875 (knob OFF, byte-exact); knob ON keeps the greedy-token state-carry check (case-l's 1e-5 logit tol is too tight for bf16).agent-preflight.sh).Base
origin/main2f029a10(branch is behind current main; rebase before merge). This lands a documented-measured-negative knob + the honest refutation records + the precise fold scope.🤖 Generated with Claude Code
https://claude.ai/code/session_014fZAcg1WcU8V629k6HWKys