Kimi-Linear-48B W7-speed STRICT lever — bf16 regime MEASURED 106→120/128 (device islands the residual) - #102
Closed
localai-bot wants to merge 1 commit into
Closed
Kimi-Linear-48B W7-speed STRICT lever — bf16 regime MEASURED 106→120/128 (device islands the residual)#102localai-bot wants to merge 1 commit into
localai-bot wants to merge 1 commit into
Conversation
…20/128, plateaus; device islands remain the residual
The recorded STRICT path ("device islands + bf16 residual stream") is implemented as
three env-gated numeric knobs in kimi_linear_device.cpp (default OFF -> the f32
correctness vehicle is byte-identical; CPU gate test_kimi_linear_forward 13/13·656 in
the CUDA binary) and MEASURED on GB10 (full 48.9B model, the §12 128-token gate vs the
STRICT deterministic golden; both flock locks + reclaim-waits, min-avail >=115 GiB, no
reboot):
VT_KIMI_BF16_RESIDUAL bf16 residual stream (vLLM fused_add_rms_norm order: residual
stored bf16, block outputs bf16, RMSNorm variance over the f32 pre-store sum) via
in-place f32<->bf16 rounds, keeping f32 storage so the islands still read f32.
VT_KIMI_BF16_ISLANDS round the host-fallback island INPUTS (KDA q/k/v/g1/beta,
NoPE-MLA q/kv/kpe) to bf16 (RNE) before the recurrence/softmax.
VT_KIMI_ISLAND_F32ACC f32 (not f64) island accumulation -- MEASURED NEGATIVE, kept
as a documented-negative A/B knob.
RESULT (token match /128): control 106 (reproduces §13); residual-alone 106 (net-zero,
shuffles flips, BREAKS p3 into a repeat loop); islands-alone 106 (fixes p2, destabilizes
p3); residual+islands = 120/128 BEST (p0-p6 all 16/16; only p7 pos-8 flips: golden 18705
vs ours 58084, a single near-tie that cascades). The two levers INTERACT -- island
input-rounding fixes p2 but repeats p3; the bf16 residual re-stabilizes p3. Further
precision-matching regresses (island-output bf16 -> 90, reverted; f32 accum -> 91-106):
host-precision-matching PLATEAUS at 120/128 because it is not vLLM's ACTUAL GDN-Triton /
FA2 kernel arithmetic.
VERDICT: NO arm STRICT (K=3-deterministic golden -> STRICT required, not distributional);
default STAYS OFF (parity-enablers). Named residual (= also the speed lever): the device
islands -- a NEW per-channel-decay GDN kernel (g[T,H,D]; vt::GdnDecode/GdnPrefill carry
only per-head g[T,Hv], ops.h:1797/1846 -- NOT a drop-in) + the paged FA2 MLA. Speed:
1.30 tok/s (O(n^2) full-recompute + host islands, invariant to the knobs); vLLM can't
serve Kimi-Linear-48B at bf16 on ONE GB10 (oracle capture needed util 0.82 for a
single-seq eager run) -> HW-forced-indirect. Records: spec §14, STATUS/BENCHMARKS/
FEATURES/ENVIRONMENT keyed rows, NOW, benchmark-record, anchored state entry.
FOLLOWING_AGENTS_PROTOCOL
Assisted-by: Claude Code:claude-opus-4-8 [ClaudeCode]
Collaborator
Author
|
Landed on main as the squash (mudler authorship). The per-channel-decay KDA kernel + paged MLA routing is the scoped follow-up — STRICT and speed in one object. 🤖 Generated with Claude Code |
mudler
added a commit
that referenced
this pull request
Aug 7, 2026
…d to the per-channel KDA kernel (#102) row/KIMI-LINEAR-STRICT-SPEED squash. Two env-gated knobs mirror vLLM's bf16 compute regime on the 48.9B model (full GB10 gate, deterministic K=3 golden => STRICT required): VT_KIMI_BF16_RESIDUAL (bf16 residual stream, fused_add_rms_norm order) + VT_KIMI_BF16_ISLANDS (bf16-round island inputs). KEY FINDING — the levers only work TOGETHER: island rounding fixes p2 but destabilizes p3; the bf16 stream re-stabilizes p3; combined 120/128 (p0-p6 all 16/16; one p7 near-tie cascades). Further host-matching REGRESSES (output rounding 90/128; f32-accum negative, kept as documented A/B knob) — the plateau proves host arithmetic cannot impersonate vLLM's GDN-Triton/FA2 kernels. Default STAYS OFF per parity-enablers. tok/s invariant to the knobs (1.30-1.31 = O(n^2) recompute + per-layer host-island round-trips, not arithmetic); vLLM cannot serve this model bf16 on one GB10, so the comparison stays HW-forced-indirect. THE follow-up (STRICT and speed, one object): a NEW per-channel-decay GDN kernel (KDA g[T,H,D] vs GdnDecode's per-head g[T,Hv], ops.h:1797,1846) + paged mla::ForwardMlaAttentionBlock routing. Unit gate 13/13-656 byte- identical at defaults. Also recorded: dgx interactive ssh gets a per-session-private /dev/shm (defeats tmux cross-call polling); systemd-run system-namespace builds are the workaround. FOLLOWING_AGENTS_PROTOCOL Assisted-by: Claude Code:claude-opus-4-8 [ClaudeCode]
mudler
added a commit
that referenced
this pull request
Aug 7, 2026
… AND 3.1x decode (#104) row/KIMI-KDA-DEVICE-KERNEL squash. The additive vt::KdaGatedDeltaRule op grounds 1:1 in FLA at the pin: fused_recurrent_kda reuses the GDN recurrence with the SOLE delta b_h *= exp(b_gk[None,:]) — per-K-CHANNEL decay g[T,Hv,Dk] vs GDN's per-head scalar (fused_recurrent.py:88-175 vs :132-137). CPU KdaHeadTokenStep + CUDA KdaScanKernel (GdnScanKernel + a third shared-mem dk-array); the shared GDN kernels untouched (test_ops_gdn 66/66). DECISIVE unit gate: broadcast-g reduces BIT-IDENTICALLY (exact float ==) to the landed GdnPrefill — zero new numerics beyond the per-channel decay itself; per-channel case vs an f64 first-principles reference; CPU==CUDA on Blackwell; 4/4 GPU-green. Full 48.9B GB10 gate: device-KDA 122/128 (p0-p6 all 16/16; one p7 comma near-tie) AND 4.24 tok/s (3.1x over 1.35) — vLLM's actual f32-on-bf16 arithmetic on GPU beats #102's host-precision best on BOTH axes and FIXES its p2 divergence. Razor verdict: the #102 bf16 knobs now REGRESS (122->90, reintroducing the p3 repeat loop) — they compensated for the f64 island's over-precision and are superseded. VT_KIMI_DEVICE_KDA opt-in default OFF pending STRICT. The named path to STRICT (spec §15): chunk_kda prefill kernel (vLLM prompts use chunk_kda, we run the recurrent form — Triton-AOT cubin regen or native port) + paged mla::ForwardMlaAttentionBlock for the 7 NoPE-MLA layers + paged-incremental decode (the O(n^2) kill). Also documents the two pre-existing VT_H3_ACT_* env vars and removes a committed conflict marker at the benchmark-record tail. FOLLOWING_AGENTS_PROTOCOL Assisted-by: Claude Code:claude-opus-4-8 [ClaudeCode]
mudler
added a commit
that referenced
this pull request
Aug 7, 2026
… AND 3.1x decode (#104) row/KIMI-KDA-DEVICE-KERNEL squash (the REAL landing; bcf3972 was an ENOSPC-broken push of an unrelated tree under this message, reverted in 17968f9). The additive vt::KdaGatedDeltaRule op grounds 1:1 in FLA at the pin: fused_recurrent_kda reuses the GDN recurrence with the SOLE delta b_h *= exp(b_gk[None,:]) — per-K-CHANNEL decay g[T,Hv,Dk] vs GDN's per-head scalar (fused_recurrent.py:88-175 vs :132-137). CPU KdaHeadTokenStep + CUDA KdaScanKernel; shared GDN kernels untouched (test_ops_gdn 66/66). DECISIVE unit gate: broadcast-g reduces BIT-IDENTICALLY to the landed GdnPrefill — zero new numerics beyond the per-channel decay; CPU==CUDA on Blackwell; 4/4 GPU-green. Full 48.9B GB10 gate: device-KDA 122/128 (p0-p6 all 16/16; one p7 comma near-tie) AND 4.24 tok/s (3.1x over 1.35). Razor verdict: the #102 bf16 knobs now REGRESS (122->90) — they compensated for the f64 island's over-precision and are superseded. VT_KIMI_DEVICE_KDA opt-in default OFF pending STRICT. Named path to STRICT (spec §15): chunk_kda prefill kernel + paged mla::ForwardMlaAttentionBlock + paged- incremental decode. Also documents the VT_H3_ACT_* env vars and removes a committed conflict marker at the benchmark-record tail. FOLLOWING_AGENTS_PROTOCOL Assisted-by: Claude Code:claude-opus-4-8 [ClaudeCode]
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Kimi-Linear-48B-A3B — W7-speed STRICT lever (bf16 regime), MEASURED
Row
MODEL-TEXT-kimi-linear-kimi-linear-for-causal-lm. Implements the recorded path toSTRICT ("device islands + bf16 residual stream") as three env-gated numeric knobs
(default OFF → the f32 correctness vehicle is byte-identical; CPU gate
test_kimi_linear_forward13/13·656 in the CUDA binary), and MEASURES them on GB10(full 48.9B model, the §12 128-token gate vs the STRICT deterministic golden).
Knobs (
kimi_linear_device.cpp)VT_KIMI_BF16_RESIDUAL— bf16 residual stream mirroring vLLM'sfused_add_rms_normorder (f32 sum for the RMSNorm variance, bf16 residual/block-outputs carried forward),
via in-place
CastBf16/CastF32rounds that keep f32 storage so the islands still read f32.VT_KIMI_BF16_ISLANDS— bf16-round the host-fallback island inputs before the recurrence/softmax.VT_KIMI_ISLAND_F32ACC— f32 (not f64) island accumulation. Measured negative, kept as a documented-negative A/B.Result (token match /128 vs the deterministic golden; STRICT required)
Verdict: NO arm reaches STRICT (best 120/128 is a divergence; the golden is K=3-deterministic
so STRICT — not the distributional gate — is required). Per parity-enablers the default STAYS OFF.
Host-precision-matching plateaus at 120/128; the sole remaining divergence is a single deterministic
near-tie (p7 pos 8: golden
18705, ours58084).Named residual (= also the speed lever)
The principled path to STRICT is routing the two islands through vLLM's actual device kernels, but it is
not a drop-in: KDA's decay is per-k-channel
g[T,H,D]whilevt::GdnDecode/GdnPrefillcarry only aper-head scalar
g[T,Hv](ops.h:1797,1846) — a NEW per-channel-decay GDN kernel is required — and NoPE-MLAneeds the paged FA2 path. These are also the speed levers (the vehicle re-computes the whole sequence every
step, O(n²)); measured 1.30 tok/s. vLLM cannot serve Kimi-Linear-48B at bf16 on one GB10 (the oracle
capture itself needed
gpu_memory_utilization=0.82for a single-seq eager run) → speed is HW-forced-indirect.Detail: spec §14;
.agents/benchmark-record.md; anchoredstate.mdentry. Box left clean, memory-safethroughout (both flock locks, reclaim-waits, min-avail ≥ 115 GiB, no reboot).
🤖 Generated with Claude Code
https://claude.ai/code/session_014fZAcg1WcU8V629k6HWKys