Skip to content

model(kimi_linear): ONE-SURFACE ROW 7 — fold Kimi-Linear onto the shared paged runner - #122

Closed
localai-bot wants to merge 2 commits into
mainfrom
row/KIMI-RUNNER-FOLD
Closed

model(kimi_linear): ONE-SURFACE ROW 7 — fold Kimi-Linear onto the shared paged runner#122
localai-bot wants to merge 2 commits into
mainfrom
row/KIMI-RUNNER-FOLD

Conversation

@localai-bot

@localai-bot localai-bot commented Aug 7, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator

ARCH-ONE-SURFACE ROW 7 / kimi-linear.md §20.3→§21 (task #281)

Fold Kimi-Linear onto the shared paged runner. STRICT stays CLOSED (intrinsic near-tie); this PR delivers the runner fold, the server surface, and the ONE-SURFACE example rewrite.

Landed

  • B1 KV enablement: LoadHfConfig synthesizes layer_types + GDN geometry from linear_attn_config (additive; qwen3_5 explicit-field path byte-identical; runner.cpp untouched).
  • B2 KDA-paged block: vt::KdaChunkPrefill prefill / vt::KdaGatedDeltaRule decode over the paged gdn_state group (state slots via non_spec_state_indices); conv taps in vLLM's chunk(3) layout. Shared GDN kernels untouched.
  • B3 NoPE-MLA-paged block: vt::ConcatAndCacheMla latent writes; production arm = mla::ForwardMlaAttentionBlock (absorbed-MQA decode / FA2 prefill, identity-RoPE, load-time AbsorbKvBProjBf16) — GB10-ruled default-ON; diagnostic exact-island arm kept (VT_KIMI_PAGED_MLA_FA2=0).
  • B4 ONE SURFACE: engine loads the bf16-resident tower (stage_on_load, §13 recipe); ForwardPaged honors device_token_ids (async device mirror — a measured 9/128 divergence, RED-first pinned); vllm_complete_tokens (ABI v12); kimi_linear_gen rewritten as a thin vllm.h client; allowlist 11→10; CLI reference leg preserved as env-gated test_kimi_linear_fold_gate.

Gates (GB10, golden bfa5bdbf)

gate result
SACRED 35B post-fold test_qwen36_paged_engine 2/2·315 PASS
SACRED 27B post-fold test_qwen27_paged_engine 1/1·235 PASS
CLI reference leg 122/128 @ 18.93 tok/s (reproduces §19; p7 got byte-equal)
Engine leg (default) == CLI 128/128 BYTE-IDENTICAL; vs golden 122/128 (same near-tie profile, ≥122 bound MET)
Diagnostic exact arm 111/128 (§19 GPU M-tiling near-tie class — recorded, not a bug)
Server smoke /v1/completions PASS — streamed 48 tok / 2.52 s = 19.0 tok/s wall, coherent (tiktoken→tokenizer.json via TikTokenConverter, round-trip verified)
Speed vs vLLM server 19.0 wall vs #111 floor ~21 → ~0.90×, ≥ vLLM NOT met (residuals: KDA host islands, grouped MoE, decode graph). In-session vLLM re-measure ABORTED — the util-0.82 leg hard-rebooted the GB10 at graph capture (§19-documented risk; not retried per the safety mandate)
CPU test_kimi_linear_paged 8/8·206 (logits byte-equal, mutation-verified ×4), test_capi 35/35·290, ctest 351/351, preflight EXIT=0

Residuals (recorded in spec §21): engine speed ≥ vLLM (last thread), device-token-ids audit for non-qwen/kimi models under the async mirror, shippable tiktoken converter, exact-arm p2 near-tie cascade (diagnostic arm only).

🤖 Generated with Claude Code

https://claude.ai/code/session_014fZAcg1WcU8V629k6HWKys

@mudler
mudler force-pushed the row/KIMI-RUNNER-FOLD branch from ef120d5 to 0eaa06c Compare August 7, 2026 21:28
@localai-bot

localai-bot commented Aug 7, 2026

Copy link
Copy Markdown
Collaborator Author

Operator disposition: PARKED / FIX-REQUIRED at reviewed head 0eaa06c8.

Fresh mutation review found:

  • documentation-checkpoint is red: docs/STATUS.md and docs/BENCHMARKS.md are missing for the checkpoint, with stale fold/surface claims remaining.
  • the real-model fold gate skips to 0 cases/0 assertions without the 91.5 GiB model and does not exercise the server/golden binding gate;
  • zeroing every vllm_complete_tokens output ID stays green, and negative max_out_tokens is untested;
  • removing decode SSM scatter and forcing prefill state slot 0 leave advertised token/slot tests green;
  • the shared MLA FA2 arm is default-off and untested at this reviewed head.

A fresh fix helper was started, but stopped without committing when a separate live session introduced an overlapping uncommitted edit in the PR worktree with an unrecorded GPU-evidence claim. That concurrent writer must finish/reconcile its work first; then the findings require a fresh scoped mutation re-review. Do not mark ready or merge on the current head.

@mudler
mudler force-pushed the row/KIMI-RUNNER-FOLD branch from 0eaa06c to 6287ead Compare August 7, 2026 22:16
mudler added a commit that referenced this pull request Aug 7, 2026
@mudler
model(kimi_linear): ROW 7 — fold Kimi-Linear onto the shared paged ru…
e76f7ae 
…nner; engine==CLI 128/128, golden 122/128 profile; ABI v12 vllm_complete_tokens; thin-client example (allowlist 11->10)

Branch: row/KIMI-RUNNER-FOLD (draft PR #122 — the helper claim).

ARCH-ONE-SURFACE ROW 7 / kimi-linear.md §20.3→§21, task #281. The runner
previously ABORTED on Kimi's KV (MambaSpec check vs config-derived zeros); the
fast paged-incremental decode was CLI-only. Now Kimi decode runs THROUGH
ModelRegistry::Forward on the runner's own paged state and the engine/server
serve it.

B1 — KV enablement: LoadHfConfig synthesizes layer_types + the GDN-group
geometry from Kimi's nested linear_attn_config (configs/kimi_linear.py:34-148,
1-indexed kda_layers). ADDITIVE: explicit-field configs (qwen3_5) never enter
the branch; runner.cpp untouched. Mutation-verified (test_hf_config 17/17·180).

B2 — KDA-paged block (KdaLayerPagedBf16): vt::KdaChunkPrefill for fresh
prefills (vLLM's prompt path; VT_KIMI_PAGED_KDA_CHUNK=0 A/B) /
vt::KdaGatedDeltaRule (T==1) for decode + continuing prefills over the paged
gdn_state group keyed by non_spec_state_indices (GdnStateGather/Scatter); conv
taps via CausalConv1dFwd (varlen) / CausalConv1dUpdate (decode) in vLLM's
conv_state.chunk(3) [q|k|v] layout. NOT per-head GdnBlockPaged — KDA's
per-K-channel decay needs the KDA ops; shared GDN kernels untouched.

B3 — NoPE-MLA-paged block: latent rows via vt::ConcatAndCacheMla at
attn_meta.slot_mapping (bf16 pages; the conv cache dtype now follows
ResolveKvCacheDType, mirroring kda_state_dtype). PRODUCTION arm =
mla::ForwardMlaAttentionBlock (vLLM's absorbed-MQA decode / FA2 prefill,
identity-RoPE, scale qk^-0.5, load-time AbsorbKvBProjBf16 into
MlaResidentWeights::w_uk_t/w_uv) — VT_KIMI_PAGED_MLA_FA2 DEFAULT ON,
GB10-ruled: it reproduces the golden's 122/128 near-tie profile EXACTLY.
Diagnostic arm ('0') = the exact f64 island (CPU fold-identity vehicle; GB10
111/128 = the §19 GPU M-tiling near-tie class, not a paging bug).

B4 — ONE SURFACE: registry loads the bf16-RESIDENT tower through the engine
(ModelFactory::stage_on_load — queue before load, per-tensor stage-and-release,
the §13 recipe); ForwardPaged honors device_token_ids (the DEFAULT-ON async
device mirror leaves host ids stale — a measured GB10 9/128 divergence,
RED-first CPU-pinned); vllm_complete_tokens (ABI v12, pre-tokenized completion
returning generated ids, test_capi 35/35·290 mutation-verified);
examples/kimi_linear_gen REWRITTEN as a thin vllm.h + vllm::shared client;
example-abi-allowlist kimi row REMOVED, MAX_INTERNAL_REACHING 11->10, equality
pin + spec claims moved together; the CLI-incremental reference leg preserved
as the env-gated test_kimi_linear_fold_gate.

GB10 gates (dgx, /dev/shm CUDA build: CUTLASS 4.5.0 + FA2 + Triton AOT
sm_121a; golden md5 bfa5bdbf; flock, drop_caches, worker parked, no reboot):
- SACRED post-fold: 35B test_qwen36_paged_engine 2/2·315 PASS; 27B
  test_qwen27_paged_engine 1/1·235 PASS.
- Gate A fold identity: CLI reference leg reproduces §19 EXACTLY (122/128 @
  18.93 tok/s, p7 got byte-equal); ENGINE leg (vllm_engine_load +
  vllm_complete_tokens, FA2 default) == CLI 128/128 BYTE-IDENTICAL; vs golden
  122/128 — the SAME near-tie profile (>=122 bound MET).
- Server smoke /v1/completions: PASS — streamed 48 tokens / 2.52 s = 19.0
  tok/s wall, coherent; non-streamed coherent; tokenizer via TikTokenConverter
  (round-trip verified; Kimi ships tiktoken-only).
- SPEED: server 19.0 tok/s wall (~0.90x the #111 vLLM ~21 floor) — >= vLLM
  NOT met; residual: per-step KDA host islands, grouped MoE, decode graph
  (spec §21). Example two-length diffs (16.9 N=64) carry cold-start pollution
  — measurement caveat recorded.
- CPU: test_kimi_linear_paged 8/8·206 (ForwardPaged logits BYTE-EQUAL the CLI
  at every step, mutation-RED on 4 seams; slot isolation; batched-prefill slot
  writes; device_token_ids pin; FA2-arm greedy == exact-arm + f32-page
  rejection); full ctest 351/351.

Records: spec §21, STATUS/BENCHMARKS/FEATURES keyed rows, model-matrix row,
state.md append + NOW refresh, benchmark-record append, ENVIRONMENT.md knobs.

FOLLOWING_AGENTS_PROTOCOL
Assisted-by: Claude Code:claude-fable-5 [ClaudeCode]
@mudler
mudler force-pushed the row/KIMI-RUNNER-FOLD branch from 6287ead to e76f7ae Compare August 7, 2026 22:18
mudler added a commit that referenced this pull request Aug 7, 2026
@mudler
records(kimi_linear): ROW 7 — vLLM re-measure leg aborted by box rebo…
76bddec 
…ot (recorded)

Branch: row/KIMI-RUNNER-FOLD (#122). The same-session vLLM denominator re-measure
(oracle venv, util 0.82 — the #111-precedented config, no tracing) loaded 20/20
shards then HARD-REBOOTED the GB10 at torch.compile/graph capture, reproducing the
§19 measured box-safety violation (min-avail at the 15-17G floor). NOT retried per
the safety mandate; the campaign denominator remains the #111 recorded ~21 floor.
Box recovered clean (single reboot, GPU visible); local-ai-worker auto-restored by
its --restart=always policy.

FOLLOWING_AGENTS_PROTOCOL
Assisted-by: Claude Code:claude-fable-5 [ClaudeCode]
mudler added a commit that referenced this pull request Aug 7, 2026
@mudler
records(kimi_linear): ROW 7 — vLLM re-measure leg aborted by box rebo…
6d0cf5b 
…ot (recorded)

Branch: row/KIMI-RUNNER-FOLD (#122). The same-session vLLM denominator re-measure
(oracle venv, util 0.82 — the #111-precedented config, no tracing) loaded 20/20
shards then HARD-REBOOTED the GB10 at torch.compile/graph capture, reproducing the
§19 measured box-safety violation (min-avail at the 15-17G floor). NOT retried per
the safety mandate; the campaign denominator remains the #111 recorded ~21 floor.
Box recovered clean (single reboot, GPU visible); local-ai-worker auto-restored by
its --restart=always policy.

FOLLOWING_AGENTS_PROTOCOL
Assisted-by: Claude Code:claude-fable-5 [ClaudeCode]
@mudler
mudler force-pushed the row/KIMI-RUNNER-FOLD branch from 76bddec to 6d0cf5b Compare August 7, 2026 22:35
@localai-bot
localai-bot marked this pull request as ready for review August 7, 2026 22:36
@localai-bot localai-bot mentioned this pull request Aug 7, 2026
Closed
mudler added 2 commits August 7, 2026 23:24
@mudler
claim(kimi-linear): ROW 7 ONE-SURFACE runner fold — row/KIMI-RUNNER-F…
1798b27 
…OLD claim

Opens the helper claim for ARCH-ONE-SURFACE ROW 7 / kimi-linear.md §20.3:
fold Kimi-Linear onto the shared paged runner (B1 KV enablement, B2 KDA-paged
block, B3 NoPE-MLA-paged block, B4 engine bind + thin-client example).
Task #281.

FOLLOWING_AGENTS_PROTOCOL
Assisted-by: Claude Code:claude-fable-5 [ClaudeCode]
@mudler
model(kimi_linear): ROW 7 — fold Kimi-Linear onto the shared paged ru…
5b39361 
…nner; engine==CLI 128/128, golden 122/128 profile; ABI v13 vllm_complete_tokens; thin-client example (merged ratchet 8)

Branch: row/KIMI-RUNNER-FOLD (draft PR #122 — the helper claim).
ARCH-ONE-SURFACE ROW 7 / kimi-linear.md §20.3→§21, task #281. The runner
previously ABORTED on Kimi's KV (MambaSpec check vs config-derived zeros); the
fast paged-incremental decode was CLI-only. Now Kimi decode runs THROUGH
ModelRegistry::Forward on the runner's own paged state and the engine/server
serve it.

B1 — KV enablement: LoadHfConfig synthesizes layer_types + the GDN-group
geometry from Kimi's nested linear_attn_config (configs/kimi_linear.py:34-148,
1-indexed kda_layers). ADDITIVE: explicit-field configs (qwen3_5) never enter
the branch — the priority conjunct is PINNED by a both-present test (explicit
fields WIN; mutation-verified RED when the conjunct is dropped); runner.cpp
untouched.

B2 — KDA-paged block (KdaLayerPagedBf16): vt::KdaChunkPrefill for fresh
prefills (VT_KIMI_PAGED_KDA_CHUNK=0 A/B) / vt::KdaGatedDeltaRule (T==1) for
decode + continuing prefills over the paged gdn_state group keyed by
non_spec_state_indices (GdnStateGather/Scatter); conv taps via CausalConv1dFwd
(varlen) / CausalConv1dUpdate (decode) in vLLM's conv_state.chunk(3) [q|k|v]
layout. NOT per-head GdnBlockPaged — KDA's per-K-channel decay needs the KDA
ops; shared GDN kernels untouched.

B3 — NoPE-MLA-paged block: latent rows via vt::ConcatAndCacheMla at
attn_meta.slot_mapping (bf16 pages; conv cache dtype follows
ResolveKvCacheDType, mirroring kda_state_dtype). PRODUCTION arm =
mla::ForwardMlaAttentionBlock (vLLM's absorbed-MQA decode / FA2 prefill,
identity-RoPE, scale qk^-0.5, load-time AbsorbKvBProjBf16 into
MlaResidentWeights::w_uk_t/w_uv) — VT_KIMI_PAGED_MLA_FA2 DEFAULT ON,
GB10-ruled: it reproduces the golden's 122/128 near-tie profile EXACTLY.
Diagnostic arm ('0') = the exact f64 island (CPU fold-identity vehicle; GB10
111/128 = the §19 GPU M-tiling near-tie class, not a paging bug).

B4 — ONE SURFACE: registry loads the bf16-RESIDENT tower through the engine
(ModelFactory::stage_on_load — queue before load, per-tensor
stage-and-release, the §13 recipe); ForwardPaged honors device_token_ids (the
DEFAULT-ON async device mirror leaves host ids stale — a measured GB10 9/128
divergence, RED-first CPU-pinned); vllm_complete_tokens (ABI v13 after the
#123 video-v12 rebase; changelog slotted; floor pin >= 13; dlopen 28/28 +
c_header_compile reference); examples/kimi_linear_gen REWRITTEN as a thin
vllm.h + vllm::shared client; example-abi-allowlist kimi row REMOVED — merged
ratchet MAX_INTERNAL_REACHING = 8 (with #123's two minimax removals), equality
pin 8, coupled spec/state claims moved together; the CLI-incremental reference
leg preserved as the env-gated test_kimi_linear_fold_gate.

GB10 gates (dgx, /dev/shm CUDA build: CUTLASS 4.5.0 + FA2 + Triton AOT
sm_121a; golden md5 bfa5bdbf; flock, drop_caches, worker parked, no reboot
during the gates):
- SACRED post-fold: 35B test_qwen36_paged_engine 2/2·315 PASS; 27B
  test_qwen27_paged_engine 1/1·235 PASS.
- Gate A fold identity: CLI reference leg reproduces §19 EXACTLY (122/128 @
  18.93 tok/s, p7 got byte-equal); ENGINE leg (vllm_engine_load +
  vllm_complete_tokens, FA2 default) == CLI 128/128 BYTE-IDENTICAL; vs golden
  122/128 — the SAME near-tie profile (>=122 bound MET).
- Server smoke /v1/completions: PASS — streamed 48 tokens / 2.52 s = 19.0
  tok/s wall, coherent; non-streamed coherent; tokenizer via TikTokenConverter
  (round-trip verified; Kimi ships tiktoken-only).
- SPEED: server 19.0 tok/s wall (~0.90x the #111 vLLM ~21 floor) — >= vLLM
  NOT met; residual: per-step KDA host islands, grouped MoE, decode graph
  (spec §21). Example two-length diffs (16.9 N=64) carry cold-start pollution
  — measurement caveat recorded.
- The in-session vLLM re-measure ABORTED BY BOX REBOOT (util 0.82 at graph
  capture — the §19-documented risk; not retried per the safety mandate; the
  #111 ~21 floor stays the denominator; box recovered clean, worker
  auto-restored).
- CPU: test_kimi_linear_paged 8/8·206 (ForwardPaged logits BYTE-EQUAL the CLI
  at every step, mutation-RED on 4 seams; slot isolation; batched-prefill slot
  writes; device_token_ids pin; FA2-arm greedy == exact-arm + f32-page
  rejection; pid-unique temp fixtures); test_hf_config 18/18·187; test_capi
  400; test_dlopen 28/28; full ctest 351/351 (pre-rebase base).

Records: spec §21, STATUS/BENCHMARKS/FEATURES keyed rows, model-matrix row,
state.md append + NOW refresh, benchmark-record append, ENVIRONMENT.md knobs
(VT_KIMI_PAGED_MLA_FA2 documented default-ON, GB10-ruled).

FOLLOWING_AGENTS_PROTOCOL
Assisted-by: Claude Code:claude-fable-5 [ClaudeCode]
@mudler
mudler force-pushed the row/KIMI-RUNNER-FOLD branch from 6d0cf5b to 5b39361 Compare August 7, 2026 23:26
@localai-bot

localai-bot commented Aug 7, 2026

Copy link
Copy Markdown
Collaborator Author

Review fixes + landing reconciliation pushed (head 5b39361b, preflight EXIT=0):

  • F1: B1 priority conjunct PINNED — both-present config (explicit qwen3_5 linear_* + linear_attn_config) keeps the explicit fields; mutation-verified RED without the conjunct (test_hf_config 18/18·187).
  • F2: ENVIRONMENT.md VT_KIMI_PAGED_MLA_FA2 corrected to the shipped default-ON with the GB10 §21 basis.
  • F3: vllm_complete_tokens in test_dlopen (28/28) + referenced in c_header_compile.c.
  • F4: CMake comment v13; ratchet test renamed test_ratchet_ceiling_pinned_at_8; TempFile names pid-unique.
  • R1: rebased onto main (feat(abi): MiniMax-H3 video+audio generation through the ONE surface — vllm_video_*, ABI v12, /v1/videos via the library seam, thin-client examples #123 video v12) — VLLM_ABI_VERSION 13, floor pin >= 13, full v12→v13 prose sweep, changelog slotted. v14 left for the device-knob lane.
  • R2/R3: merged allowlist (kimi + both minimax rows gone) → MAX_INTERNAL_REACHING = 8, equality pin 8, coupled spec/state claims moved together; keyed files taken from main wholesale with my rows re-applied; state entry re-anchored after main's tail; STATUS compacted within my row to hold its shrink ratchet.
  • Re-run green: test_hf_config 18/18·187, test_kimi_linear_paged 8/8·206, test_kimi_linear_forward 875, test_capi 400, test_dlopen 28/28, surface-coverage OK (8 internal-reachers) + 46-test suite, agent-preflight EXIT=0.

mudler added a commit that referenced this pull request Aug 7, 2026
@mudler
feat(abi): explicit device selection through the ONE surface - vllm_m…
0153d6c 
…odel_params.device, ABI v14, --device on server + cli

ONE-SURFACE fold ROW 8 (row/DEVICE-KNOB, task #284), the smallest: an
embedder can now ASK for a device instead of inheriting the
accelerator-first probe. Mirrors vLLM's DeviceConfig.device names
(vllm/config/device.py:13 @ 555967922; an explicit device is assigned
VERBATIM and never substituted, device.py:61-66 - upstream's own
--device flag was removed in v0.10, bc8a8ce5e, so the field mirrors
the surviving config surface).

include/vllm.h gains vllm_model_params.device (int32, 0=auto/1=cpu/
2=cuda; 0 MUST be auto so zero-initialized structs keep the pre-v14
probe byte-identical; cpu-before-cuda follows the shipped v12
vllm_video_model_params.device precedent shifted by the auto slot).
VLLM_ABI_VERSION -> 14 per the operator's version-collision heads-up
(#122 reconciles the Kimi vllm_complete_tokens slice to v13 at its
landing; the changelog names the gap and this branch rebases after).
Floor pin advanced to >= 14 (the #121 == lesson). New
vllm/config/device.h (+.cpp): vllm::Device, DeviceFromString,
DeviceName. Plumb: EngineParams::device -> SelectQueue explicit arms
via the pure LoadedEngine::ResolveExplicitDeviceType (explicit cpu
NEVER probes; explicit cuda requires the registered kCUDA platform and
fails LOUD otherwise; explicit-arm queue-creation failures propagate,
unlike auto's byte-identical catch-and-fall-back). FromModelDir
resolves an explicit device BEFORE any path/config I/O (DeviceConfig
resolves at config time, arg_utils.py:1878), so a device error is
never masked by a path error. capi validates 0/1/2 and refuses
explicit cuda on the CPU-hosted transcription stack. Thin clients:
--device auto|cpu|cuda on vllm-cli (ABI field only) and the server
(DeviceFromString; --video-device untouched, separate engine).

Also folds the #123 review's three MINOR findings (tests/docs only):
c_header_compile.c actually references the v11+v12 surface + the v14
field (its "every entry point" claim went stale at v10); the v12
changelog block moved to chronological position; the H3 fold fixture
flag list gained --keep-quant.

Gates: full CPU build -Werror clean; test_capi (4 new v14 cases +
floor), test_loaded_engine_dense (+3 device cases incl. the pure
policy matrix that pins "explicit cpu beats a REGISTERED accelerator"
CPU-side), test_dlopen, test_openai_api_server (+explicit-cpu serving
smoke), test_minimax_h3_video_fold, test_model_loader_gguf,
test_platform all green; surface-coverage green, ratchet UNTOUCHED at
9; STATUS ratchet paid by removing a stale back-to-back merge
duplicate (279200 -> 279150). Ten mutations killed and reverted
(defaults, zero-map, range, both plumb sites, ctor arm, unconditional
fallback, policy flip, parse flip, forgotten bump); kills recorded in
.agents/state.md. Residuals: CUDA-build A/B (no dgx in scope) and the
pending #122 rebase, both named in the state entry.

FOLLOWING_AGENTS_PROTOCOL
Assisted-by: Claude Code:claude-fable-5 [ClaudeCode]
@localai-bot

localai-bot commented Aug 7, 2026

Copy link
Copy Markdown
Collaborator Author

Landed on main as 9f772ca (mudler-authored squash via commit-tree). Two-round review: fresh mutation review (SACRED scope verified untouched, fold seams/staleness pin/zero-value contract all killable, stage_on_load De-Morgan-verified) -> 4 findings fixed + v13/ratchet-8 reconciliation -> scoped re-review MERGE-CLEAN with every fix kill-verified. Operator landing: merged tree byte-identical to the reviewed head (TREE_DELTA=0), five row suites 5/5 SUCCESS on a fresh build, eleven gates 00000000000. Kimi-Linear is now fully on the one surface: shared paged runner (engine==CLI 128/128 byte-identical, SACRED green), /v1/completions serving, ABI v13 vllm_complete_tokens, thin-client example, ratchet 8. Honest open thread: engine speed ~0.90x vLLM, levers in spec §21. The device-knob PR #136 (v14) rebases next.

mudler added a commit that referenced this pull request Aug 7, 2026
@mudler
feat(kimi_linear): Kimi on the shared paged runner + server + ABI v13…
9f772ca 
… - KDA-paged, NoPE-MLA-paged, engine==CLI byte-identical (#122)

ONE-SURFACE fold ROW 7 / spec kimi-linear.md sec.20.3->21. The runner
abort on Kimi KV dies CONFIG-SIDE: LoadHfConfig synthesizes layer_types
+ GDN geometry from linear_attn_config (runner.cpp untouched; explicit
qwen3_5-style fields WIN over synthesis, pinned by a both-present
config case). KdaLayerPagedBf16 (KdaChunkPrefill prefill /
KdaGatedDeltaRule decode over the paged gdn_state group) + paged
NoPE-MLA (ConcatAndCacheMla latent writes; FA2 ForwardMlaAttentionBlock
identity-RoPE default-ON by GB10 measurement, f64-island diagnostic arm
kept). The engine stages the bf16-resident tower at load; ForwardPaged
honors device_token_ids (async-mirror staleness was a measured 9/128
divergence, RED-first pinned). vllm_complete_tokens lands on the C ABI
(v13, floor-pinned >= 13); examples/kimi_linear_gen is a vllm.h-only
thin client; surface ratchet 9->8.

Gates: SACRED post-fold 35B 2/2 + 27B 1/1 on GB10; engine==CLI
128/128 BYTE-IDENTICAL; vs the vLLM golden 122/128, the same intrinsic
near-tie profile as the CLI reference; /v1/completions serves Kimi at
19.0 tok/s wall. HONEST NOT-MET: engine speed ~0.90x of vLLM ~21 -
levers named in sec.21 (device KDA gates, grouped MoE seam, decode
graph). The in-session vLLM re-measure hard-rebooted the GB10 (the
documented util-0.82 capture trigger) and was not retried per the
safety mandate; recorded, box recovered clean.

Two-round review: fresh mutation review (SACRED scope, fold seams,
staleness pin, stage_on_load De-Morgan check) -> 4 findings fixed +
v13/ratchet-8 reconciliation -> scoped re-review MERGE-CLEAN, every
fix kill-verified. Operator: merged tree byte-identical to the
reviewed head; kimi_paged/forward/hf_config/capi/dlopen 5/5 SUCCESS.

FOLLOWING_AGENTS_PROTOCOL
Assisted-by: Claude Code:claude-fable-5 [ClaudeCode]
@localai-bot localai-bot closed this Aug 7, 2026
mudler added a commit that referenced this pull request Aug 7, 2026
@mudler
feat(abi): explicit device selection through the ONE surface - vllm_m…
3f8de74 
…odel_params.device, ABI v14, --device on server + cli

ONE-SURFACE fold ROW 8 (row/DEVICE-KNOB, task #284), the smallest: an
embedder can now ASK for a device instead of inheriting the
accelerator-first probe. Mirrors vLLM's DeviceConfig.device names
(vllm/config/device.py:13 @ 555967922; an explicit device is assigned
VERBATIM and never substituted, device.py:61-66 - upstream's own
--device flag was removed in v0.10, bc8a8ce5e, so the field mirrors
the surviving config surface).

include/vllm.h gains vllm_model_params.device (int32, 0=auto/1=cpu/
2=cuda; 0 MUST be auto so zero-initialized structs keep the pre-v14
probe byte-identical; cpu-before-cuda follows the shipped v12
vllm_video_model_params.device precedent shifted by the auto slot).
VLLM_ABI_VERSION 13 -> 14 on top of #122's vllm_complete_tokens v13
(rebased onto 9f772ca; the changelog now reads v10..v14 chronological
- the v12-after-v11 move is the #123-review minor kept through the
rebase). Floor pin advanced to >= 14 (the #121 == lesson). New
vllm/config/device.h (+.cpp): vllm::Device, DeviceFromString,
DeviceName. Plumb: EngineParams::device -> SelectQueue explicit arms
via the pure LoadedEngine::ResolveExplicitDeviceType (explicit cpu
NEVER probes; explicit cuda requires the registered kCUDA platform and
fails LOUD otherwise; explicit-arm queue-creation failures propagate,
unlike auto's byte-identical catch-and-fall-back). FromModelDir
resolves an explicit device BEFORE any path/config I/O (DeviceConfig
resolves at config time, arg_utils.py:1878), so a device error is
never masked by a path error. capi validates 0/1/2 and refuses
explicit cuda on the CPU-hosted transcription stack. Thin clients:
--device auto|cpu|cuda on vllm-cli (ABI field only) and the server
(DeviceFromString; --video-device untouched, separate engine).

Also folds the #123 review's three MINOR findings (tests/docs only):
c_header_compile.c actually references the v11+v12 surface + the v14
field (composed with #122's vllm_complete_tokens reference; its "every
entry point" claim went stale at v10); the v12 changelog block moved
to chronological position; the H3 fold fixture flag list gained
--keep-quant.

Gates: full CPU build -Werror clean; test_capi (4 new v14 cases +
floor >= 14, #122's v13 cases green alongside), test_loaded_engine_dense
(+3 device cases incl. the pure policy matrix that pins "explicit cpu
beats a REGISTERED accelerator" CPU-side), test_dlopen,
test_openai_api_server (+explicit-cpu serving smoke),
test_minimax_h3_video_fold, test_model_loader_gguf, test_platform all
green; surface-coverage green, allowlist ratchet UNTOUCHED at 8 (post
#122); STATUS ratchet paid by removing a stale back-to-back merge
duplicate (279200 -> 279150, measured 279111 post-rebase). Ten
mutations killed and reverted (defaults, zero-map, range, both plumb
sites, ctor arm, unconditional fallback, policy flip, parse flip,
forgotten bump); kills recorded in .agents/state.md. Residual:
CUDA-build A/B (no dgx in scope), named in the state entry.

FOLLOWING_AGENTS_PROTOCOL
Assisted-by: Claude Code:claude-fable-5 [ClaudeCode]
@localai-bot localai-bot mentioned this pull request Aug 8, 2026
Closed
4 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@localai-bot @mudler