row/SERVE-ASYNC-DENSE-MIRROR: classic-dense async device-mirror fix + MXFP4 default-config e2e CLOSED - #44
Closed
localai-bot wants to merge 3 commits into
Closed
row/SERVE-ASYNC-DENSE-MIRROR: classic-dense async device-mirror fix + MXFP4 default-config e2e CLOSED#44localai-bot wants to merge 3 commits into
localai-bot wants to merge 3 commits into
Conversation
…ic dense (ROW-SERVE-ASYNC-DENSE-MIRROR) Classic dense Qwen3ForCausalLM (qwen3.cpp) lacked the async device-token-ids mirror the gate models (qwen3_5) already had, so on the depth-2 AsyncLLM serving path its batch-1 greedy decode nondeterministically degenerated into token-0 garbage (surfaced by the MXFP4 campaign; quant-independent, hits bf16/NVFP4). The shared pure-dense embed EmbedInto uploaded the stale host token_ids, racing the async combine's device input-ids write (unsynchronized device-write/host-read) -> token-0 degeneration when the host read won. Note the classic-dense decode graph is default-OFF, so the racing path is the EAGER ForwardBody->EmbedInto. Fix (byte-identical when the mirror is off), mirroring the 27B-dense template (qwen3_5.cpp:6737/6746): - qwen3.cpp: EmbedInto now consumes ApplyDeviceTokenIdsOverride (main-queue-ordered d.b.Copy over the DBuf prefix), replacing the racing host read with the combine's device ids. Shares the detail::DeviceTokenIds seam (qwen3_5_internal.h). - qwen3_dense.cpp: ForwardQwen3ForCausalLM establishes DeviceTokenIdsScope over the whole forward (eager + decode-graph replay), publishing the override. Gate (RED-first): tests/parity/test_qwen3_dense_async_serving.cpp (Qwen3-0.6B/4B, batch-1 x5 + N=4 concurrency) requires every async continuation to reproduce the race-free in-process SYNC engine continuation token-for-token (the near-tie-proof, drift-proof anchor). RED on VT_ASYNC_DEVICE_MIRROR=0, GREEN on the default; checkpoint-gated + dgx-only. CPU gates green (Release -Werror): library + test build clean; regression suites input_batch/combine_tokens/runner/engine_core_proc/async_llm/llm_engine pass. Residual (named): sibling registries sharing this driver (InternLM2, Mistral, Llama) get the fixed consumer but still need the one-line DeviceTokenIdsScope; own-embed decode models per decode-framework-routing-audit. dgx-owed: async gate RED->GREEN + SACRED dense gates + the MXFP4 W4 online_gate bench + p3 near-tie. FOLLOWING_AGENTS_PROTOCOL Assisted-by: Claude Code:claude-opus-4-8 [ClaudeCode]
…CRED + memcheck + MXFP4 default e2e + near-tie ratified Records the on-hardware verification of ROW-SERVE-ASYNC-DENSE-MIRROR (f9c969a) on dgx.casa GB10 (sm_121a, CUDA build in /dev/shm; both flock locks per run, tmux + done-markers, free-g 98-100 GiB): - Async gate RED->GREEN, same binary env-toggled: 0.6B + 4B GREEN 41/41 (async == in-process SYNC anchor); RED (VT_ASYNC_DEVICE_MIRROR=0) FAIL 3 CHECKs — concurrency requests degenerate into "...the Germany is!!!!!!" token-0 garbage. - SACRED test_qwen3_paged_engine 0.6B + 4B 184/184, 16/16 prompts each, 0 forward-divergent (byte-neutral sync path confirmed). - compute-sanitizer memcheck on the 0.6B async GREEN arm: 0 errors. - MXFP4 Yi30/Qwen3-8B-MXFP4 (classic dense Qwen3ForCausalLM) DEFAULT-config (async ON) e2e now coherent + 3/4 token-exact vs the golden; degenerate without the fix — closes the QUANT-CT-MXFP4 async-default residual. - p2/p3 story near-tie RATIFIED: oracle (VLLM_DISABLED_KERNELS=FlashInferMxFp4Linear Kernel) re-reproduces the golden; teacher-forcing it on our sequence makes our token the oracle's own argmax at every position (max gap 0.0000 nats). Owed residual (recorded, not a regression): the W4 THROUGHPUT bench (online_gate.py c1..c8x3 vs oracle) — the harness has no Yi30/8B model key (needs corpus + oracle-record plumbing). The fix unblocks the default-config number; oracle proven to run the model today. Plus the sibling scope one-liner (InternLM2/Mistral/Llama). FOLLOWING_AGENTS_PROTOCOL Assisted-by: Claude Code:claude-opus-4-8 [ClaudeCode]
Collaborator
Author
|
Landed on main as the squash (mudler authorship, local commit-tree flow). 🤖 Generated with Claude Code |
mudler
added a commit
that referenced
this pull request
Aug 5, 2026
…closed (#44) row/SERVE-ASYNC-DENSE-MIRROR squash. ApplyDeviceTokenIdsOverride consumer in the shared EmbedInto (qwen3.cpp:170/:200) + DeviceTokenIdsScope in ForwardQwen3ForCausalLM (qwen3_dense.cpp:96), verbatim from the 27B-dense template. New gate test_qwen3_dense_async_serving (async vs in-process sync anchor): GREEN 41/41 default on 0.6B+4B, RED with mirror=0 reproduces token-0 garbage. SACRED 184/184 byte-neutral; memcheck 0. MXFP4 default-config e2e CLOSED: 3/4 token-exact + p2 near-tie-RATIFIED (teacher-forced 0.0000 nats). Gap enumeration recorded: InternLM2/ Mistral/Llama share the fixed consumer (scope one-liner residual); other registries tracked via decode-framework-routing-audit. W4 throughput bench = next (harness model-key plumbing). FOLLOWING_AGENTS_PROTOCOL Assisted-by: Claude Code:claude-opus-4-8 [ClaudeCode]
mudler
added a commit
that referenced
this pull request
Aug 6, 2026
…ecode gap (QUANT-CT-MXFP4-BENCH) Executed the binding ours-vs-oracle online-serving grid on Yi30/Qwen3-8B-MXFP4 (dense Qwen3, native Marlin W4A16 MXFP4 keep-quant) at 33e9360 on dgx GB10. Production graphed vLLM 0.25.0, oracle forced to Marlin via VLLM_DISABLED_KERNELS=FlashInferMxFp4LinearKernel (sm_121 cute-dsl crashes), c1/c2/c4/c8 x3 interleaved, single load/arm, drop_caches between legs, both flock locks. Smoke model gate reproduced #44 (3/3 deterministic token-exact + coherent near-tie). VERDICT: BELOW-FLOOR (gate NO, 74/84 axes below). Medians (ours->vllm, ratio): - total tok/s: c1 0.989, c2 0.911, c4 0.919, c8 0.913 - median TPOT ms: c1 0.986, c2 0.900, c4 0.905, c8 0.891 - median TTFT: at parity (c1/c4 PASS, c2/c8 within noise) - peak GPU mem: 28.3 vs 73.7 GiB = 2.607 (ours 2.6x LESS) Per-rep spread ~1-3% (tight/reproducible; full per-rep in benchmark-record). FIRST attribution (grounded, no ceiling): the gap is a BATCHED-decode cost absent at batch-1 -- c1 is at parity (0.989 tput / 0.986 TPOT), c2-c8 is ~0.91x driven ENTIRELY by TPOT +10-12% with TTFT/prefill at parity. Divergent hot path = the grouped Marlin W4A16 keep-quant decode GEMM at M=2..8, not batch-1 GEMV nor prefill. Memory a clean 2.6x win. Next: same-tool nsys decode-window on c8 both engines to name the exact kernel/shape. Also: fixed the orchestrator ordering (plan must precede corpus -- online_gate.py `plan` refuses a non-empty evidence root) and recorded the strict online-serving build contract learned this run (DISK build not tmpfs, RelWithDebInfo, oracle-venv ninja, oracle flashinfer cutlass, export-compile-commands, profile-control OFF). Records: BENCHMARKS/STATUS/FEATURES + quantization-matrix + benchmark-record (full table + recipe + build contract) + NOW + state, same change. Box left clean. FOLLOWING_AGENTS_PROTOCOL Assisted-by: Claude Code:claude-opus-4-8 [ClaudeCode]
mudler
added a commit
that referenced
this pull request
Aug 6, 2026
…-c8 TPOT 0.89-0.91x (#45) row/QUANT-CT-MXFP4-BENCH squash. Harness q3mxfp4 key (online_gate.py POINTS/REVISIONS/REPOSITORIES, smoke gate vs the #44 battery, dgx orchestrator; 45/45 contract tests). Binding medians of 3 reps ours vs the 0.25.0 oracle (Marlin W4A16 arm via VLLM_DISABLED_KERNELS): c1 0.989x total, c2 0.911x, c4 0.919x, c8 0.913x - deficit entirely TPOT (+10-12% at c2-c8), TTFT at parity, peak GPU memory 2.607x LESS (28.3 vs 73.7 GiB keep-quant). Attribution: the grouped Marlin W4A16 decode GEMM as M grows 2->8 (batch-1 + prefill at parity); kernel-level nsys + lever campaign dispatched, no ceiling declared. Strict record-execution build contract documented (real-disk tree, RelWithDebInfo, oracle cutlass, cache-drop mincore proof). FOLLOWING_AGENTS_PROTOCOL Assisted-by: Claude Code:claude-opus-4-8 [ClaudeCode]
mudler
added a commit
that referenced
this pull request
Aug 6, 2026
…te (#48) row/KERNEL-FA2-GQA-SWAP squash. LaunchDecodeVarlenFA2Bf16 gains the upstream mha_fwd_kvcache ngroups swap (logical q/o reshape via strides, no transpose; h=kv_heads, seqlen_q=ngroups; num_splits on swapped dims; 1:1 mirror of the shipped d256 LaunchDecodeFA2Bf16). Gate VT_FA2_DECODE_GQA_SWAP default OFF, OFF path untouched. Blast radius: Qwen3-dense d128 family only (0.6B/4B/8B); gate models already swap on d256; Voxtral MHA inert. Gates: op RED-first 5/5 (280 assn; wrong-stride injection fails loudly 26,528 violations), full binary 28/28 (454,679), memcheck 0/0, #44 smoke swap-ON 3/3 token-exact AND byte-identical to OFF (graphed). Flip conditioned on the c1-c8 re-bench + 0.6B/4B e2e (next campaign); #47 projects ~28% c2 / ~55% c8 gap closure. FOLLOWING_AGENTS_PROTOCOL Assisted-by: Claude Code:claude-opus-4-8 [ClaudeCode]
mudler
added a commit
that referenced
this pull request
Aug 6, 2026
…side noise (#49) row/KERNEL-FA2-GQA-SWAP-FLIP squash. Fa2DecodeGqaSwapEnabled OFF->ON (VT_FA2_DECODE_GQA_SWAP=0 opts out). Correctness: SACRED 0.6B+4B 16/16 each across FOUR arms (off/on/new-default/opt-out) character-identical; op 5/5-280; #44 smoke 3/3 token-exact at the new default. Mechanism proof at no-env: decode-flash grid (1,5,16)=batch x kv_heads, 80 CTAs 1 wave, per-call 63.7->45.3us (-29%). Binding vs #45: c2 0.911->0.922, c4 0.919->0.930, c8 0.913->0.942 (worst-ON rep > best-OFF rep each), c1 flat, TTFT parity+, memory 2.614x LESS. MXFP4 verdict still BELOW (best 0.942): named residual = grouped-Marlin E=1 +7-9%/call (indirect gather + fp32 C_tmp vs vLLM dense direct-A) + ~0.7ms/step host; next lever = dense-direct-A marlin decode. Ops note: clean checkout beats a file overlay for SACRED bindings (stale-binary trap recorded). FOLLOWING_AGENTS_PROTOCOL Assisted-by: Claude Code:claude-opus-4-8 [ClaudeCode]
mudler
added a commit
that referenced
this pull request
Aug 6, 2026
…ve EVERY tput axis vs #49, c1 crosses to parity+; goal still <1.0x on c2-c8 Clean-checkout binding grid at the committed sha d3b412f (not an overlay) via mxfp4-online-serving-grid.sh, oracle vLLM 0.25.0 (FlashInfer mxfp4 disabled), single-load/arm, drop_caches+mincore. #44 model gate re-passed inside the grid; 24/24 legs failed:0; reps tight (ours c8 CoV ~0.4%). Grid EXIT=1 is the gate-FAIL signal (gate_pass:false) + single-model cross-summary wait, not a crash. Binding (ours/vLLM normalized ratio) vs #49 tput [0.990/0.922/0.930/0.942]: total_token_throughput c1 1.005 c2 0.925 c4 0.939 c8 0.953 median_tpot_ms c1 1.002 c2 0.922 c4 0.915 c8 0.939 mean_ttft_ms c1 1.034 c2 0.962 c4 1.004 c8 0.999 median_itl_ms c1 1.004 c2 0.919 c4 0.920 c8 0.929 peak host-mem ours 35.2 GiB vs vLLM 76.7 GiB = 2.18x LESS VERDICT: c1 PASSES every axis; c2-c8 BELOW on tput/tpot/itl (best c8 0.953), TTFT parity c4/c8. gate_pass FALSE => MXFP4 parity goal NOT DONE (below-floor on c2-c8). The byte-exact slivers moved EVERY tput axis up vs #49 (c1 +1.5pp crosses to parity+, c4 +0.9pp, c8 +1.1pp, c2 +0.3pp), no regression. Residual map (c8 ~4.75% gap): (1) block=16 padding CLOSED (sliver a); (2) grouped-Marlin decode +7-9% per-call (E=1 indirect sorted_token_ids gather + fp32 C_tmp vs vLLM dense marlin_gemm direct-A, #46/#50); (3) host slice. Host-slice attribution (step 2, VT_LOOP_TRACE under decode): every window interval_ms ≈ step_ms (delta ≤0.02ms; admits=0 pure-decode windows interval-step ≈ 0), so the born-on-runner engine-core loop has negligible host overhead — the #47 ~0.7ms/step residual is NOT an engine-loop lever (it is the shared async frontend or the cross-tool attribution boundary). Records: STATUS, BENCHMARKS, benchmark-record, state, NOW. FOLLOWING_AGENTS_PROTOCOL Assisted-by: Claude Code:claude-opus-4-8 [ClaudeCode]
mudler
added a commit
that referenced
this pull request
Aug 6, 2026
… projections take vLLM's own dense marlin (48-CTA, byte-faithful, beats MoE) Executes the four scoped GB10 gates for the #55 dense-template marlin port and flips VT_MARLIN_DENSE default ON (=0 opts back out to the MoE route). WHY. The single-expert MoE-marlin route the dense E=1 projections use pads M<=8 into a 128-CTA grid; vLLM's own dense marlin covers the same tiles with 48 CTAs. The #54 par1 clamp reached 48 CTAs but regrouped the fp32 C_tmp reduce, costing a bf16 ULP that flipped a strict 32B token. This dense-template port keeps vLLM's OWN dense reduce, so it is byte-faithful to vLLM's numerics. GATES (GB10 sm_121a, RelWithDebInfo, full production stack): (a) UNIT EXEC: 2/2 cases 263/263 GREEN. First run was RED on NVFP4 but the diagnostic proved a test-authoring artifact, not a kernel bug: max|dense-moe| ==0 everywhere (dense output byte-identical to the validated MoE route), the failures were cancellation elements vs a naive fp32 ref (identical for MoE). Fix: NVFP4 vs-reference uses the cancellation-robust L2 metric; per-element byte-fidelity stays the EXACT dense==MoE check; RED injection preserved. compute-sanitizer memcheck: 0 invalid-access/race errors. (b) STRICT BATTERY dense-ON vs the oracle: 32B-NVFP4A16 6/6 (graphed AND eager, byte-identical) via the ratified near-tie razor — every dense token == vLLM's teacher-forced argmax, max gap 0.000 nats, TIGHTER than the MoE route (62 mnats). The MoE greedy anchor shifts at two exact bf16 ties, so the 32B our_ids/neartie_gap goldens are regenerated under dense-ON. Counters dense_gemms=2048, marlin_gemms=0 (gate-conditional assertion added). 0.6B/4B 184/184, async 82/82, #44 MXFP4-8B 3/3 token-exact. (c) NSYS c8: marlin kernel marlin_moe_wna16 -> marlin (dense template ran), grid 128->48 CTA, per-call median 117.8->85.8us. (d) BINDING c1..c8 x3 vs #51: total_token_throughput c1 1.020 / c2 0.962 / c4 0.966 / c8 0.969 (beats #51 1.005/0.925/0.939/0.953 on EVERY axis), GPU mem 2.63x less. MXFP4 vLLM-parity goal still <1.0x c2-c8 (closer than #51), residual = decode-flash then glue. Flip conditions (parity-enablers) met: correctness ratified + beats MoE every axis + no regression + memory win. Records: state, NOW, STATUS, BENCHMARKS, FEATURES updated; STATUS ratchet lowered to the shrunk size. FOLLOWING_AGENTS_PROTOCOL Assisted-by: Claude Code:claude-opus-4-8 [ClaudeCode]
mudler
added a commit
that referenced
this pull request
Aug 6, 2026
… closed (#60) row/QUANT-CT-MXFP4-FINAL-STACK squash. Lever 1: VT_FA2_NSPLITS_CAP (off/auto/N) at both FA2 decode launch sites — wave-optimal cap on the real 48 SMs; battery green incl. 32B strict CHARACTER-IDENTICAL at cap=auto, memcheck 0, async 325/325, #44 smoke 3/3. Stays default-OFF per parity-enablers: non-byte-exact split-reduction with no failing-axis win (only touches c1-c2; c1 already 1.020). Lever 2: glue already default-ON through vt::FusedChain at both add+RMSNorm sites + fused gate_up; residual is Inductor GEMM-epilogue fusion the catalog cannot express (a Marlin kernel rewrite, the #46-declined redirect). TERMINAL MXFP4 VERDICT: c1 1.020 PASS + mem 2.63x WIN; c2-c8 0.962-0.969 GPU-intrinsic; every measured lever exhausted. The one unexhausted path = from-scratch Marlin prologue/epilogue fusion kernel. FOLLOWING_AGENTS_PROTOCOL Assisted-by: Claude Code:claude-opus-4-8 [ClaudeCode]
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The #31 async race class fixed for classic dense Qwen3ForCausalLM (third confirmed instance): ApplyDeviceTokenIdsOverride consumer in the shared EmbedInto + DeviceTokenIdsScope in the dense forward, ported verbatim from the 27B-dense template. New async token-exact gate (async-vs-in-process-sync anchor, drift-proof): GREEN 41/41 on 0.6B+4B default, RED reproduces the token-0 garbage with the mirror off. SACRED 184/184 byte-neutral; memcheck 0. MXFP4 default-config e2e now 3/4 token-exact + p2 NEAR-TIE-RATIFIED (teacher-forced: our tokens are the oracle's own argmax throughout, 0.0000 nats). Residuals: W4 throughput bench (harness key plumbing, next dispatch); sibling scope one-liners for InternLM2/Mistral/Llama (share the fixed consumer, currently null-override no-regression).
🤖 Generated with Claude Code
https://claude.ai/code/session_014fZAcg1WcU8V629k6HWKys