row/SERVE-ASYNC-LLM: async batch-1 P0 root-caused + gated + fixed (device-mirror default ON) - #31
Merged
Merged
Conversation
…_DEVICE_MIRROR) — c16 NEUTRAL, uncovers async-serving decode bug the mirror fixes Followed up the prior c16 drain A/B's "real lever = GPU-resident sampled tokens". Built it as a drain MOVE (not the drain REMOVAL the earlier hazard note assumed): make last_sampled_tokens device-resident on the integrated path so update_states' host condense stops read-after-writing the device scatter, letting the bulk host prep overlap the GPU tail while the drain relocates from execute_model top to just before the forward. Hazard re-verification: hazard-A (double-buffer exec_state_) and hazard-C (block-table device buffer) are real ONLY for a drain removal; the move keeps exec_state_ and the decode-graph buffers guarded by the moved drain, so no double-buffering is needed. Gated VT_ASYNC_DEVICE_MIRROR, default OFF, OFF path byte-identical. Mirrors vLLM states.py:64 + gpu_model_runner.py:1786-1881. async_device_mirror() now engages on any real CUDA GPU (!UnifiedMemory() || is_integrated_gpu()); combine+scatter sites put the mirror branch first; execute_model defers the drain + exec_state_ reset to just before the forward on the mirror path. A/B (dgx GB10, same binary env-toggled, dual-lock, single load/arm, 3 reps c16 + 1 c32): c16 OFF median 2305.8 vs ON 2303.3 = 0.999x; c32 2928.9 vs 2919.1 = 0.997x; bands overlap. NEUTRAL — the move overlaps only host prep; the drain still serializes GPU input staging, so c16 does not recover (real fix needs drain-removal + double-buffer). Default stays OFF (no speed win → no flip per parity-enablers). Correctness finding from the token-exactness probe: the baseline async batch-1 greedy decode DEGENERATES into token-0 garbage nondeterministically, reproducing byte-identically on the unchanged production server @1ea26427; the device-resident mirror produces deterministic coherent output and fixes it. Never gated because SACRED exercises the SYNC engine. An async-serving token-exact gate is owed. Gates: CPU -Werror clean; runner 17/17 (incl. drain-invariant), input_batch 25/25, combine_tokens 7/7, llm_engine 11/11, engine_core_proc 10/10, async_llm 8/8. dgx CUDA build (Release/121a/cutlass-4.5.0/TRITON) clean, 1669 GDN cubin syms. SACRED test_qwen36_paged_engine mirror ON exit=0. UAF ignore_eos bracket mt 4-128 alive + compute-sanitizer memcheck 0 errors + 2/2 SACRED cases (mirror ON). FOLLOWING_AGENTS_PROTOCOL Assisted-by: Claude Code:claude-opus-4-8 [ClaudeCode]
…eration + RED-first async token gate ROW-SERVE-ASYNC-LLM P0: production async (AsyncLLM -> step_with_batch_queue, depth-2) batch-1 greedy decode nondeterministically degenerates into repeated token-0 garbage on the 35B NVFP4 model. Root-caused: on the async path sample_tokens_async DELETES the synchronous token_ids_cpu write-back, so the next step's prepare_inputs reads a stale/zero decode-row placeholder and relies on the device combine splicing the real token from last_sampled_tokens. On the VT_ASYNC_DEVICE_MIRROR=0 integrated path the combine patches step.input_token_ids on the MAIN QUEUE while the Qwen3.5 decode graph reads that same host vector on the CPU (BuildPaddedDecode -> CopyInPlace -> EmbedInto host->device upload) with NO intervening sync -- an unsynchronized device-write/host-read race. When the CPU wins, it embeds the zero placeholder -> token-0 degeneration. The sync LLMEngine (EngineCore::step, depth-1) writes token_ids_cpu synchronously so the combine is redundant and the race is harmless -- which is why SACRED (sync engine) never caught it and the bug shipped. FIX (parity-enablers policy): flip VT_ASYNC_DEVICE_MIRROR default ON. The mirror routes the combine's device output into the embed via ApplyDeviceTokenIdsOverride, main-queue-ordered after the combine, so the embed never does the racing host read -- exactly what upstream does on every platform (states.py:64 device-resident prev_sampled_token_ids + gpu_model_runner.py _prepare_input_ids GPU gather). The only OFF-path alternative is a per-step Synchronize after the combine, which reintroduces the host sync the async path exists to remove (a c16 regression) and is a GB10-only band-aid, not parity. Speed-neutral (prior c16 A/B 0.999x). VT_ASYNC_DEVICE_MIRROR=0 is the rollback. One functional line (AsyncDeviceMirrorEnvDefault: on unless "0"); the rest is truthful comment repair. GATE (RED-first, the missing async-serving token-exact gate that let this ship): tests/parity/test_qwen36_async_serving.cpp drives LoadedEngine::async_engine() (the depth-2 AsyncLLM path the sync SACRED gate never exercises), batch-1 (5 reps) + a concurrency bracket, and asserts token-exactness vs the SAME pinned oracle continuation the SACRED sync gate uses (greedy is deterministic regardless of step interleave). RED on VT_ASYNC_DEVICE_MIRROR=0 (P0 repro), GREEN on the new default. Checkpoint-gated + dgx-only (the bug needs the CUDA decode-graph + real GPU overlap); skips with a recorded reason on CPU/CI. CPU: compiles, links, skips. FOLLOWING_AGENTS_PROTOCOL Assisted-by: Claude Code:claude-opus-4-8 [ClaudeCode]
localai-bot
changed the title
mudler
added a commit
that referenced
this pull request
Aug 5, 2026
The #36 squash touched qwen3_5.cpp without moving docs/FEATURES.md in the same commit (the pre-merge doc-checkpoint ran on the merge-commit shape, where the diff looked records-only) and CI flagged the drift. The row also lagged reality: async scheduling has been default-ON with the UAF-safe drain since #31, and #36 added the opt-in VT_ASYNC_EXECUTOR slot ring. FOLLOWING_AGENTS_PROTOCOL Assisted-by: Claude Code:claude-opus-4-8 [ClaudeCode]
mudler
added a commit
that referenced
this pull request
Aug 5, 2026
…ken-exact), async-default is a pre-existing non-MXFP4 bug Root-caused the e2e residual. The default (async ON) degeneration is NOT the MXFP4 compute: with VT_ASYNC_SCHED=0 the same binary is TOKEN-EXACT vs the golden on 3/4 prompts (p1/p2/p4 exact; p3 open-ended story diverges after the identical first token = bf16/impl non-determinism). The async executor overlaps the prior step's output-copy with the forward, and classic dense Qwen3ForCausalLM lacks the async device-mirror fix (the #31 class wired only for the gate models) — a quant-independent, pre-existing classic-dense-Qwen3 async bug, separate from this row. MXFP4 keep-quant compute is proven correct: op-level GEMM (0.36%), scale permute byte-exact vs vLLM at all shapes, model-facing Apply/BuildMarlinDenseResident gate, and e2e async-off 3/4 token-exact. Reverted an exploratory per-stream workspace change (not the async cause). Evidence: docs/bench-evidence/mxfp4-qwen/W3-e2e-result.md. Residual: classic-dense-Qwen3 async device-mirror fix (separate row) + W4 bench. FOLLOWING_AGENTS_PROTOCOL Assisted-by: Claude Code:claude-opus-4-8 [ClaudeCode]
mudler
added a commit
that referenced
this pull request
Aug 5, 2026
…t, bench owed (#38) row/QUANT-CT-MXFP4 squash. Native compressed-tensors mxfp4-pack-quantized route through the existing Marlin FP4 infra (Laguna-B2 pattern): MXFP4 kernel-gen config (group_blocks=2, s_type kFE8M0fnu, 15 instances), MarlinProcessExpertScalesMxfp4 (byte-exact vs vLLM at all model shapes), launcher branch, Nvfp4Weight.{group_size,is_mxfp4}, MatmulMxfp4W4A16D, CT-MXFP4 loaders, qwen3 detect+load. Correctness PROVEN: op-level GEMM vs independent CPU dequant max_rel 0.36% at M=1/M=8 across K,N up to 12288; model-facing path bad=0; e2e 3/4 prompts token-exact vs the oracle golden (4th = open-ended, near-tie regime). W1 runtime finding: vLLM 0.25.0's default MXFP4 dispatch CRASHES on sm_121 (FlashInfer cute-dsl rejects cap 121); the working oracle path is Marlin W4A16 via VLLM_DISABLED_KERNELS. Default-config e2e is blocked by a PRE-EXISTING quant-independent async bug: classic dense Qwen3ForCausalLM lacks the #31 async device-mirror (VT_ASYNC_SCHED=0 token-exact, async-on degenerates) - separate row, fix in flight. W4 bench owed after that fix. FOLLOWING_AGENTS_PROTOCOL Assisted-by: Claude Code:claude-opus-4-8 [ClaudeCode]
mudler
added a commit
that referenced
this pull request
Aug 5, 2026
…closed (#44) row/SERVE-ASYNC-DENSE-MIRROR squash. ApplyDeviceTokenIdsOverride consumer in the shared EmbedInto (qwen3.cpp:170/:200) + DeviceTokenIdsScope in ForwardQwen3ForCausalLM (qwen3_dense.cpp:96), verbatim from the 27B-dense template. New gate test_qwen3_dense_async_serving (async vs in-process sync anchor): GREEN 41/41 default on 0.6B+4B, RED with mirror=0 reproduces token-0 garbage. SACRED 184/184 byte-neutral; memcheck 0. MXFP4 default-config e2e CLOSED: 3/4 token-exact + p2 near-tie-RATIFIED (teacher-forced 0.0000 nats). Gap enumeration recorded: InternLM2/ Mistral/Llama share the fixed consumer (scope one-liner residual); other registries tracked via decode-framework-routing-audit. W4 throughput bench = next (harness model-key plumbing). FOLLOWING_AGENTS_PROTOCOL Assisted-by: Claude Code:claude-opus-4-8 [ClaudeCode]
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What (2 commits)
e2dae57e—VT_ASYNC_DEVICE_MIRRORon integrated GB10: device-residentlast_sampled_tokens+ GPU gather, mirroring vLLMgpu_model_runner.py:1786-1881/states.py:64; drain moved fromexecute_modeltop to just before the forward.8a8bed99— the async batch-1 P0 root-caused, gated RED-first, and fixed by flipping the mirror default ON (VT_ASYNC_DEVICE_MIRROR=0= rollback).The P0 root cause (exact race)
sample_tokens_async(runner.cpp:2148-2154) skips the synctoken_ids_cpuwrite-back; the depth-2 loop relies on the device combine (runner.cpp:982-986, main-queue) to patchstep.input_token_ids— but the Qwen3.5 decode graph reads that same host vector on the CPU (BuildPaddedDecode→CopyInPlace→EmbedIntoupload, qwen3_5.cpp:7634/7641/5923) with no intervening sync. When the CPU wins the race it embeds the zero placeholder → nondeterministic token-0 degeneration. SACRED never saw it becauseLLMEngine::stepis depth-1 sync, where the write-back makes the combine redundant.The missing gate (now exists)
tests/parity/test_qwen36_async_serving.cpp: AsyncLLM depth-2, batch-1 x5 + 4-way concurrency, token-exact vs the same pinned oracle continuation as the sync SACRED gate. RED on the old default (P0 reproduced: token-0 garbage in first 16 tokens), GREEN on the new default. Checkpoint-gated, dgx-only, skips with reason on CPU.Why default-ON and not a host-path band-aid
The minimal OFF-path fix is a per-step
Synchronizeafter the combine — it reintroduces the host sync the async path exists to remove (c16 regression) and is a GB10-only hack. The mirror is the vLLM-parity structure and is speed-neutral: c16 3-rep median 2303.9 ON vs 2290.0 rollback (baseline ~2305), c32 2942.7.Gates (all on GB10, final default)
SACRED sync token-exact exit=0 · async gate GREEN 39/39 (also under compute-sanitizer, 0 errors) · ignore_eos UAF bracket mt 4-128 all 200+alive · CPU -Werror clean, 5/5 suites · GDN cubins verified. Records updated same-commits (STATUS in-ratchet, BENCHMARKS, benchmark-record, NOW, state).
Residuals (recorded)
exec_state_/block-table (the mirror is a speed-neutral drain MOVE).Evidence:
dgx:~/work/mirror-ab/.🤖 Generated with Claude Code
https://claude.ai/code/session_014fZAcg1WcU8V629k6HWKys