From f9c969aebee729cdc5fe470bb2162db6deb69438 Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Wed, 5 Aug 2026 22:29:33 +0000 Subject: [PATCH 1/2] fix(qwen3-dense): port the #31 async device token-ids mirror to classic dense (ROW-SERVE-ASYNC-DENSE-MIRROR) Classic dense Qwen3ForCausalLM (qwen3.cpp) lacked the async device-token-ids mirror the gate models (qwen3_5) already had, so on the depth-2 AsyncLLM serving path its batch-1 greedy decode nondeterministically degenerated into token-0 garbage (surfaced by the MXFP4 campaign; quant-independent, hits bf16/NVFP4). The shared pure-dense embed EmbedInto uploaded the stale host token_ids, racing the async combine's device input-ids write (unsynchronized device-write/host-read) -> token-0 degeneration when the host read won. Note the classic-dense decode graph is default-OFF, so the racing path is the EAGER ForwardBody->EmbedInto. Fix (byte-identical when the mirror is off), mirroring the 27B-dense template (qwen3_5.cpp:6737/6746): - qwen3.cpp: EmbedInto now consumes ApplyDeviceTokenIdsOverride (main-queue-ordered d.b.Copy over the DBuf prefix), replacing the racing host read with the combine's device ids. Shares the detail::DeviceTokenIds seam (qwen3_5_internal.h). - qwen3_dense.cpp: ForwardQwen3ForCausalLM establishes DeviceTokenIdsScope over the whole forward (eager + decode-graph replay), publishing the override. Gate (RED-first): tests/parity/test_qwen3_dense_async_serving.cpp (Qwen3-0.6B/4B, batch-1 x5 + N=4 concurrency) requires every async continuation to reproduce the race-free in-process SYNC engine continuation token-for-token (the near-tie-proof, drift-proof anchor). RED on VT_ASYNC_DEVICE_MIRROR=0, GREEN on the default; checkpoint-gated + dgx-only. CPU gates green (Release -Werror): library + test build clean; regression suites input_batch/combine_tokens/runner/engine_core_proc/async_llm/llm_engine pass. Residual (named): sibling registries sharing this driver (InternLM2, Mistral, Llama) get the fixed consumer but still need the one-line DeviceTokenIdsScope; own-embed decode models per decode-framework-routing-audit. dgx-owed: async gate RED->GREEN + SACRED dense gates + the MXFP4 W4 online_gate bench + p3 near-tie. FOLLOWING_AGENTS_PROTOCOL Assisted-by: Claude Code:claude-opus-4-8 [ClaudeCode] --- .agents/NOW.md | 1 + .agents/benchmark-record.md | 29 +++ .agents/model-matrix.md | 2 +- .agents/state.md | 70 ++++++ docs/BENCHMARKS.md | 4 +- docs/FEATURES.md | 2 +- docs/STATUS.md | 2 +- src/vllm/model_executor/models/qwen3.cpp | 41 ++++ .../model_executor/models/qwen3_dense.cpp | 10 + tests/CMakeLists.txt | 11 + .../parity/test_qwen3_dense_async_serving.cpp | 218 ++++++++++++++++++ 11 files changed, 385 insertions(+), 5 deletions(-) create mode 100644 tests/parity/test_qwen3_dense_async_serving.cpp diff --git a/.agents/NOW.md b/.agents/NOW.md index 9de0aa583..2a5277e5a 100644 --- a/.agents/NOW.md +++ b/.agents/NOW.md @@ -22,6 +22,7 @@ checkpoint on `upstream/main` at `59674cf1d`. | Kimi-Linear-48B (KDA+NoPE-MLA+MoE) | **Full-model GB10 e2e RUNS** (bf16-resident §13, f32-loader block CLEARED): CPU+CUDA 13/13·656; host RSS peak 1.7 GiB, min-avail 21 GiB, no OOM. **Token gate NEAR-TIE 106/128** (6/8 prompts token-exact; numerics near-tie vs deterministic oracle, not a bug) | STRICT path = device GDN/MLA islands + bf16 stream (W7-speed residuals); 1.59 tok/s; default OFF | | 35B fresh grid | **BOUND** @`1ea26427`: tput 0.93-1.03x, c16 0.93x. INTAKE + Option A both **RESOLVED NEGATIVE** (H2D-out-of-capture tput WASH) | Real lever left: prefill glue (task #61) | | Qwen3.5-4B revalidation | 0.9971x @`59674cf1` (#35); TTFT/PSS pass, TPOT/ITL open | `docs/bench-evidence/` | +| ROW-SERVE-ASYNC-DENSE-MIRROR | **Code+gate LANDED** (CPU -Werror + suites green): classic dense `Qwen3ForCausalLM` consumes the async device token-ids mirror (#31 ported); RED-first `test_qwen3_dense_async_serving` | dgx owed: gate RED→GREEN + SACRED dense + **MXFP4 W4 bench** (c1..c8x3 vs oracle) + p3 near-tie. Residual: InternLM2/Mistral/Llama scope line | In-flight branches (gated default-OFF, not pushed): `laguna-fp4proj-prod` (fp4 opt-in), laguna bf16/legacy/pipeline-gemv, `ds4-hc-expand-fuse`. diff --git a/.agents/benchmark-record.md b/.agents/benchmark-record.md index d37b70b60..54fa7ec36 100644 --- a/.agents/benchmark-record.md +++ b/.agents/benchmark-record.md @@ -12210,3 +12210,32 @@ a NAMED speed residual, NOT an optimized decode. DEFAULT: `VT_KIMI_DEVICE_COMPUT (near-tie != token-exact). PATH TO STRICT + speed = the named W7 residuals: device GDN per-channel recurrence + exp/softplus gate op + paged `mla::ForwardMlaAttentionBlock` + a bf16 residual stream end-to-end (matches vLLM's rounding, removes the host round-trips). + +## 2026-08-07T05:00 — ROW-SERVE-ASYNC-DENSE-MIRROR: classic-dense async P0 fix (code+CPU gates) + W4 MXFP4 bench PLAN + +CORRECTNESS (not a speed lever): the #31 async device token-ids mirror, ported to the +classic dense family. Classic dense `Qwen3ForCausalLM` (qwen3.cpp `EmbedInto`) raced +the async combine's device input-ids write against a stale host upload → nondeterministic +token-0 degeneration on the depth-2 AsyncLLM serving path (surfaced by the MXFP4 +campaign; quant-independent). Fix: EmbedInto consumes `ApplyDeviceTokenIdsOverride` +(main-queue-ordered `d.b.Copy` over the DBuf prefix) published by +`ForwardQwen3ForCausalLM`'s `DeviceTokenIdsScope` — verbatim the 27B-dense template. +Byte-identical when the mirror is off. Gate `test_qwen3_dense_async_serving` (Qwen3-0.6B/4B, +async==in-process-SYNC anchor; RED on VT_ASYNC_DEVICE_MIRROR=0, GREEN default). CPU +-Werror clean; regression suites GREEN (input_batch 183 / combine_tokens 14 / runner 323 / +engine_core_proc 576 / async_llm 309); test_llm_engine = documented flaky (unrelated TU). + +OWED ON DGX (this row): +- Async gate RED→GREEN (same binary, VT_ASYNC_DEVICE_MIRROR env-toggled) + SACRED + test_qwen3_paged_engine (0.6B/4B) unchanged + ignore_eos bracket + compute-sanitizer + memcheck on the new gate. +- THE MXFP4 W4 BENCH (now unblocked): `tools/bench/online_gate.py` c1..c8 x3 reps, single + load per arm, OURS (fixed default config) vs ORACLE on Yi30/Qwen3-8B-MXFP4. Oracle arm + MUST set `VLLM_DISABLED_KERNELS=FlashInferMxFp4LinearKernel` (its default MXFP4 dispatch + crashes on sm_121). Match-or-beat is the bar; honest either way. Update the BENCHMARKS + MXFP4 Qwen3-8B row + quantization-matrix with the numbers. +- p3 (open-ended story) near-tie distributional verdict against the oracle K-run set while + the oracle is loaded (converts the QUANT-CT-MXFP4 "3/4 token-exact, p3 diverges after + identical first token" into a ratified near-tie). +Box safety: both locks (flock $HOME/gpu.lock AND /tmp/gpu), free -g >= 90, no oracle +alongside our server, local-ai-worker stopped, tmux + done-markers, git archive not rsync. diff --git a/.agents/model-matrix.md b/.agents/model-matrix.md index 6f7ef95c8..aaec429ab 100644 --- a/.agents/model-matrix.md +++ b/.agents/model-matrix.md @@ -54,7 +54,7 @@ Engaged architectures (the 43 non-`INVENTORIED` rows): | Support | Architecture | Family / example | Status | Row | |---|---|---|---|---| -| ✅ | `Qwen3ForCausalLM` | Qwen3 dense (0.6B/1.7B/4B/32B) | near-tie-robust token-exact 16/16 on 0.6B+4B vs vLLM 0.25.0; NVFP4A16 (W4A16) dense quant also gated; c1 every-axis speed parity, c8 decode residual | `MODEL-TEXT-qwen3-qwen3-for-causal-lm` | +| ✅ | `Qwen3ForCausalLM` | Qwen3 dense (0.6B/1.7B/4B/32B) | near-tie-robust token-exact 16/16 on 0.6B+4B vs vLLM 0.25.0; NVFP4A16 (W4A16) dense quant also gated; c1 every-axis speed parity, c8 decode residual; async-serving device token-ids mirror ported (`ROW-SERVE-ASYNC-DENSE-MIRROR`, #31 fix into the shared dense `EmbedInto`) — `test_qwen3_dense_async_serving` RED→GREEN; sibling InternLM2/Mistral/Llama scope one-liner is a named residual | `MODEL-TEXT-qwen3-qwen3-for-causal-lm` | | ✅ | `Qwen3MoeForCausalLM` | Qwen3-Coder-30B-A3B (MoE) | STRICT token-exact 6/6 vs vLLM 0.25.0; 11/16 speed-grid cells at/above graphed vLLM, c1/c2 residual | `MODEL-TEXT-qwen3-moe-qwen3-moe-for-causal-lm` | | ✅ | `Qwen3_5ForConditionalGeneration` | Qwen3.6-27B (text path) | text-gen STRICT token-exact 235/235 vs vLLM 0.25.0; mm INPUT pipeline (M0/M1) landed + processor-parity gate PASS; **M3-W0 landed** (vision-inclusive checkpoint `Qwen/Qwen3.6-27B` 51.7 GiB bf16 with 333 `visual.*` FOUND+fits+downloaded; 27B vision config resolved — depth 27/out 5120/**EMPTY deepstack**; MRoPE `[11,11,10]`/rot 64/theta 1e7; the bf16 GDN-hybrid loader ALREADY handles it). **M3-b LANDED 2026-07-25: image→text STRICT token-exact 32/32 vs vLLM 0.25.0** — Qwen3.6-27B image understanding works end-to-end (forked GDN-hybrid VL forward gated on mm input ⇒ text byte-identical; 27B/35B/Coder inertness re-passed 235/315/138). **M3d LANDED 2026-07-25: video→text STRICT token-exact 32/32 vs vLLM 0.25.0** — video works end-to-end too (`Qwen3_5VLGenerateGreedyVideo` reuses the M3c processor/windowed-tower/video-MRoPE on the GDN-hybrid backbone). **Qwen video modalities COMPLETE: image+video both work e2e** (audio N/A for Qwen). **VISION-FORWARD SPEED (2026-07-28, `CLAIM-MM-SPEED-QWEN-IMAGE`, multimodal-speed.md §16): the mm-forward tower BEATS vLLM** — per-image tower forward 142.3 ms (flash `AttentionDenseFlash`, hd-72) vs vLLM 0.25.0 ~250 ms eager encode = 0.57×; attribution-first nsys REFUTED a bigger lever (the t=784 vision attention is serial-latency-bound, flash only 1.04× over warp), STRICT 32/32 image/video HELD + goldens md5 unchanged. Row stays `PARTIAL` — vision-forward speed BEATS vLLM; **umbrella speed pending** on batched c2+/serving | `MODEL-MM-qwen3-5-qwen3-5-for-conditional-generation` | | ✅ | `Qwen3_5MoeForConditionalGeneration` | Qwen3.6-35B-A3B (text path) | text-gen STRICT token-exact 315/315 vs vLLM 0.25.0; mm INPUT pipeline (M0/M1) landed + processor-parity gate PASS, vision tower pending (M2/M3) so the row is `PARTIAL` (text-only) | `MODEL-MM-qwen3-5-qwen3-5-moe-for-conditional-generation` | diff --git a/.agents/state.md b/.agents/state.md index 942736107..87358fca5 100644 --- a/.agents/state.md +++ b/.agents/state.md @@ -36559,3 +36559,73 @@ decode). `VT_KIMI_DEVICE_COMPUTE` STAYS OFF (parity-enablers: a near-tie is not `ACTIVE`. Residuals now precisely: (a) STRICT token-exactness (device islands + bf16 stream), (b) the paged het-KV incremental decode, (c) speed. + +## ROW-SERVE-ASYNC-DENSE-MIRROR: ported the #31 async device-mirror to the classic dense family (Qwen3ForCausalLM); RED-first async-serving gate; W4 MXFP4 bench in flight + + +Closed the classic-dense half of the #31 P0 (the residual named by QUANT-CT-MXFP4 +2026-08-07T03:00): classic dense `Qwen3ForCausalLM` (qwen3.cpp) lacked the async +device-token-ids mirror the gate models (qwen3_5) already had, so on the depth-2 +AsyncLLM serving path its batch-1 greedy decode nondeterministically degenerated into +token-0 garbage. Quant-independent (surfaced by the MXFP4 campaign; hits bf16/NVFP4 +classic dense too). + +MECHANISM (file:line, identical to the 27B-dense template qwen3_5.cpp:6737/6746). +On async serving the sampled token is NOT written to token_ids_cpu synchronously; the +runner's device combine splices each decode row's real token into the DEVICE input-ids +on the main queue while host `token_ids` stays stale (runner.cpp:1159 publishes the +pointer as `ModelForwardInput::device_token_ids`, non-null only on the async CUDA +path). The SHARED pure-dense embed `EmbedInto` (qwen3.cpp) uploaded the stale host +vector (`DBuf dids(d, kI32, {T}, token_ids.data())`) — racing the combine's device +write (unsynchronized device-write/host-read) → token-0 degeneration when the host +read wins. Note the classic-dense decode graph is default-OFF, so the racing path is +the EAGER `ForwardBody`→`EmbedInto`, not only the decode graph. + +FIX (two edits, byte-identical when the mirror is off). +(1) qwen3.cpp: added `ApplyDeviceTokenIdsOverride(Dev,DBuf&,int64_t)` (verbatim analog +of qwen3_5.cpp:5904) and called it in `EmbedInto` right after the host-`token_ids` +DBuf construction — overwrites the real prefix with the device combine's ids via a +main-queue-ordered `d.b.Copy`, so the embed never does the racing host read. The +`detail::DeviceTokenIds` seam (qwen3_5_internal.h:175-193, thread_local defined in +qwen3_5.cpp:344) is shared across TUs; qwen3.cpp now includes that header. +(2) qwen3_dense.cpp: `ForwardQwen3ForCausalLM` establishes +`detail::DeviceTokenIdsScope(input.device_token_ids, token_ids.size())` at the top +(mirrors qwen3_5_dense.cpp:118), publishing the override for the whole forward (eager +Forward/ForwardDevice AND the decode-graph replay, all of which route through the +shared EmbedInto). Consume-on-first-use; null everywhere except the async CUDA runner. + +GATE (RED-first). New `tests/parity/test_qwen3_dense_async_serving.cpp` (registered in +tests/CMakeLists.txt), the classic-dense sibling of test_qwen36_async_serving. Drives +`LoadedEngine::async_engine()` (depth-2 AsyncLLM, batch-1 x5 reps + N=4 concurrency) +and requires every async continuation to reproduce the IN-PROCESS SYNC engine +continuation token-for-token. The sync engine is the race-free, drift-proof, +near-tie-proof anchor (these bf16 dense checkpoints are near-tie, so a committed strict +golden is ill-posed; async and sync run the IDENTICAL per-step device forward → equal +token-for-token with the fix). Qwen3-0.6B (primary vehicle) + Qwen3-4B (bigger +confirmation), checkpoint-gated + dgx-only. POLARITY: VT_ASYNC_DEVICE_MIRROR=0 => RED +(EmbedInto races), default => GREEN (async==sync). + +CPU GATES (mudler-ubuntu-box, Release -Werror clean): full library + new test build +0 errors; test compiles/links/skips (checkpoints absent). Regression suites GREEN: +test_input_batch 183, test_combine_tokens 14, test_runner 323, test_engine_core_proc +576, test_async_llm 309. test_llm_engine is the documented flaky suite (starves >120s +on a loaded box; my diff touches only qwen3.cpp/qwen3_dense.cpp model TUs it does not +exercise on CPU) — cross-check on dgx. + +RESIDUALS (named, NOT fixed this pass). The 3 sibling registries sharing this driver — +InternLM2, Mistral, Llama (all "== Qwen3DenseModel", route through the now-fixed +EmbedInto) — get the fixed CONSUMER for free but still lack the one-line +`DeviceTokenIdsScope` PRODUCER, so they stay byte-identical to today (override null on +their path) and remain exposed on the async CUDA path; each needs exactly the +qwen3_dense.cpp scope line + the qwen3_5_internal.h include. Own-embed decode models +(deepseek_v2/v4, gemma2/3/4, glm4*, granite, kimi*, laguna, minicpm*, olmo2, opt, +phi*, qwen3_moe, qwen3_vl, stablelm, commandr) need the same two-part seam (consumer + +scope); track via [[decode-framework-routing-audit]] (qwen3_vl already flagged +off-framework). + +OWED ON DGX (this row): (1) the async gate RED→GREEN + SACRED dense gates +(test_qwen3_paged_engine 0.6B/4B) + ignore_eos bracket + memcheck; (2) THE MXFP4 W4 +bench — tools/bench/online_gate.py c1..c8 x3, single load/arm, ours vs oracle on +Yi30/Qwen3-8B-MXFP4, oracle arm VLLM_DISABLED_KERNELS=FlashInferMxFp4LinearKernel; +(3) the p3 near-tie distributional verdict (oracle K-run set) while the oracle is +loaded. Branch `row/SERVE-ASYNC-DENSE-MIRROR`. diff --git a/docs/BENCHMARKS.md b/docs/BENCHMARKS.md index 78e92c44d..8dd9a3354 100644 --- a/docs/BENCHMARKS.md +++ b/docs/BENCHMARKS.md @@ -102,7 +102,7 @@ and coherent. The missing gate now exists, `test_qwen36_async_serving` (depth-2 AsyncLLM, batch-1 + concurrency, token-exact vs the SACRED oracle): RED on `=0`, GREEN on the default. c16 re-checked on the default: 2312.9/2303.9/2294.4 (median **2303.9**), c32 2942.7 (no regression). Root cause + file:line in the benchmark -record. The intake-drain lever likewise measured NEUTRAL (2026-08-06, `VT_INTAKE_DRAIN` A/B 3+3 reps): admitting during the forward wait collapses intake -91% but shifts it into queued, arrival-to-scheduled invariant, so the recorded INTAKE term is an attribution boundary over a GPU-bound prefill wait, not reducible; lever reverted, byte-exact `VT_LOOP_TRACE` probe kept. +record. The same P0 hit classic dense `Qwen3ForCausalLM` (quant-independent), fixed by `ROW-SERVE-ASYNC-DENSE-MIRROR` (see the MXFP4 Qwen3-8B row). The intake-drain lever likewise measured NEUTRAL (2026-08-06, `VT_INTAKE_DRAIN` A/B 3+3 reps): admitting during the forward wait collapses intake -91% but shifts it into queued, arrival-to-scheduled invariant, so the recorded INTAKE term is an attribution boundary over a GPU-bound prefill wait, not reducible; lever reverted, byte-exact `VT_LOOP_TRACE` probe kept. ### DeepSeek-V2-Lite (MLA) @@ -307,7 +307,7 @@ built on it rather than keeping the flattering one. | Qwen3-dense decode CUDA-graph | Token-exact pass, ~4.3% e2e directional | Steady-state per-step tok/s | | Kimi-Linear-48B-A3B (KDA+MLA+MoE) | Full-model GB10 e2e RUNS (bf16-resident §13), NEAR-TIE 106/128, pool math CLOSES; default OFF | Full model RUNS on GB10 (bf16-resident, RSS peak 1.7 GiB, min-avail 21 GiB, no OOM). Token NEAR-TIE 106/128 (6/8 prompts exact, numerics vs deterministic oracle). 1.59 tok/s. Detail: spec §13 | | vLLM 0.26 re-benchmark | Pending | Re-run the binding grids on the advanced pin | -| MXFP4 Qwen3-8B (W4A16 Marlin) | Compute proven (#38); e2e 3/4 token-exact async-off | W4 bench after the classic-dense async-mirror fix; oracle arm needs `VLLM_DISABLED_KERNELS=FlashInferMxFp4LinearKernel` | +| MXFP4 Qwen3-8B (W4A16 Marlin) | Compute proven (#38); e2e now token-exact async-DEFAULT after the classic-dense async-mirror fix (`ROW-SERVE-ASYNC-DENSE-MIRROR`) | W4 online_gate c1..c8 x3 vs oracle (`VLLM_DISABLED_KERNELS=FlashInferMxFp4LinearKernel`), running on dgx; p3 near-tie verdict owed | | SGLang floor arms | Never ran | Both arms of the SGLang comparison | | cuBLAS invocation-parity guard | CI guard landed (CPU); `kGemvHeuristicAlgos` refactor build-verify owed | `nvcc` rebuild + SACRED gate on dgx | diff --git a/docs/FEATURES.md b/docs/FEATURES.md index 70b206240..3cc0dea93 100644 --- a/docs/FEATURES.md +++ b/docs/FEATURES.md @@ -38,7 +38,7 @@ are our reading of their documented behavior, not measurements. | Priority scheduling | ◐ gating | ✅ | ✅ | ☐ | | LPM cache-aware admission | ✅ | ☐ | ✅ | ☐ | | In-batch prefix de-prioritization | ✅ | ☐ | ✅ | ☐ | -| Async / overlap scheduling | ✅ default on (UAF-safe drain; opt-in `VT_ASYNC_EXECUTOR` out-of-capture H2D staging) | ✅ | ✅ | ☐ | +| Async / overlap scheduling | ✅ default on (UAF-safe drain; device token-ids mirror on gate + classic-dense models; opt-in `VT_ASYNC_EXECUTOR` out-of-capture H2D staging) | ✅ | ✅ | ☐ | | CUDA graph decode capture | ◐ per-family | ✅ | ✅ | ✅ | | Partial-prefill concurrency | ☐ | ✅ | ✅ | ☐ | | Cascade attention | ☐ | ✅ | ◐ | ☐ | diff --git a/docs/STATUS.md b/docs/STATUS.md index 49811d565..402ca7031 100644 --- a/docs/STATUS.md +++ b/docs/STATUS.md @@ -53,7 +53,7 @@ token-for-token correctness against the pinned oracle. |---|---|---| | Qwen3.6-27B (NVFP4) text generation | Correctness-complete, at/above vLLM speed | Token-exact greedy on GB10; beats vLLM 0.25.0 total throughput at every concurrency (1.007-1.045x), effective parity 115/124 axes | | Qwen3.6-35B-A3B (NVFP4, GDN MoE) | Correctness-complete; 3-rep grid 0.93-1.03x. Async batch-1 token-0 degeneration FIXED: `VT_ASYNC_DEVICE_MIRROR` default ON | Token-exact SYNC+ASYNC (RED→GREEN); c16 0.93x; `VT_ASYNC_EXECUTOR` Option A (H2D out of capture) GREEN+RED but A/B NEUTRAL → OFF; c16 residual is prefill glue | -| Qwen3 / Qwen2 dense (BF16) | Correctness-complete, speed-pending | Near-tie-robust token-exact vs vLLM (Qwen3-0.6B, Qwen3-4B); c1 effective parity, c8 decode residual. **D1 (2026-07-31, `CLAIM-D1-BF16-MERGED-QKV`): the bf16 merged-QKV path (`Qwen3QkvMergeEnabled`/`VT_QWEN3_QKV_MERGE`) is now default-ON** — one `vt::MatmulBT` over the merged `[qdim+2kdim,H]` owner + a contiguous `vt::QkvSplit` (OLMo-2 exemplar), replacing three per-shard GEMMs. Bit-exact GEMM math (A/B unit `test_ops_qkv_merge` byte-identical, RED-first); the wider-N cuBLASLt K-reduction flips the 0.6B genuine bf16 near-tie so the SACRED 0.6B golden was regenerated (all tokens within the near-tie band, max 0.125 nats), while Qwen3-4B is byte-neutral (0 diffs, stays STRICT). Re-gated 0.6B 16/16 + 4B 16/16; consistency/launch-count fold (measured NEUTRAL on 4B decode), no new throughput owed | +| Qwen3 / Qwen2 dense (BF16) | Correctness-complete, speed-pending. Async-serving P0 FIXED (`ROW-SERVE-ASYNC-DENSE-MIRROR`): classic-dense `Qwen3ForCausalLM` now honors the async device token-ids mirror | Near-tie-robust token-exact vs vLLM (Qwen3-0.6B, Qwen3-4B); c1 effective parity, c8 decode residual. **Async device-mirror (`ROW-SERVE-ASYNC-DENSE-MIRROR`): the #31 fix ported to the classic dense family.** The shared dense `EmbedInto` (qwen3.cpp) raced the async combine's device input-ids write against a stale host upload → nondeterministic token-0 degeneration on the depth-2 AsyncLLM serving path (proven by the MXFP4 campaign; quant-independent, hits bf16/NVFP4). Now `EmbedInto` consumes the device override (`ApplyDeviceTokenIdsOverride`) published by `ForwardQwen3ForCausalLM`'s `DeviceTokenIdsScope`, exactly mirroring the 27B-dense template. New gate `test_qwen3_dense_async_serving` (Qwen3-0.6B/4B, batch-1 + concurrency, token-exact vs the race-free in-process SYNC anchor): RED on `VT_ASYNC_DEVICE_MIRROR=0`, GREEN on the default. Byte-identical when the mirror is off. RESIDUAL: the sibling registries sharing this driver (InternLM2, Mistral, Llama) get the fixed consumer but still need the one-line scope; own-embed models per `decode-framework-routing-audit`. **D1 (2026-07-31, `CLAIM-D1-BF16-MERGED-QKV`): the bf16 merged-QKV path (`Qwen3QkvMergeEnabled`/`VT_QWEN3_QKV_MERGE`) is now default-ON** — one `vt::MatmulBT` over the merged `[qdim+2kdim,H]` owner + a contiguous `vt::QkvSplit` (OLMo-2 exemplar), replacing three per-shard GEMMs. Bit-exact GEMM math (A/B unit `test_ops_qkv_merge` byte-identical, RED-first); the wider-N cuBLASLt K-reduction flips the 0.6B genuine bf16 near-tie so the SACRED 0.6B golden was regenerated (all tokens within the near-tie band, max 0.125 nats), while Qwen3-4B is byte-neutral (0 diffs, stays STRICT). Re-gated 0.6B 16/16 + 4B 16/16; consistency/launch-count fold (measured NEUTRAL on 4B decode), no new throughput owed | | Qwen3.5-4B plain BF16 direct loading on discrete CUDA | Correctness-complete, speed-pending | Revalidated after merging current upstream: local throughput is unchanged at 0.99997x its prior run; against the freshly measured pinned oracle it is 0.9971x. TTFT 0.7719x and host PSS 0.3127x pass; TPOT/ITL 1.1244x and VRAM 1.0014x remain open. Direct ON/OFF outputs remain 128/128 identical | | Qwen3-Coder-30B-A3B MoE (BF16) | Correctness-complete, speed-pending | Near-tie-robust token-exact 6/6; 11 of 16 binding grid cells at or above vLLM. **D1 (2026-07-31): inherits the default-ON bf16 merged-QKV via the shared dense `AttnBlock` — byte-neutral (0 token diffs, golden UNCHANGED); re-gated 6/6** | | Llama-3.x dense (BF16) | Correctness-complete, speed-pending | Near-tie-robust token-exact 16/16 (Llama-3.2-1B); llama3 RoPE scaling | diff --git a/src/vllm/model_executor/models/qwen3.cpp b/src/vllm/model_executor/models/qwen3.cpp index 8ab6f8ba0..845b7e60b 100644 --- a/src/vllm/model_executor/models/qwen3.cpp +++ b/src/vllm/model_executor/models/qwen3.cpp @@ -51,6 +51,7 @@ #include "vllm/model_executor/models/dense_nvfp4_gemm.h" // NVFP4 W4A16 dispatch #include "vllm/model_executor/models/device_pool.h" // DevicePool/Pool/ActivePool (shared) #include "vllm/model_executor/models/qwen3_5_common.h" // HostLogits +#include "vllm/model_executor/models/qwen3_5_internal.h" // detail::DeviceTokenIds seam #include "vllm/platforms/interface.h" #include "vt/backend.h" #include "vt/ops.h" @@ -145,6 +146,40 @@ void GatherRows(Dev d, void* dst, const Tensor& src, const std::vector& d.b.Copy(d.q, dp + s * rb, sp + static_cast(idx[s]) * rb, rb); } +// ROW-SERVE-ASYNC-DENSE-MIRROR (ENG-ASYNC-SCHED W4 / the #31 P0, ported to the +// classic dense family): overwrite the REAL prefix of a freshly uploaded input-id +// buffer with the device-resident ids the async runner's combine produced. The +// exact analogue of qwen3_5.cpp's ApplyDeviceTokenIdsOverride — that TU wired it +// for the gate models (MoE + 27B dense); this is the identical consumer for the +// SHARED pure-dense driver (Qwen3ForCausalLM and every registry that routes +// through Qwen3DenseModel / EmbedInto: InternLM2, Mistral, Llama). +// +// WHY: on the async serving loop (AsyncLLM depth-2) the sampled token is NOT +// written to token_ids_cpu synchronously; the runner's device combine splices each +// decode row's real token into the device input-ids on the MAIN QUEUE while the +// host `token_ids` vector stays stale. The default host upload below then RACES +// that device write (unsynchronized device-write/host-read), nondeterministically +// embedding the stale/zero placeholder -> token-0 degeneration. Copying the +// device ids over the DBuf prefix here is main-queue-ordered AFTER the combine, so +// the embed never does the racing host read — exactly upstream (states.py:64 +// device-resident prev_sampled_token_ids + gpu_model_runner.py GPU gather). +// +// The override is published by the registry forward's detail::DeviceTokenIdsScope +// and CONSUMED here on first use; null on every path except the CUDA async runner, +// so with no override this is byte-identical to the pre-fix host upload. +static void ApplyDeviceTokenIdsOverride(Dev d, DBuf& dids, int64_t T) { + const detail::DeviceTokenIds ov = detail::DeviceTokenIdsOverride(); + if (ov.ids == nullptr) return; + detail::DeviceTokenIdsOverride() = detail::DeviceTokenIds{}; + // A device buffer LONGER than the embed's input would run past the end. That can + // only mean the runner and the model disagree about this step's shape, so fail + // loudly rather than corrupt the embedding. + VT_CHECK(ov.count <= T, + "qwen3 dense embed: device input ids longer than the embed input"); + d.b.Copy(d.q, dids.ptr(), ov.ids, + static_cast(ov.count) * sizeof(int32_t)); +} + // Embed: hidden[T,H] bf16 = embed_tokens[token_ids] (device-resident table). KEPT // OUTSIDE THE CUDA-GRAPH (mirrors qwen3_moe.cpp / qwen3_5.cpp EmbedInto): the CUDA // Embedding op allocates a device bounds-check flag (cudaMalloc/cudaFree) and syncs @@ -156,7 +191,13 @@ void EmbedInto(Dev d, DBuf& hidden, const std::vector& token_ids, const int64_t T = static_cast(token_ids.size()); Tensor dtab = ResidentWeight(d, weights.embed_tokens, {config.vocab_size, config.hidden_size}); + // ROW-SERVE-ASYNC-DENSE-MIRROR: when the async runner has already placed this + // step's input ids on the device (and spliced each decode row's sampled token + // into them there), embed straight from that buffer. `token_ids` is stale for + // decode rows in that case BY DESIGN — materializing it on the host is the + // synchronize the async path removes — so its real prefix is overwritten here. DBuf dids(d, DType::kI32, {T}, token_ids.data()); + ApplyDeviceTokenIdsOverride(d, dids, T); vt::Embedding(d.q, hidden.t(), dtab, dids.t()); } diff --git a/src/vllm/model_executor/models/qwen3_dense.cpp b/src/vllm/model_executor/models/qwen3_dense.cpp index 3af0bb03e..74bbced94 100644 --- a/src/vllm/model_executor/models/qwen3_dense.cpp +++ b/src/vllm/model_executor/models/qwen3_dense.cpp @@ -22,6 +22,7 @@ #include "vllm/model_executor/models/qwen3.h" #include "vllm/model_executor/models/qwen3_5.h" // ForwardLogits (shared carrier) #include "vllm/model_executor/models/qwen3_5_common.h" // HostLogits +#include "vllm/model_executor/models/qwen3_5_internal.h" // detail::DeviceTokenIdsScope #include "vllm/v1/kv_cache_dtype.h" #include "vllm/v1/kv_cache_interface.h" #include "vt/dtype.h" @@ -85,6 +86,15 @@ ForwardLogits ForwardQwen3ForCausalLM(LoadedModel& model, const ModelForwardInput& input) { auto& qwen = static_cast(model); const Qwen3DenseWeights& weights = qwen.weights(); + // ROW-SERVE-ASYNC-DENSE-MIRROR: publish the async runner's device-resident input + // ids for the duration of THIS forward, so the shared EmbedInto (eager + // Forward/ForwardDevice AND the decode-graph replay, all of which route through + // qwen3.cpp) reads them instead of racing the stale host `token_ids` against the + // combine's device write (the #31 async batch-1 token-0 degeneration, ported to + // the classic dense family). Null on every non-async-CUDA path, RAII-scoped so it + // cannot outlive the call — byte-identical when the mirror is off. + const detail::DeviceTokenIdsScope device_ids_scope( + input.device_token_ids, static_cast(input.token_ids.size())); // Shared pure-dense decode CUDA-graph (opt-in via VLLM_CPP_QWEN3_DENSE_DECODE_ // GRAPH): route a graph-eligible pure-decode CUDA step through the model's driver // (pad-to-nearest capture set + replay), else fall through to the byte-identical diff --git a/tests/CMakeLists.txt b/tests/CMakeLists.txt index 1dcd688a8..7cd77e0e0 100644 --- a/tests/CMakeLists.txt +++ b/tests/CMakeLists.txt @@ -949,6 +949,17 @@ target_compile_definitions(test_qwen3_paged_engine PRIVATE target_include_directories(test_qwen3_paged_engine PRIVATE ${CMAKE_SOURCE_DIR}/tests/parity ${CMAKE_SOURCE_DIR}/src) +# ROW-SERVE-ASYNC-DENSE-MIRROR — the classic-dense async-serving greedy gate: +# Qwen3-0.6B/4B async batch-1 + concurrency reproduce the race-free SYNC engine +# continuation token-for-token. RED on VT_ASYNC_DEVICE_MIRROR=0 (dense EmbedInto +# races the combine), GREEN on the default fix (checkpoint-gated, dgx-only; skips +# on CPU/CI when the snapshot is absent). +vllm_cpp_add_test(test_qwen3_dense_async_serving parity/test_qwen3_dense_async_serving.cpp) +target_compile_definitions(test_qwen3_dense_async_serving PRIVATE + PARITY_GOLDENS_DIR="${CMAKE_SOURCE_DIR}/tests/parity/goldens") +target_include_directories(test_qwen3_dense_async_serving PRIVATE + ${CMAKE_SOURCE_DIR}/tests/parity ${CMAKE_SOURCE_DIR}/src) + # ROAD-V1-D4-APC W3 — the cache-ON automatic-prefix-caching e2e gate on Qwen3-4B # (dense, full-attention, APC-eligible): APC-ON == APC-OFF token-exact (RED-first # cache-bug catch) + non-zero hit counters + vLLM K-run membership + prefill/TTFT diff --git a/tests/parity/test_qwen3_dense_async_serving.cpp b/tests/parity/test_qwen3_dense_async_serving.cpp new file mode 100644 index 000000000..c81248047 --- /dev/null +++ b/tests/parity/test_qwen3_dense_async_serving.cpp @@ -0,0 +1,218 @@ +// vllm.cpp original (checkpoint-gated acceptance gate); no upstream mirror. The +// CLASSIC-DENSE sibling of test_qwen36_async_serving.cpp (which owns the gate +// models). Mirrors its scheduling intent — greedy served decode is +// token-deterministic regardless of async step interleave — for the shared +// pure-dense driver (Qwen3ForCausalLM and every registry that routes through +// Qwen3DenseModel / qwen3.cpp EmbedInto). +// +// THE ASYNC-SERVING CLASSIC-DENSE GREEDY GATE — the missing counterpart to +// test_qwen3_paged_engine.cpp (the SYNC SACRED gate). That gate drives the SYNC +// LLMEngine (depth-1); this one drives the ASYNC SERVING frontend +// (LoadedEngine::async_engine() -> AsyncLLM -> step_with_batch_queue, depth-2 / +// max_concurrent_batches==2), the path the production OpenAI server runs and the +// ONLY path that pipelines two steps — so it is the only path that exposes the +// ROW-SERVE-ASYNC-DENSE-MIRROR P0: classic-dense Qwen3 async batch-1 greedy decode +// nondeterministically degenerating into repeated token-0 garbage. +// +// ROOT CAUSE (the #31 class, ported). On the async path the sampled token is NOT +// written to token_ids_cpu synchronously (sample_tokens_async); the runner's device +// combine splices each decode row's real token into the DEVICE input-ids on the +// main queue while the host `token_ids` vector stays stale. Before this row, +// classic dense Qwen3's EmbedInto (qwen3.cpp) IGNORED the device ids and uploaded +// the stale host vector — racing the combine's device write (unsynchronized +// device-write/host-read). When the host read wins it embeds the zero placeholder +// -> token-0 degeneration. The gate models (qwen3_5) already consumed the device +// override via ApplyDeviceTokenIdsOverride; this row wires the identical consumer +// into the shared dense EmbedInto + publishes the override from the classic-dense +// registry forward (ForwardQwen3ForCausalLM's DeviceTokenIdsScope). +// +// WHY THE ORACLE IS THE IN-PROCESS SYNC ENGINE, not a committed strict golden. The +// classic dense checkpoints (Qwen3-0.6B/4B, bf16) are NEAR-TIE models — vLLM's own +// prefill argmax disagrees with its incremental decode token at genuine ties (see +// test_qwen3_paged_engine.cpp), so a STRICT token-exact bar against a committed +// vLLM golden is ill-posed here. But the async serving path and the sync engine run +// the IDENTICAL per-step DEVICE forward on the SAME runner: with the fix the async +// embed reads exactly the ids the sync path appends to token_ids_cpu, so async == +// sync token-for-token (CUDA argmax is deterministic given identical inputs). The +// sync LLMEngine is race-free (it writes token_ids_cpu synchronously, so its combine +// is redundant — the SACRED gate proves it correct), making its continuation the +// exact, drift-proof, near-tie-proof anchor the async output must reproduce. The P0 +// degeneration is repeated token-0 garbage, far outside any near-tie band, so a +// divergence is unambiguous. +// +// GATE POLARITY (greedy is deterministic; async interleave must not change tokens): +// VT_ASYNC_DEVICE_MIRROR=0 (rollback: dense EmbedInto races) => RED (P0 repro), +// default / VT_ASYNC_DEVICE_MIRROR=1 (the fix) => GREEN (async==sync). +// +// Checkpoint-GATED + dgx-only, exactly like the SACRED gate: it resolves the real +// Qwen3-0.6B / Qwen3-4B snapshots under ~/.cache/huggingface/hub. On the CPU dev +// box / CI the snapshots are absent, so each case emits a loud SKIP and returns +// (compiles + links on CPU, but only RUNS on dgx.casa GB10, where the CUDA forward +// + real GPU overlap the bug needs exist). +#include + +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include "vllm/entrypoints/model_loader.h" +#include "vllm/sampling_params.h" +#include "vllm/v1/engine/async_llm.h" + +namespace fs = std::filesystem; + +namespace { + +// Greedy (argmax) sampling params — temperature 0 => deterministic. Identical to +// the SACRED gate's Greedy(). +vllm::SamplingParams Greedy(int max_tokens) { + vllm::SamplingParams sp; + sp.temperature = 0.0; + sp.max_tokens = max_tokens; + sp.PostInit(); + return sp; +} + +// Resolve the newest HF snapshot dir for a "models--" cache entry, or "". +// IDENTICAL resolution to test_qwen3_paged_engine.cpp's FindSnapshot. +std::string FindSnapshot(const std::string& repo_dir) { + const char* home = std::getenv("HOME"); + if (home == nullptr) return ""; + const fs::path snaps = + fs::path(home) / ".cache/huggingface/hub" / repo_dir / "snapshots"; + std::error_code ec; + if (!fs::is_directory(snaps, ec)) return ""; + for (const auto& e : fs::directory_iterator(snaps, ec)) { + if (fs::exists(e.path() / "config.json", ec)) return e.path().string(); + } + return ""; +} + +// The classic-dense async-serving gate. Loads the checkpoint ONCE, computes the +// race-free SYNC continuation as the anchor, then requires every async batch-1 rep +// and every concurrent request to reproduce it token-for-token. +void RunAsyncGate(const std::string& repo_dir, const std::string& label) { + const std::string snap = FindSnapshot(repo_dir); + if (snap.empty()) { + MESSAGE(label << " checkpoint absent; skipping (dgx-only) — " << repo_dir + << " snapshot not present. The P0 needs the CUDA forward + real GPU " + "overlap; a CPU-only run cannot reproduce it (the eager backend " + "serializes the queue), so this trivially passes."); + return; + } + + // Prompt long enough that the served decode runs many steps (the race is per + // decode step). A short greedy continuation exposes any degeneration immediately. + const std::string kPrompt = "The capital of France is Paris, and the"; + constexpr int kMaxTokens = 24; + + MESSAGE(label << ": loading via FromModelDir(" << snap << ")..."); + std::unique_ptr loaded = + vllm::entrypoints::LoadedEngine::FromModelDir( + snap, vllm::entrypoints::EngineParams{}); + + // The gate is only meaningful when the depth-2 async serving path is engaged — + // the path that pipelines two steps and exposes the P0. If async scheduling + // resolved OFF (VT_ASYNC_RUNNER=0 / VT_ASYNC_SCHED=0) the frontend runs depth-1 + // and the race cannot occur, so a pass would be vacuous. Fail loud instead. + REQUIRE(loaded->async_scheduling_enabled()); + REQUIRE(loaded->max_concurrent_batches() == 2); + + // ── THE ANCHOR: the race-free SYNC engine continuation ──────────────────────── + // Driven to completion BEFORE the async frontend threads start, so the shared + // scheduler/executor is idle when async takes over. The sync LLMEngine writes + // token_ids_cpu synchronously (its combine is redundant), so this is the correct, + // drift-proof, near-tie-proof reference the async output must match. + const vllm::RequestOutput sync_out = + loaded->engine().generate(kPrompt, Greedy(kMaxTokens), "sync-anchor"); + REQUIRE(sync_out.finished); + REQUIRE(sync_out.outputs.size() == 1); + const std::vector want = sync_out.outputs[0].token_ids; + REQUIRE(static_cast(want.size()) == kMaxTokens); + MESSAGE(label << ": sync anchor continuation=\"" << sync_out.outputs[0].text + << "\" (" << want.size() << " tokens)"); + + vllm::v1::AsyncLLM& aengine = loaded->async_engine(); + + // ── ARM 1: BATCH-1 (single-request) served greedy decode ────────────────────── + // THE P0. Each independent single-request generation MUST reproduce the sync + // anchor token-for-token. Several reps because the bug is nondeterministic (the + // host<->combine race is a coin flip per decode step); on the buggy (mirror-OFF) + // path the first generated token degenerates and stays there, so any one rep + // diverging fails the gate. On the fixed (mirror-ON) path every rep is byte-exact. + constexpr int kBatch1Reps = 5; + for (int r = 0; r < kBatch1Reps; ++r) { + const std::string id = "b1-r" + std::to_string(r); + const vllm::RequestOutput out = + aengine.generate(kPrompt, Greedy(kMaxTokens), id); + REQUIRE(out.finished); + REQUIRE(out.outputs.size() == 1); + const std::vector& got = out.outputs[0].token_ids; + MESSAGE(label << "[batch1 rep " << r << "]: produced " << got.size() << "/" + << kMaxTokens << " tokens; continuation=\"" << out.outputs[0].text + << "\""); + REQUIRE(static_cast(got.size()) == kMaxTokens); + CHECK(got == want); + } + + // ── ARM 2: CONCURRENCY bracket ──────────────────────────────────────────────── + // N independent greedy requests submitted together so the engine runs them as + // pure-decode batched steps (num_reqs==N) while the depth-2 loop pipelines. Each + // request is independent (own KV state), so each MUST reproduce the same sync + // anchor regardless of the step interleave (vLLM's greedy determinism guarantee). + int kN = 4; + if (const char* c = std::getenv("VT_ASYNC_SERVING_CONC")) { + const int v = std::atoi(c); + if (v > 0) kN = v; + } + MESSAGE(label << ": concurrency bracket N=" << kN); + std::vector reqs; + reqs.reserve(static_cast(kN)); + for (int i = 0; i < kN; ++i) { + reqs.push_back(aengine.add_request("c" + std::to_string(i), kPrompt, + Greedy(kMaxTokens))); + } + // Drain each request to its TERMINAL output. Blocking on request i does NOT + // serialize the engine: all kN were enqueued first and the engine thread steps + // them concurrently, so requests j!=i keep decoding while we collect i. + for (int i = 0; i < kN; ++i) { + const vllm::v1::AsyncRequest& req = reqs[static_cast(i)]; + vllm::RequestOutput out; + for (;;) { + std::optional ready = aengine.get_output_nowait(req); + out = ready.has_value() ? std::move(*ready) : aengine.get_output(req); + if (out.finished) break; + } + REQUIRE(out.finished); + REQUIRE(out.outputs.size() == 1); + const std::vector& got = out.outputs[0].token_ids; + MESSAGE(label << "[conc " << i << "]: produced " << got.size() << "/" + << kMaxTokens << " tokens; continuation=\"" << out.outputs[0].text + << "\""); + REQUIRE(static_cast(got.size()) == kMaxTokens); + CHECK(got == want); + } + + aengine.shutdown(); +} + +} // namespace + +// Qwen3-0.6B (dense) — the primary classic-dense P0 vehicle (smallest/fastest). +TEST_CASE("qwen3-0.6B dense async-serving greedy token-exact gate (dgx-only) — " + "ROW-SERVE-ASYNC-DENSE-MIRROR") { + RunAsyncGate("models--Qwen--Qwen3-0.6B", "qwen3-0.6B"); +} + +// Qwen3-4B (dense) — the bigger-model confirmation (36 layers, GQA 32/8), same +// shared driver, same async serving path. +TEST_CASE("qwen3-4B dense async-serving greedy token-exact gate (dgx-only) — " + "ROW-SERVE-ASYNC-DENSE-MIRROR") { + RunAsyncGate("models--Qwen--Qwen3-4B", "qwen3-4B"); +} From bf89affb9490f07eee4c15cf096f9c93322dace4 Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Wed, 5 Aug 2026 22:57:20 +0000 Subject: [PATCH 2/2] =?UTF-8?q?docs(serve-async-dense):=20dgx=20GB10=20ver?= =?UTF-8?q?ification=20=E2=80=94=20async=20RED=E2=86=92GREEN=20+=20SACRED?= =?UTF-8?q?=20+=20memcheck=20+=20MXFP4=20default=20e2e=20+=20near-tie=20ra?= =?UTF-8?q?tified?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Records the on-hardware verification of ROW-SERVE-ASYNC-DENSE-MIRROR (f9c969ae) on dgx.casa GB10 (sm_121a, CUDA build in /dev/shm; both flock locks per run, tmux + done-markers, free-g 98-100 GiB): - Async gate RED->GREEN, same binary env-toggled: 0.6B + 4B GREEN 41/41 (async == in-process SYNC anchor); RED (VT_ASYNC_DEVICE_MIRROR=0) FAIL 3 CHECKs — concurrency requests degenerate into "...the Germany is!!!!!!" token-0 garbage. - SACRED test_qwen3_paged_engine 0.6B + 4B 184/184, 16/16 prompts each, 0 forward-divergent (byte-neutral sync path confirmed). - compute-sanitizer memcheck on the 0.6B async GREEN arm: 0 errors. - MXFP4 Yi30/Qwen3-8B-MXFP4 (classic dense Qwen3ForCausalLM) DEFAULT-config (async ON) e2e now coherent + 3/4 token-exact vs the golden; degenerate without the fix — closes the QUANT-CT-MXFP4 async-default residual. - p2/p3 story near-tie RATIFIED: oracle (VLLM_DISABLED_KERNELS=FlashInferMxFp4Linear Kernel) re-reproduces the golden; teacher-forcing it on our sequence makes our token the oracle's own argmax at every position (max gap 0.0000 nats). Owed residual (recorded, not a regression): the W4 THROUGHPUT bench (online_gate.py c1..c8x3 vs oracle) — the harness has no Yi30/8B model key (needs corpus + oracle-record plumbing). The fix unblocks the default-config number; oracle proven to run the model today. Plus the sibling scope one-liner (InternLM2/Mistral/Llama). FOLLOWING_AGENTS_PROTOCOL Assisted-by: Claude Code:claude-opus-4-8 [ClaudeCode] --- .agents/NOW.md | 2 +- .agents/benchmark-record.md | 26 ++++++++++++++++++ .agents/quantization-matrix.md | 2 +- .agents/state.md | 49 ++++++++++++++++++++++++++++++++++ docs/BENCHMARKS.md | 2 +- docs/FEATURES.md | 2 +- docs/STATUS.md | 2 +- 7 files changed, 80 insertions(+), 5 deletions(-) diff --git a/.agents/NOW.md b/.agents/NOW.md index 2a5277e5a..9499b5223 100644 --- a/.agents/NOW.md +++ b/.agents/NOW.md @@ -22,7 +22,7 @@ checkpoint on `upstream/main` at `59674cf1d`. | Kimi-Linear-48B (KDA+NoPE-MLA+MoE) | **Full-model GB10 e2e RUNS** (bf16-resident §13, f32-loader block CLEARED): CPU+CUDA 13/13·656; host RSS peak 1.7 GiB, min-avail 21 GiB, no OOM. **Token gate NEAR-TIE 106/128** (6/8 prompts token-exact; numerics near-tie vs deterministic oracle, not a bug) | STRICT path = device GDN/MLA islands + bf16 stream (W7-speed residuals); 1.59 tok/s; default OFF | | 35B fresh grid | **BOUND** @`1ea26427`: tput 0.93-1.03x, c16 0.93x. INTAKE + Option A both **RESOLVED NEGATIVE** (H2D-out-of-capture tput WASH) | Real lever left: prefill glue (task #61) | | Qwen3.5-4B revalidation | 0.9971x @`59674cf1` (#35); TTFT/PSS pass, TPOT/ITL open | `docs/bench-evidence/` | -| ROW-SERVE-ASYNC-DENSE-MIRROR | **Code+gate LANDED** (CPU -Werror + suites green): classic dense `Qwen3ForCausalLM` consumes the async device token-ids mirror (#31 ported); RED-first `test_qwen3_dense_async_serving` | dgx owed: gate RED→GREEN + SACRED dense + **MXFP4 W4 bench** (c1..c8x3 vs oracle) + p3 near-tie. Residual: InternLM2/Mistral/Llama scope line | +| ROW-SERVE-ASYNC-DENSE-MIRROR | **LANDED + dgx-VERIFIED** (`f9c969ae`): #31 async mirror ported to classic dense `Qwen3ForCausalLM`. Async gate RED→GREEN 0.6B+4B, SACRED 184/184, memcheck 0, MXFP4 default e2e 3/4 + near-tie RATIFIED | Residual: **W4 throughput bench** (online_gate lacks Yi30/8B key); sibling scope one-liner | In-flight branches (gated default-OFF, not pushed): `laguna-fp4proj-prod` (fp4 opt-in), laguna bf16/legacy/pipeline-gemv, `ds4-hc-expand-fuse`. diff --git a/.agents/benchmark-record.md b/.agents/benchmark-record.md index 54fa7ec36..aabee97e7 100644 --- a/.agents/benchmark-record.md +++ b/.agents/benchmark-record.md @@ -12239,3 +12239,29 @@ OWED ON DGX (this row): identical first token" into a ratified near-tie). Box safety: both locks (flock $HOME/gpu.lock AND /tmp/gpu), free -g >= 90, no oracle alongside our server, local-ai-worker stopped, tmux + done-markers, git archive not rsync. + +## 2026-08-07T07:00 — ROW-SERVE-ASYNC-DENSE-MIRROR dgx GB10 verification (f9c969ae) + +CUDA build (/dev/shm, 121a, cutlass 4.5.0, nvcc 13.0, Release). Both flock locks per run, +tmux + done-markers, free-g 98-100 GiB. CORRECTNESS (this row is a correctness fix, not a +speed lever): +- ASYNC GATE (same binary, env-toggled): 0.6B + 4B GREEN (default) 41/41 async==sync; + 0.6B + 4B RED (VT_ASYNC_DEVICE_MIRROR=0) FAIL 3 CHECKs — concurrency requests degenerate + into "...the Germany is!!!!!!" token-0 garbage. P0 reproduced + fixed on real HW. +- SACRED test_qwen3_paged_engine 0.6B+4B 184/184, 16/16 prompts each, 0 forward-divergent + (byte-neutral sync path). +- compute-sanitizer memcheck on 0.6B async GREEN: 0 errors. +- MXFP4 Yi30/Qwen3-8B-MXFP4 DEFAULT-config (async ON + fix) vllm-cli greedy 48 tok: p0/p1/p3 + TOKEN-EXACT vs golden, p2 story coherent near-tie; VT_ASYNC_DEVICE_MIRROR=0 same binary + DEGENERATES (" Paris. What I I I What... !!!!!!") = the W3 async-default degeneration, CLOSED. +- p3/p2 NEAR-TIE RATIFIED: oracle (VLLM_DISABLED_KERNELS=FlashInferMxFp4LinearKernel, ninja+nvcc + on PATH) re-reproduces the golden; story greedy K=8 singleton; teacher-forcing the oracle on + OUR sequence => our token is the oracle argmax at EVERY position (max gap 0.0000 nats). The + free-run " wise old man" vs our " young girl" is the oracle's own prefill-vs-decode tie. + +W4 THROUGHPUT BENCH NOT DONE (harness gap, honest): online_gate.py MODEL_REVISIONS carries +only "27"/"35"; no Yi30/8B key. Retargeting = MODEL_REVISIONS/REPOSITORIES entry + corpus +(make_serve_low_corpus) + record-oracle (needs an 8B paged_engine test binary) + num-blocks/ +max-num-seqs sizing. The fix unblocks the DEFAULT-config bench (no more VT_ASYNC_SCHED=0 +workaround); oracle proven to run the model today. Evidence: dgx:/dev/shm/serve-async-dense/ +{gates_06b.log,gates2.log,mxfp4_e2e.log,oracle_neartie2.log,oracle_tf.log}. diff --git a/.agents/quantization-matrix.md b/.agents/quantization-matrix.md index 17bee582c..abb359661 100644 --- a/.agents/quantization-matrix.md +++ b/.agents/quantization-matrix.md @@ -127,7 +127,7 @@ Registry source: | `QUANT-FP8-PCPT` | ModelOpt FP8 per-channel/per-token | W8/A8 | capability selected | - | - | - | - | - | `INVENTORIED` | - | leaf spec open | - | | `QUANT-MXFP8-MODELOPT` | ModelOpt MXFP8 | W8/A8 | CUDA/ROCm/XPU dispatch | - | - | - | - | - | `INVENTORIED` | - | leaf spec open | - | | `QUANT-MIXED-MODELOPT` | ModelOpt mixed precision | FP8/NVFP4/MXFP8 groups | per-layer | part | part | part | Y | Y | `PARTIAL` | 35B FP8+NVFP4 slice: [loader](../src/vllm/model_executor/models/qwen3_5_weights.cpp#L118), [FP8 tests](../tests/vt/test_ops_fp8_cutlass.cpp#L188), [NVFP4 tests](../tests/vt/test_ops_moe_grouped.cpp#L453), [gate](../tests/parity/test_qwen36_paged_engine.cpp#L78) | leaf spec open | - | -| `QUANT-CT-MXFP4` | compressed-tensors MXFP4 `mxfp4-pack-quantized` (group 32, E8M0 block scales, NO global) | W4/A16 native **Marlin mxf4 keep-quant** landed (GB10 target; W4A4 cute-dsl crashes sm_121) | CUDA GB10 Marlin W4A16 (E8M0, group_blocks=2); CPU dequant fallback | Y | Y | Y | Y | - | `ANCHOR-BACKFILL` | **W2 native compute + W3 gates + e2e (row/QUANT-CT-MXFP4, `1c5ee09e`):** W0 vehicle `Yi30/Qwen3-8B-MXFP4` runs on the 0.25.0 oracle; W1 traced FlashInfer-W4A4-selected-but-crashes-on-sm_121 -> **Marlin W4A16 is the GB10 target**. Native path: `generate_kernels.py` MXFP4 config (`kFE8M0fnu`, group_blocks 2) + regenerated instances; [`MarlinProcessExpertScalesMxfp4`](../src/vt/cuda/cuda_marlin_repack.cu) (byte-exact vs vLLM at all shapes); `MoeMarlinArgs.{group_size,mxfp4}` launcher branch; `Nvfp4Weight.{group_size,is_mxfp4}` + [`dense_nvfp4_gemm.h`](../include/vllm/model_executor/models/dense_nvfp4_gemm.h) branch + `MatmulMxfp4W4A16D`; [`dense_weight_loaders.h`](../include/vllm/model_executor/models/dense_weight_loaders.h) MXFP4 loaders; `qwen3_weights.cpp` detect+load. **Gates GREEN:** op-level GEMM vs independent CPU dequant 0.36% M=1/M=8 all real shapes ([`test_ops_moe_grouped.cpp`](../tests/vt/test_ops_moe_grouped.cpp)); model-facing `MakeLinearMethod->Apply->BuildMarlinDenseResident` bad=0 K=4096+12288 ([`test_linear_method.cpp`](../tests/vllm/model_executor/layers/test_linear_method.cpp)); **e2e 3/4 token-exact vs oracle golden (async-off)** ([evidence](../docs/bench-evidence/mxfp4-qwen/W3-e2e-result.md)). RESIDUAL: default-async degeneration is a PRE-EXISTING classic-dense-Qwen3 async bug (device-mirror not wired for `qwen3.cpp`, quant-independent, SEPARATE row); W4 bench owed; p3 formal distributional gate owed. Earlier CPU weight unpack + E8M0 dequant: NEW [mxfp4_dequant.h](../include/vllm/model_executor/model_loader/mxfp4_dequant.h) + [.cpp](../src/vllm/model_executor/model_loader/mxfp4_dequant.cpp#L14) (`E8M0ToF32` = `2^(byte-127)`, `DequantMxfp4ToBf16`/`ToF32`, group 32, no global; reuses `kE2M1Lut`). Unit gate [test_mxfp4_dequant.cpp](../tests/vllm/test_mxfp4_dequant.cpp#L34) — E8M0 known-byte decode, hand-computed 32-group dequant (bf16+f32), the E8M0-vs-fp8 + group-32-vs-16 RED traps, multi-row/group offsets, randomized rel-error vs a double-precision port of `dq_mxfp4_torch` with bf16==f32 exactness. CPU `-Werror` 0-warn. Ports FROM `compressed_tensors_w4a4_mxfp4.py:20-97` + `mxfp8_utils.py:61-65,222` + golden `tests/quantization/reference_mxfp4.py:28-117`. C/E/P PENDING: GPU W4A4 fp4 GEMM + Marlin W4A16 fallback + MoE expert path + e2e are NAMED later bricks; DeepSeek-V4 + Kimi-K3 loaders consume this once wired | [MXFP4 spike](specs/mxfp4-compressed-tensors.md) | `CLAIM-QUANT-MXFP4` | +| `QUANT-CT-MXFP4` | compressed-tensors MXFP4 `mxfp4-pack-quantized` (group 32, E8M0 block scales, NO global) | W4/A16 native **Marlin mxf4 keep-quant** landed (GB10 target; W4A4 cute-dsl crashes sm_121) | CUDA GB10 Marlin W4A16 (E8M0, group_blocks=2); CPU dequant fallback | Y | Y | Y | Y | - | `ANCHOR-BACKFILL` | **W2 native compute + W3 gates + e2e (row/QUANT-CT-MXFP4, `1c5ee09e`):** W0 vehicle `Yi30/Qwen3-8B-MXFP4` runs on the 0.25.0 oracle; W1 traced FlashInfer-W4A4-selected-but-crashes-on-sm_121 -> **Marlin W4A16 is the GB10 target**. Native path: `generate_kernels.py` MXFP4 config (`kFE8M0fnu`, group_blocks 2) + regenerated instances; [`MarlinProcessExpertScalesMxfp4`](../src/vt/cuda/cuda_marlin_repack.cu) (byte-exact vs vLLM at all shapes); `MoeMarlinArgs.{group_size,mxfp4}` launcher branch; `Nvfp4Weight.{group_size,is_mxfp4}` + [`dense_nvfp4_gemm.h`](../include/vllm/model_executor/models/dense_nvfp4_gemm.h) branch + `MatmulMxfp4W4A16D`; [`dense_weight_loaders.h`](../include/vllm/model_executor/models/dense_weight_loaders.h) MXFP4 loaders; `qwen3_weights.cpp` detect+load. **Gates GREEN:** op-level GEMM vs independent CPU dequant 0.36% M=1/M=8 all real shapes ([`test_ops_moe_grouped.cpp`](../tests/vt/test_ops_moe_grouped.cpp)); model-facing `MakeLinearMethod->Apply->BuildMarlinDenseResident` bad=0 K=4096+12288 ([`test_linear_method.cpp`](../tests/vllm/model_executor/layers/test_linear_method.cpp)); **e2e 3/4 token-exact vs oracle golden (async-off)** ([evidence](../docs/bench-evidence/mxfp4-qwen/W3-e2e-result.md)). **Default-async degeneration RESOLVED (`ROW-SERVE-ASYNC-DENSE-MIRROR`, `f9c969ae`):** the pre-existing classic-dense-Qwen3 async bug (device-mirror not wired for `qwen3.cpp`) is fixed; DEFAULT-config (async ON) e2e on dgx is now coherent + 3/4 token-exact vs the golden (p0/p1/p3 exact; p2 story = oracle-ratified near-tie, teacher-forced max gap 0.0000 nats). p3 formal near-tie gate now RATIFIED. RESIDUAL: the W4 THROUGHPUT bench (c1..c8x3 vs oracle) still owed — online_gate lacks a Yi30/8B model key (needs corpus + oracle-record plumbing). Earlier CPU weight unpack + E8M0 dequant: NEW [mxfp4_dequant.h](../include/vllm/model_executor/model_loader/mxfp4_dequant.h) + [.cpp](../src/vllm/model_executor/model_loader/mxfp4_dequant.cpp#L14) (`E8M0ToF32` = `2^(byte-127)`, `DequantMxfp4ToBf16`/`ToF32`, group 32, no global; reuses `kE2M1Lut`). Unit gate [test_mxfp4_dequant.cpp](../tests/vllm/test_mxfp4_dequant.cpp#L34) — E8M0 known-byte decode, hand-computed 32-group dequant (bf16+f32), the E8M0-vs-fp8 + group-32-vs-16 RED traps, multi-row/group offsets, randomized rel-error vs a double-precision port of `dq_mxfp4_torch` with bf16==f32 exactness. CPU `-Werror` 0-warn. Ports FROM `compressed_tensors_w4a4_mxfp4.py:20-97` + `mxfp8_utils.py:61-65,222` + golden `tests/quantization/reference_mxfp4.py:28-117`. C/E/P PENDING: GPU W4A4 fp4 GEMM + Marlin W4A16 fallback + MoE expert path + e2e are NAMED later bricks; DeepSeek-V4 + Kimi-K3 loaders consume this once wired | [MXFP4 spike](specs/mxfp4-compressed-tensors.md) | `CLAIM-QUANT-MXFP4` | | `QUANT-CT-W4A8-FP8` | compressed-tensors W4A8 FP8 | W4/A8 | CUTLASS | - | - | - | - | - | `INVENTORIED` | - | leaf spec open | - | | `QUANT-CT-W4A8-INT8` | compressed-tensors W4A8 INT8 | W4/A8 | platform selected | - | - | - | - | - | `INVENTORIED` | - | leaf spec open | - | | `QUANT-CT-W8A8-FP8` | compressed-tensors W8A8 FP8 | W8/A8 | platform selected | - | - | - | - | - | `INVENTORIED` | - | leaf spec open | - | diff --git a/.agents/state.md b/.agents/state.md index 87358fca5..1071107fa 100644 --- a/.agents/state.md +++ b/.agents/state.md @@ -36629,3 +36629,52 @@ bench — tools/bench/online_gate.py c1..c8 x3, single load/arm, ours vs oracle Yi30/Qwen3-8B-MXFP4, oracle arm VLLM_DISABLED_KERNELS=FlashInferMxFp4LinearKernel; (3) the p3 near-tie distributional verdict (oracle K-run set) while the oracle is loaded. Branch `row/SERVE-ASYNC-DENSE-MIRROR`. + +## ROW-SERVE-ASYNC-DENSE-MIRROR: dgx GB10 verification — async gate RED→GREEN (0.6B+4B) + SACRED no-regression + memcheck + MXFP4 default-config e2e + p3 near-tie RATIFIED + + +Ran the full dgx GB10 campaign for `f9c969ae` (git-archive → /dev/shm build, CUDA +`-DVLLM_CPP_CUDA=ON -DVLLM_CPP_TRITON=ON -DVLLM_CPP_CUDA_ARCHITECTURES=121a +-DVLLM_CPP_CUTLASS_DIR=$HOME/cutlass-4.5.0`, Release, nvcc 13.0; both flock locks +held per run, tmux + done-markers, free-g 98-100 GiB throughout, local-ai-worker +parked). All correctness proofs GREEN. + +ASYNC GATE RED→GREEN (same binary, `VT_ASYNC_DEVICE_MIRROR` env-toggled): +- 0.6B GREEN (default) 41/41 — every async batch-1 rep + concurrent request reproduces + the SYNC anchor (" capital of Italy is Rome. ... Beijing. The capital of Japan"). +- 0.6B RED (`=0`) FAILURE, 3 CHECKs — concurrency requests degenerate into + " capital of Italy the Germany is!!!!!!!!!!" (token-0 garbage). P0 reproduced. +- 4B GREEN 41/41; 4B RED FAILURE 3 CHECKs. Both models. + +SACRED NO-REGRESSION: `test_qwen3_paged_engine` 0.6B+4B 184/184, 16/16 prompts each +(11/16 strict + 5/16 near-tie, max gap 0.125/0.25 nats, 0 forward-divergent) — my +change is byte-neutral on the sync path (the mirror override is null there), confirmed. + +MEMCHECK: compute-sanitizer memcheck on the 0.6B async gate GREEN arm = ERROR SUMMARY +0 errors, 41/41 assertions under the sanitizer. + +MXFP4 Yi30/Qwen3-8B-MXFP4 (a classic dense `Qwen3ForCausalLM`) DEFAULT-config e2e +(vllm-cli, async ON + fix, greedy 48 tok): p0 " Paris. What is the capital of Italy?...", +p1 " 4. Q: What is 3+3? A: 6...", p3 fibonacci = TOKEN-EXACT vs golden; p2 story +coherent " there lived a young girl named Lily..." (near-tie). Same binary with +`VT_ASYNC_DEVICE_MIRROR=0` DEGENERATES (" Paris. What I I I What What ... !!!!!!") — +exactly the QUANT-CT-MXFP4 W3 async-default degeneration, now CLOSED by the fix. + +p3/p2 NEAR-TIE RATIFIED (oracle, `VLLM_DISABLED_KERNELS=FlashInferMxFp4LinearKernel` ++ ninja/nvcc on PATH — the DGX non-interactive quirk): oracle re-reproduces the 4-prompt +golden today (Marlin W4A16); its story greedy is deterministic (K=8 singleton +" there was a wise old man"). Teacher-forcing the oracle on OUR story sequence: our +token is the oracle's OWN argmax at EVERY continuation position (max gap 0.0000 nats, +no divergence in the 0.5-nat band) — the free-run divergence is the oracle contradicting +itself at a genuine bf16 tie (prefill-vs-decode), the documented near-tie regime. VERDICT +NEAR-TIE-RATIFIED. + +RESIDUAL (owed): the W4 THROUGHPUT bench (online_gate.py c1..c8x3 ours vs oracle) is +NOT done — `online_gate.py` MODEL_REVISIONS only carries the "27"/"35" NVFP4 gate-model +keys; the fingerprinted harness (`dgx-online-serving.sh`, per-model corpora + +`record-oracle` via a paged_engine test binary) has no Yi30/8B entry. Retargeting needs +a MODEL_REVISIONS/REPOSITORIES entry + corpus generation + oracle recording + sizing. +The fix UNBLOCKS the default-config number (the async-default degeneration no longer +forces `VT_ASYNC_SCHED=0`); the oracle is proven to run the model today. Plus the +sibling scope one-liner (InternLM2/Mistral/Llama). dgx build tree persists at +`dgx:/dev/shm/serve-async-dense`. diff --git a/docs/BENCHMARKS.md b/docs/BENCHMARKS.md index 8dd9a3354..5f1d9099a 100644 --- a/docs/BENCHMARKS.md +++ b/docs/BENCHMARKS.md @@ -307,7 +307,7 @@ built on it rather than keeping the flattering one. | Qwen3-dense decode CUDA-graph | Token-exact pass, ~4.3% e2e directional | Steady-state per-step tok/s | | Kimi-Linear-48B-A3B (KDA+MLA+MoE) | Full-model GB10 e2e RUNS (bf16-resident §13), NEAR-TIE 106/128, pool math CLOSES; default OFF | Full model RUNS on GB10 (bf16-resident, RSS peak 1.7 GiB, min-avail 21 GiB, no OOM). Token NEAR-TIE 106/128 (6/8 prompts exact, numerics vs deterministic oracle). 1.59 tok/s. Detail: spec §13 | | vLLM 0.26 re-benchmark | Pending | Re-run the binding grids on the advanced pin | -| MXFP4 Qwen3-8B (W4A16 Marlin) | Compute proven (#38); e2e now token-exact async-DEFAULT after the classic-dense async-mirror fix (`ROW-SERVE-ASYNC-DENSE-MIRROR`) | W4 online_gate c1..c8 x3 vs oracle (`VLLM_DISABLED_KERNELS=FlashInferMxFp4LinearKernel`), running on dgx; p3 near-tie verdict owed | +| MXFP4 Qwen3-8B (W4A16 Marlin) | Compute proven (#38); **DEFAULT-config (async ON) e2e coherent + 3/4 token-exact** on dgx after `ROW-SERVE-ASYNC-DENSE-MIRROR`; degenerate without the fix | p2/p3 near-tie RATIFIED (oracle teacher-forced: max gap 0.0000 nats). W4 THROUGHPUT bench still owed: online_gate lacks a Yi30/8B key (see benchmark-record) | | SGLang floor arms | Never ran | Both arms of the SGLang comparison | | cuBLAS invocation-parity guard | CI guard landed (CPU); `kGemvHeuristicAlgos` refactor build-verify owed | `nvcc` rebuild + SACRED gate on dgx | diff --git a/docs/FEATURES.md b/docs/FEATURES.md index 3cc0dea93..9ff06e962 100644 --- a/docs/FEATURES.md +++ b/docs/FEATURES.md @@ -65,7 +65,7 @@ are our reading of their documented behavior, not measurements. | GGUF k-quants and i-quants | ✅ | ☐ | ☐ | ✅ | | AWQ | ◐ CPU dequant | ✅ | ✅ | ☐ | | GPTQ | ◐ CPU dequant | ✅ | ✅ | ☐ | -| MXFP4 compressed-tensors | ◐ W4A16 Marlin compute proven, bench owed | ✅ | ✅ | ☐ | +| MXFP4 compressed-tensors | ◐ W4A16 Marlin compute proven; default-config e2e coherent + 3/4 token-exact (async-mirror fix); throughput bench owed | ✅ | ✅ | ☐ | | fp8 weights | ✅ | ✅ | ✅ | ☐ | | bf16 / fp16 | ✅ | ✅ | ✅ | ✅ | | Safetensors direct load, no conversion | ✅ | ✅ | ✅ | ☐ | diff --git a/docs/STATUS.md b/docs/STATUS.md index 402ca7031..dbcd58ea3 100644 --- a/docs/STATUS.md +++ b/docs/STATUS.md @@ -53,7 +53,7 @@ token-for-token correctness against the pinned oracle. |---|---|---| | Qwen3.6-27B (NVFP4) text generation | Correctness-complete, at/above vLLM speed | Token-exact greedy on GB10; beats vLLM 0.25.0 total throughput at every concurrency (1.007-1.045x), effective parity 115/124 axes | | Qwen3.6-35B-A3B (NVFP4, GDN MoE) | Correctness-complete; 3-rep grid 0.93-1.03x. Async batch-1 token-0 degeneration FIXED: `VT_ASYNC_DEVICE_MIRROR` default ON | Token-exact SYNC+ASYNC (RED→GREEN); c16 0.93x; `VT_ASYNC_EXECUTOR` Option A (H2D out of capture) GREEN+RED but A/B NEUTRAL → OFF; c16 residual is prefill glue | -| Qwen3 / Qwen2 dense (BF16) | Correctness-complete, speed-pending. Async-serving P0 FIXED (`ROW-SERVE-ASYNC-DENSE-MIRROR`): classic-dense `Qwen3ForCausalLM` now honors the async device token-ids mirror | Near-tie-robust token-exact vs vLLM (Qwen3-0.6B, Qwen3-4B); c1 effective parity, c8 decode residual. **Async device-mirror (`ROW-SERVE-ASYNC-DENSE-MIRROR`): the #31 fix ported to the classic dense family.** The shared dense `EmbedInto` (qwen3.cpp) raced the async combine's device input-ids write against a stale host upload → nondeterministic token-0 degeneration on the depth-2 AsyncLLM serving path (proven by the MXFP4 campaign; quant-independent, hits bf16/NVFP4). Now `EmbedInto` consumes the device override (`ApplyDeviceTokenIdsOverride`) published by `ForwardQwen3ForCausalLM`'s `DeviceTokenIdsScope`, exactly mirroring the 27B-dense template. New gate `test_qwen3_dense_async_serving` (Qwen3-0.6B/4B, batch-1 + concurrency, token-exact vs the race-free in-process SYNC anchor): RED on `VT_ASYNC_DEVICE_MIRROR=0`, GREEN on the default. Byte-identical when the mirror is off. RESIDUAL: the sibling registries sharing this driver (InternLM2, Mistral, Llama) get the fixed consumer but still need the one-line scope; own-embed models per `decode-framework-routing-audit`. **D1 (2026-07-31, `CLAIM-D1-BF16-MERGED-QKV`): the bf16 merged-QKV path (`Qwen3QkvMergeEnabled`/`VT_QWEN3_QKV_MERGE`) is now default-ON** — one `vt::MatmulBT` over the merged `[qdim+2kdim,H]` owner + a contiguous `vt::QkvSplit` (OLMo-2 exemplar), replacing three per-shard GEMMs. Bit-exact GEMM math (A/B unit `test_ops_qkv_merge` byte-identical, RED-first); the wider-N cuBLASLt K-reduction flips the 0.6B genuine bf16 near-tie so the SACRED 0.6B golden was regenerated (all tokens within the near-tie band, max 0.125 nats), while Qwen3-4B is byte-neutral (0 diffs, stays STRICT). Re-gated 0.6B 16/16 + 4B 16/16; consistency/launch-count fold (measured NEUTRAL on 4B decode), no new throughput owed | +| Qwen3 / Qwen2 dense (BF16) | Correctness-complete, speed-pending. Async-serving P0 FIXED (`ROW-SERVE-ASYNC-DENSE-MIRROR`): classic-dense `Qwen3ForCausalLM` now honors the async device token-ids mirror | Near-tie-robust token-exact vs vLLM (Qwen3-0.6B, Qwen3-4B); c1 effective parity, c8 decode residual. **Async device-mirror (`ROW-SERVE-ASYNC-DENSE-MIRROR`, `f9c969ae`): the #31 fix ported to the classic dense family, dgx-VERIFIED.** The shared dense `EmbedInto` (qwen3.cpp) raced the async combine's device input-ids write against a stale host upload → token-0 degeneration on the depth-2 AsyncLLM serving path (quant-independent). `EmbedInto` now consumes the device override published by `ForwardQwen3ForCausalLM`'s `DeviceTokenIdsScope` (27B-dense template); gate `test_qwen3_dense_async_serving` RED on `VT_ASYNC_DEVICE_MIRROR=0`, GREEN default, byte-identical mirror-off. dgx GB10: async gate RED→GREEN 0.6B+4B, SACRED 0.6B+4B 184/184 unchanged (byte-neutral sync path), memcheck 0 errors; Yi30/Qwen3-8B-MXFP4 default-config e2e coherent + 3/4 token-exact (p2 = oracle-ratified near-tie, gap 0.0000), closing the QUANT-CT-MXFP4 async-default residual. RESIDUAL: sibling InternLM2/Mistral/Llama scope one-liner; W4 throughput bench (online_gate harness gap). Detail in state.md. **D1 (2026-07-31, `CLAIM-D1-BF16-MERGED-QKV`): the bf16 merged-QKV path (`Qwen3QkvMergeEnabled`/`VT_QWEN3_QKV_MERGE`) is now default-ON** — one `vt::MatmulBT` over the merged `[qdim+2kdim,H]` owner + a contiguous `vt::QkvSplit` (OLMo-2 exemplar), replacing three per-shard GEMMs. Bit-exact GEMM math (A/B unit `test_ops_qkv_merge` byte-identical, RED-first); the wider-N cuBLASLt K-reduction flips the 0.6B genuine bf16 near-tie so the SACRED 0.6B golden was regenerated (all tokens within the near-tie band, max 0.125 nats), while Qwen3-4B is byte-neutral (0 diffs, stays STRICT). Re-gated 0.6B 16/16 + 4B 16/16; consistency/launch-count fold (measured NEUTRAL on 4B decode), no new throughput owed | | Qwen3.5-4B plain BF16 direct loading on discrete CUDA | Correctness-complete, speed-pending | Revalidated after merging current upstream: local throughput is unchanged at 0.99997x its prior run; against the freshly measured pinned oracle it is 0.9971x. TTFT 0.7719x and host PSS 0.3127x pass; TPOT/ITL 1.1244x and VRAM 1.0014x remain open. Direct ON/OFF outputs remain 128/128 identical | | Qwen3-Coder-30B-A3B MoE (BF16) | Correctness-complete, speed-pending | Near-tie-robust token-exact 6/6; 11 of 16 binding grid cells at or above vLLM. **D1 (2026-07-31): inherits the default-ON bf16 merged-QKV via the shared dense `AttnBlock` — byte-neutral (0 token diffs, golden UNCHANGED); re-gated 6/6** | | Llama-3.x dense (BF16) | Correctness-complete, speed-pending | Near-tie-robust token-exact 16/16 (Llama-3.2-1B); llama3 RoPE scaling |