mudler · localai-bot · Aug 5, 2026 · Aug 5, 2026 · Aug 5, 2026
diff --git a/.agents/NOW.md b/.agents/NOW.md
@@ -22,6 +22,7 @@ checkpoint on `upstream/main` at `59674cf1d`.
 | Kimi-Linear-48B (KDA+NoPE-MLA+MoE) | **Full-model GB10 e2e RUNS** (bf16-resident §13, f32-loader block CLEARED): CPU+CUDA 13/13·656; host RSS peak 1.7 GiB, min-avail 21 GiB, no OOM. **Token gate NEAR-TIE 106/128** (6/8 prompts token-exact; numerics near-tie vs deterministic oracle, not a bug) | STRICT path = device GDN/MLA islands + bf16 stream (W7-speed residuals); 1.59 tok/s; default OFF |
 | 35B fresh grid | **BOUND** @`1ea26427`: tput 0.93-1.03x, c16 0.93x. INTAKE + Option A both **RESOLVED NEGATIVE** (H2D-out-of-capture tput WASH) | Real lever left: prefill glue (task #61) |
 | Qwen3.5-4B revalidation | 0.9971x @`59674cf1` (#35); TTFT/PSS pass, TPOT/ITL open | `docs/bench-evidence/` |
+| ROW-SERVE-ASYNC-DENSE-MIRROR | **LANDED + dgx-VERIFIED** (`f9c969ae`): #31 async mirror ported to classic dense `Qwen3ForCausalLM`. Async gate RED→GREEN 0.6B+4B, SACRED 184/184, memcheck 0, MXFP4 default e2e 3/4 + near-tie RATIFIED | Residual: **W4 throughput bench** (online_gate lacks Yi30/8B key); sibling scope one-liner |
 
 In-flight branches (gated default-OFF, not pushed): `laguna-fp4proj-prod`
 (fp4 opt-in), laguna bf16/legacy/pipeline-gemv, `ds4-hc-expand-fuse`.

diff --git a/.agents/benchmark-record.md b/.agents/benchmark-record.md
@@ -12210,3 +12210,58 @@ a NAMED speed residual, NOT an optimized decode. DEFAULT: `VT_KIMI_DEVICE_COMPUT
 (near-tie != token-exact). PATH TO STRICT + speed = the named W7 residuals: device GDN per-channel
 recurrence + exp/softplus gate op + paged `mla::ForwardMlaAttentionBlock` + a bf16 residual stream
 end-to-end (matches vLLM's rounding, removes the host round-trips).
+
+## 2026-08-07T05:00 — ROW-SERVE-ASYNC-DENSE-MIRROR: classic-dense async P0 fix (code+CPU gates) + W4 MXFP4 bench PLAN
+
+CORRECTNESS (not a speed lever): the #31 async device token-ids mirror, ported to the
+classic dense family. Classic dense `Qwen3ForCausalLM` (qwen3.cpp `EmbedInto`) raced
+the async combine's device input-ids write against a stale host upload → nondeterministic
+token-0 degeneration on the depth-2 AsyncLLM serving path (surfaced by the MXFP4
+campaign; quant-independent). Fix: EmbedInto consumes `ApplyDeviceTokenIdsOverride`
+(main-queue-ordered `d.b.Copy` over the DBuf prefix) published by
+`ForwardQwen3ForCausalLM`'s `DeviceTokenIdsScope` — verbatim the 27B-dense template.
+Byte-identical when the mirror is off. Gate `test_qwen3_dense_async_serving` (Qwen3-0.6B/4B,
+async==in-process-SYNC anchor; RED on VT_ASYNC_DEVICE_MIRROR=0, GREEN default). CPU
+-Werror clean; regression suites GREEN (input_batch 183 / combine_tokens 14 / runner 323 /
+engine_core_proc 576 / async_llm 309); test_llm_engine = documented flaky (unrelated TU).
+
+OWED ON DGX (this row):
+- Async gate RED→GREEN (same binary, VT_ASYNC_DEVICE_MIRROR env-toggled) + SACRED
+  test_qwen3_paged_engine (0.6B/4B) unchanged + ignore_eos bracket + compute-sanitizer
+  memcheck on the new gate.
+- THE MXFP4 W4 BENCH (now unblocked): `tools/bench/online_gate.py` c1..c8 x3 reps, single
+  load per arm, OURS (fixed default config) vs ORACLE on Yi30/Qwen3-8B-MXFP4. Oracle arm
+  MUST set `VLLM_DISABLED_KERNELS=FlashInferMxFp4LinearKernel` (its default MXFP4 dispatch
+  crashes on sm_121). Match-or-beat is the bar; honest either way. Update the BENCHMARKS
+  MXFP4 Qwen3-8B row + quantization-matrix with the numbers.
+- p3 (open-ended story) near-tie distributional verdict against the oracle K-run set while
+  the oracle is loaded (converts the QUANT-CT-MXFP4 "3/4 token-exact, p3 diverges after
+  identical first token" into a ratified near-tie).
+Box safety: both locks (flock $HOME/gpu.lock AND /tmp/gpu), free -g >= 90, no oracle
+alongside our server, local-ai-worker stopped, tmux + done-markers, git archive not rsync.
+
+## 2026-08-07T07:00 — ROW-SERVE-ASYNC-DENSE-MIRROR dgx GB10 verification (f9c969ae)
+
+CUDA build (/dev/shm, 121a, cutlass 4.5.0, nvcc 13.0, Release). Both flock locks per run,
+tmux + done-markers, free-g 98-100 GiB. CORRECTNESS (this row is a correctness fix, not a
+speed lever):
+- ASYNC GATE (same binary, env-toggled): 0.6B + 4B GREEN (default) 41/41 async==sync;
+  0.6B + 4B RED (VT_ASYNC_DEVICE_MIRROR=0) FAIL 3 CHECKs — concurrency requests degenerate
+  into "...the Germany is!!!!!!" token-0 garbage. P0 reproduced + fixed on real HW.
+- SACRED test_qwen3_paged_engine 0.6B+4B 184/184, 16/16 prompts each, 0 forward-divergent
+  (byte-neutral sync path).
+- compute-sanitizer memcheck on 0.6B async GREEN: 0 errors.
+- MXFP4 Yi30/Qwen3-8B-MXFP4 DEFAULT-config (async ON + fix) vllm-cli greedy 48 tok: p0/p1/p3
+  TOKEN-EXACT vs golden, p2 story coherent near-tie; VT_ASYNC_DEVICE_MIRROR=0 same binary
+  DEGENERATES (" Paris. What I I I What... !!!!!!") = the W3 async-default degeneration, CLOSED.
+- p3/p2 NEAR-TIE RATIFIED: oracle (VLLM_DISABLED_KERNELS=FlashInferMxFp4LinearKernel, ninja+nvcc
+  on PATH) re-reproduces the golden; story greedy K=8 singleton; teacher-forcing the oracle on
+  OUR sequence => our token is the oracle argmax at EVERY position (max gap 0.0000 nats). The
+  free-run " wise old man" vs our " young girl" is the oracle's own prefill-vs-decode tie.
+
+W4 THROUGHPUT BENCH NOT DONE (harness gap, honest): online_gate.py MODEL_REVISIONS carries
+only "27"/"35"; no Yi30/8B key. Retargeting = MODEL_REVISIONS/REPOSITORIES entry + corpus
+(make_serve_low_corpus) + record-oracle (needs an 8B paged_engine test binary) + num-blocks/
+max-num-seqs sizing. The fix unblocks the DEFAULT-config bench (no more VT_ASYNC_SCHED=0
+workaround); oracle proven to run the model today. Evidence: dgx:/dev/shm/serve-async-dense/
+{gates_06b.log,gates2.log,mxfp4_e2e.log,oracle_neartie2.log,oracle_tf.log}.
diff --git a/.agents/model-matrix.md b/.agents/model-matrix.md
@@ -54,7 +54,7 @@ Engaged architectures (the 43 non-`INVENTORIED` rows):
 
 | Support | Architecture | Family / example | Status | Row |
 |---|---|---|---|---|
-| ✅ | `Qwen3ForCausalLM` | Qwen3 dense (0.6B/1.7B/4B/32B) | near-tie-robust token-exact 16/16 on 0.6B+4B vs vLLM 0.25.0; NVFP4A16 (W4A16) dense quant also gated; c1 every-axis speed parity, c8 decode residual | `MODEL-TEXT-qwen3-qwen3-for-causal-lm` |
+| ✅ | `Qwen3ForCausalLM` | Qwen3 dense (0.6B/1.7B/4B/32B) | near-tie-robust token-exact 16/16 on 0.6B+4B vs vLLM 0.25.0; NVFP4A16 (W4A16) dense quant also gated; c1 every-axis speed parity, c8 decode residual; async-serving device token-ids mirror ported (`ROW-SERVE-ASYNC-DENSE-MIRROR`, #31 fix into the shared dense `EmbedInto`) — `test_qwen3_dense_async_serving` RED→GREEN; sibling InternLM2/Mistral/Llama scope one-liner is a named residual | `MODEL-TEXT-qwen3-qwen3-for-causal-lm` |
 | ✅ | `Qwen3MoeForCausalLM` | Qwen3-Coder-30B-A3B (MoE) | STRICT token-exact 6/6 vs vLLM 0.25.0; 11/16 speed-grid cells at/above graphed vLLM, c1/c2 residual | `MODEL-TEXT-qwen3-moe-qwen3-moe-for-causal-lm` |
 | ✅ | `Qwen3_5ForConditionalGeneration` | Qwen3.6-27B (text path) | text-gen STRICT token-exact 235/235 vs vLLM 0.25.0; mm INPUT pipeline (M0/M1) landed + processor-parity gate PASS; **M3-W0 landed** (vision-inclusive checkpoint `Qwen/Qwen3.6-27B` 51.7 GiB bf16 with 333 `visual.*` FOUND+fits+downloaded; 27B vision config resolved — depth 27/out 5120/**EMPTY deepstack**; MRoPE `[11,11,10]`/rot 64/theta 1e7; the bf16 GDN-hybrid loader ALREADY handles it). **M3-b LANDED 2026-07-25: image→text STRICT token-exact 32/32 vs vLLM 0.25.0** — Qwen3.6-27B image understanding works end-to-end (forked GDN-hybrid VL forward gated on mm input ⇒ text byte-identical; 27B/35B/Coder inertness re-passed 235/315/138). **M3d LANDED 2026-07-25: video→text STRICT token-exact 32/32 vs vLLM 0.25.0** — video works end-to-end too (`Qwen3_5VLGenerateGreedyVideo` reuses the M3c processor/windowed-tower/video-MRoPE on the GDN-hybrid backbone). **Qwen video modalities COMPLETE: image+video both work e2e** (audio N/A for Qwen). **VISION-FORWARD SPEED (2026-07-28, `CLAIM-MM-SPEED-QWEN-IMAGE`, multimodal-speed.md §16): the mm-forward tower BEATS vLLM** — per-image tower forward 142.3 ms (flash `AttentionDenseFlash`, hd-72) vs vLLM 0.25.0 ~250 ms eager encode = 0.57×; attribution-first nsys REFUTED a bigger lever (the t=784 vision attention is serial-latency-bound, flash only 1.04× over warp), STRICT 32/32 image/video HELD + goldens md5 unchanged. Row stays `PARTIAL` — vision-forward speed BEATS vLLM; **umbrella speed pending** on batched c2+/serving | `MODEL-MM-qwen3-5-qwen3-5-for-conditional-generation` |
 | ✅ | `Qwen3_5MoeForConditionalGeneration` | Qwen3.6-35B-A3B (text path) | text-gen STRICT token-exact 315/315 vs vLLM 0.25.0; mm INPUT pipeline (M0/M1) landed + processor-parity gate PASS, vision tower pending (M2/M3) so the row is `PARTIAL` (text-only) | `MODEL-MM-qwen3-5-qwen3-5-moe-for-conditional-generation` |

diff --git a/.agents/quantization-matrix.md b/.agents/quantization-matrix.md
@@ -127,7 +127,7 @@ Registry source:
 | `QUANT-FP8-PCPT` | ModelOpt FP8 per-channel/per-token | W8/A8 | capability selected | - | - | - | - | - | `INVENTORIED` | - | leaf spec open | - |
 | `QUANT-MXFP8-MODELOPT` | ModelOpt MXFP8 | W8/A8 | CUDA/ROCm/XPU dispatch | - | - | - | - | - | `INVENTORIED` | - | leaf spec open | - |
 | `QUANT-MIXED-MODELOPT` | ModelOpt mixed precision | FP8/NVFP4/MXFP8 groups | per-layer | part | part | part | Y | Y | `PARTIAL` | 35B FP8+NVFP4 slice: [loader](../src/vllm/model_executor/models/qwen3_5_weights.cpp#L118), [FP8 tests](../tests/vt/test_ops_fp8_cutlass.cpp#L188), [NVFP4 tests](../tests/vt/test_ops_moe_grouped.cpp#L453), [gate](../tests/parity/test_qwen36_paged_engine.cpp#L78) | leaf spec open | - |
-| `QUANT-CT-MXFP4` | compressed-tensors MXFP4 `mxfp4-pack-quantized` (group 32, E8M0 block scales, NO global) | W4/A16 native **Marlin mxf4 keep-quant** landed (GB10 target; W4A4 cute-dsl crashes sm_121) | CUDA GB10 Marlin W4A16 (E8M0, group_blocks=2); CPU dequant fallback | Y | Y | Y | Y | - | `ANCHOR-BACKFILL` | **W2 native compute + W3 gates + e2e (row/QUANT-CT-MXFP4, `1c5ee09e`):** W0 vehicle `Yi30/Qwen3-8B-MXFP4` runs on the 0.25.0 oracle; W1 traced FlashInfer-W4A4-selected-but-crashes-on-sm_121 -> **Marlin W4A16 is the GB10 target**. Native path: `generate_kernels.py` MXFP4 config (`kFE8M0fnu`, group_blocks 2) + regenerated instances; [`MarlinProcessExpertScalesMxfp4`](../src/vt/cuda/cuda_marlin_repack.cu) (byte-exact vs vLLM at all shapes); `MoeMarlinArgs.{group_size,mxfp4}` launcher branch; `Nvfp4Weight.{group_size,is_mxfp4}` + [`dense_nvfp4_gemm.h`](../include/vllm/model_executor/models/dense_nvfp4_gemm.h) branch + `MatmulMxfp4W4A16D`; [`dense_weight_loaders.h`](../include/vllm/model_executor/models/dense_weight_loaders.h) MXFP4 loaders; `qwen3_weights.cpp` detect+load. **Gates GREEN:** op-level GEMM vs independent CPU dequant 0.36% M=1/M=8 all real shapes ([`test_ops_moe_grouped.cpp`](../tests/vt/test_ops_moe_grouped.cpp)); model-facing `MakeLinearMethod->Apply->BuildMarlinDenseResident` bad=0 K=4096+12288 ([`test_linear_method.cpp`](../tests/vllm/model_executor/layers/test_linear_method.cpp)); **e2e 3/4 token-exact vs oracle golden (async-off)** ([evidence](../docs/bench-evidence/mxfp4-qwen/W3-e2e-result.md)). RESIDUAL: default-async degeneration is a PRE-EXISTING classic-dense-Qwen3 async bug (device-mirror not wired for `qwen3.cpp`, quant-independent, SEPARATE row); W4 bench owed; p3 formal distributional gate owed. Earlier CPU weight unpack + E8M0 dequant: NEW [mxfp4_dequant.h](../include/vllm/model_executor/model_loader/mxfp4_dequant.h) + [.cpp](../src/vllm/model_executor/model_loader/mxfp4_dequant.cpp#L14) (`E8M0ToF32` = `2^(byte-127)`, `DequantMxfp4ToBf16`/`ToF32`, group 32, no global; reuses `kE2M1Lut`). Unit gate [test_mxfp4_dequant.cpp](../tests/vllm/test_mxfp4_dequant.cpp#L34) — E8M0 known-byte decode, hand-computed 32-group dequant (bf16+f32), the E8M0-vs-fp8 + group-32-vs-16 RED traps, multi-row/group offsets, randomized rel-error vs a double-precision port of `dq_mxfp4_torch` with bf16==f32 exactness. CPU `-Werror` 0-warn. Ports FROM `compressed_tensors_w4a4_mxfp4.py:20-97` + `mxfp8_utils.py:61-65,222` + golden `tests/quantization/reference_mxfp4.py:28-117`. C/E/P PENDING: GPU W4A4 fp4 GEMM + Marlin W4A16 fallback + MoE expert path + e2e are NAMED later bricks; DeepSeek-V4 + Kimi-K3 loaders consume this once wired | [MXFP4 spike](specs/mxfp4-compressed-tensors.md) | `CLAIM-QUANT-MXFP4` |
+| `QUANT-CT-MXFP4` | compressed-tensors MXFP4 `mxfp4-pack-quantized` (group 32, E8M0 block scales, NO global) | W4/A16 native **Marlin mxf4 keep-quant** landed (GB10 target; W4A4 cute-dsl crashes sm_121) | CUDA GB10 Marlin W4A16 (E8M0, group_blocks=2); CPU dequant fallback | Y | Y | Y | Y | - | `ANCHOR-BACKFILL` | **W2 native compute + W3 gates + e2e (row/QUANT-CT-MXFP4, `1c5ee09e`):** W0 vehicle `Yi30/Qwen3-8B-MXFP4` runs on the 0.25.0 oracle; W1 traced FlashInfer-W4A4-selected-but-crashes-on-sm_121 -> **Marlin W4A16 is the GB10 target**. Native path: `generate_kernels.py` MXFP4 config (`kFE8M0fnu`, group_blocks 2) + regenerated instances; [`MarlinProcessExpertScalesMxfp4`](../src/vt/cuda/cuda_marlin_repack.cu) (byte-exact vs vLLM at all shapes); `MoeMarlinArgs.{group_size,mxfp4}` launcher branch; `Nvfp4Weight.{group_size,is_mxfp4}` + [`dense_nvfp4_gemm.h`](../include/vllm/model_executor/models/dense_nvfp4_gemm.h) branch + `MatmulMxfp4W4A16D`; [`dense_weight_loaders.h`](../include/vllm/model_executor/models/dense_weight_loaders.h) MXFP4 loaders; `qwen3_weights.cpp` detect+load. **Gates GREEN:** op-level GEMM vs independent CPU dequant 0.36% M=1/M=8 all real shapes ([`test_ops_moe_grouped.cpp`](../tests/vt/test_ops_moe_grouped.cpp)); model-facing `MakeLinearMethod->Apply->BuildMarlinDenseResident` bad=0 K=4096+12288 ([`test_linear_method.cpp`](../tests/vllm/model_executor/layers/test_linear_method.cpp)); **e2e 3/4 token-exact vs oracle golden (async-off)** ([evidence](../docs/bench-evidence/mxfp4-qwen/W3-e2e-result.md)). **Default-async degeneration RESOLVED (`ROW-SERVE-ASYNC-DENSE-MIRROR`, `f9c969ae`):** the pre-existing classic-dense-Qwen3 async bug (device-mirror not wired for `qwen3.cpp`) is fixed; DEFAULT-config (async ON) e2e on dgx is now coherent + 3/4 token-exact vs the golden (p0/p1/p3 exact; p2 story = oracle-ratified near-tie, teacher-forced max gap 0.0000 nats). p3 formal near-tie gate now RATIFIED. RESIDUAL: the W4 THROUGHPUT bench (c1..c8x3 vs oracle) still owed — online_gate lacks a Yi30/8B model key (needs corpus + oracle-record plumbing). Earlier CPU weight unpack + E8M0 dequant: NEW [mxfp4_dequant.h](../include/vllm/model_executor/model_loader/mxfp4_dequant.h) + [.cpp](../src/vllm/model_executor/model_loader/mxfp4_dequant.cpp#L14) (`E8M0ToF32` = `2^(byte-127)`, `DequantMxfp4ToBf16`/`ToF32`, group 32, no global; reuses `kE2M1Lut`). Unit gate [test_mxfp4_dequant.cpp](../tests/vllm/test_mxfp4_dequant.cpp#L34) — E8M0 known-byte decode, hand-computed 32-group dequant (bf16+f32), the E8M0-vs-fp8 + group-32-vs-16 RED traps, multi-row/group offsets, randomized rel-error vs a double-precision port of `dq_mxfp4_torch` with bf16==f32 exactness. CPU `-Werror` 0-warn. Ports FROM `compressed_tensors_w4a4_mxfp4.py:20-97` + `mxfp8_utils.py:61-65,222` + golden `tests/quantization/reference_mxfp4.py:28-117`. C/E/P PENDING: GPU W4A4 fp4 GEMM + Marlin W4A16 fallback + MoE expert path + e2e are NAMED later bricks; DeepSeek-V4 + Kimi-K3 loaders consume this once wired | [MXFP4 spike](specs/mxfp4-compressed-tensors.md) | `CLAIM-QUANT-MXFP4` |
 | `QUANT-CT-W4A8-FP8` | compressed-tensors W4A8 FP8 | W4/A8 | CUTLASS | - | - | - | - | - | `INVENTORIED` | - | leaf spec open | - |
 | `QUANT-CT-W4A8-INT8` | compressed-tensors W4A8 INT8 | W4/A8 | platform selected | - | - | - | - | - | `INVENTORIED` | - | leaf spec open | - |
 | `QUANT-CT-W8A8-FP8` | compressed-tensors W8A8 FP8 | W8/A8 | platform selected | - | - | - | - | - | `INVENTORIED` | - | leaf spec open | - |