mudler · localai-bot · Aug 6, 2026
diff --git a/.agents/NOW.md b/.agents/NOW.md
@@ -1,6 +1,6 @@
 # NOW — the one-Read resume surface
 
-<!-- now-updated: 2026-08-06 -->
+<!-- now-updated: 2026-08-08 -->
 
 Read this FIRST, every session. A SNAPSHOT, rewritten in place: what is live,
 the gate being chased, what to do next. Never a log — evidence lives in the
@@ -14,14 +14,15 @@ checkpoint on `upstream/main` at `59674cf1d`.
 
 | Claim / track | State | Next command or step |
 |---|---|---|
-| Laguna NVFP4 decode speed | **Closed: PARITY+ 1.03x** (44.46 vs 43.10, byte-exact, default). Root cause = bf16 weight residency via `VT_LAGUNA_RESIDENT_BF16W` (default-ON). Detail in benchmark record | Residual: formal vLLM K-run set when convenient |
-| DeepSeek-V4-Flash decode | **Closed: BEATS ds4 1.144x** (`VT_V4_RESIDENT_W` on, byte-exact). Phase-2 routed-expert residency NEGATIVE 2026-08-05 (−3.4%), HELD default-OFF. See state | — |
+| Laguna NVFP4 decode speed | **Closed: PARITY+ 1.03x** (byte-exact, default; `VT_LAGUNA_RESIDENT_BF16W` bf16-residency). Benchmark record | vLLM K-run set when convenient |
+| DeepSeek-V4-Flash decode | **Closed: BEATS ds4 1.144x** (`VT_V4_RESIDENT_W`, byte-exact). Phase-2 routed-expert residency NEGATIVE (−3.4%), default-OFF | — |
 | f32-out GEMV audit | Only laguna + deepseek_v4 bf16 tower affected; gate models & on-framework dense unaffected (bf16-out, e2e-verified) | Re-verify deepseek_v4 bf16 tower same-tool |
 | Invocation-parity prevention | CI guard (`check-gemv-invocation-consistency.py`) + AGENTS.md checklist landing | Review + merge; CUDA build-verify `kGemvHeuristicAlgos` on dgx |
-| MiniMax-H3 lane | Portable path complete; e2e prompt-conditioned video on real weights (Thor). Speed = NVFP4 FP4 device path, sm_121-gated | PR #26 rebase + supports-audit synthesis (workflow ran; integrate) |
-| Kimi-Linear-48B (KDA+NoPE-MLA+MoE) | **Full-model GB10 e2e RUNS** (bf16-resident §13, f32-loader block CLEARED): CPU+CUDA 13/13·656; host RSS peak 1.7 GiB, min-avail 21 GiB, no OOM. **Token gate NEAR-TIE 106/128** (6/8 prompts token-exact; numerics near-tie vs deterministic oracle, not a bug) | STRICT path = device GDN/MLA islands + bf16 stream (W7-speed residuals); 1.59 tok/s; default OFF |
+| MiniMax-H3 lane | Portable path complete; e2e prompt-conditioned video on real weights (Thor). Speed = NVFP4 FP4 device path, sm_121-gated | PR #26 rebase + supports-audit synthesis |
+| Kimi-Linear-48B (KDA+NoPE-MLA+MoE) | **Full-model GB10 e2e RUNS** (bf16-resident §13): CPU+CUDA 13/13·656, no OOM. **Token gate NEAR-TIE 106/128** (6/8 token-exact) | device GDN/MLA islands + bf16 stream; 1.59 tok/s; default OFF |
 | 35B fresh grid | **BOUND** @`1ea26427`: tput 0.93-1.03x, c16 0.93x. INTAKE + Option A both **RESOLVED NEGATIVE** (H2D-out-of-capture tput WASH) | Real lever left: prefill glue (task #61) |
 | Qwen3.5-4B revalidation | 0.9971x @`59674cf1` (#35); TTFT/PSS pass, TPOT/ITL open | `docs/bench-evidence/` |
+| MXFP4 c2-c8 lever = FA2 decode GQA group-swap | **Ported gated OFF** (`VT_FA2_DECODE_GQA_SWAP`): vLLM decode swap in the d128 varlen launcher; correctness-complete GB10 (op RED/GREEN, memcheck, swap-ON smoke token-exact), OFF byte-identical | bench + flip: state `KERNEL-FA2-GQA-SWAP` |
 | ROW-SERVE-ASYNC-DENSE-MIRROR | **LANDED+dgx-VERIFIED** (`f9c969ae`): #31 async mirror on classic dense Qwen3; gate RED→GREEN, SACRED 184/184 | Residual: sibling scope one-liner |
 | QUANT-CT-MXFP4-BENCH | ~0.91x@c2-c8; marlin+glue levers **both REFUTED**; step = flash GQA group-swap | LEVER + detail: state `QUANT-CT-MXFP4-GLUE` (`cuda_flash_attn_fa2.cu:1075`) |
 

diff --git a/.agents/benchmark-record.md b/.agents/benchmark-record.md
@@ -12376,3 +12376,40 @@ only "27"/"35"; no Yi30/8B key. Retargeting = MODEL_REVISIONS/REPOSITORIES entry
 max-num-seqs sizing. The fix unblocks the DEFAULT-config bench (no more VT_ASYNC_SCHED=0
 workaround); oracle proven to run the model today. Evidence: dgx:/dev/shm/serve-async-dense/
 {gates_06b.log,gates2.log,mxfp4_e2e.log,oracle_neartie2.log,oracle_tf.log}.
+
+## 2026-08-08T18:00 — KERNEL-FA2-GQA-SWAP: d128 varlen decode group-swap gated-OFF, correctness gates GREEN (bench PENDING)
+
+The #47-localized lever (`row/KERNEL-FA2-GQA-SWAP`, `VT_FA2_DECODE_GQA_SWAP`, default
+OFF): vLLM's FA2 `seqlenq_ngroups_swapped` decode grid ported into
+`LaunchDecodeVarlenFA2Bf16` so the Qwen3-dense d128 decode launches
+`(batch, kv_heads)` not `(batch, hq)`. `benchmark_binding=false` this session — the
+c1-c8 x3 binding grid + default flip is the recorded next step; what ran here is the
+CORRECTNESS bar that GATES the flip. dgx GB10, CUDA 13.0, sm_121a, RelWithDebInfo,
+VLLM_CPP_FLASH_ATTN=ON; the 4 changed files git-archived by SHA onto `~/mxfp4-bench/src`
+(md5-matched local), incremental `ninja` (both CUDA TUs recompiled clean on nvcc).
+
+GATES:
+- OP UNIT TEST (RED-first): `test_ops_paged_attn --test-case="*varlen d128*"` = 5/5
+  cases, **280/280 assertions GREEN** (OFF-path parity + group-swap-matches-ref both
+  ratios 16/8+32/8 × batch{1,2,4,8} × len{5,21,1024}, `swap_launches==1` asserted;
+  swap-vs-plain near-tie max_abs<2e-2; MHA qpk==1 inert `swap_launches==0`). Full
+  binary **28/28 cases, 454,679 assertions** — no regression.
+- RED PROVEN: injecting a wrong swapped `o_head_stride` (drop the ngroups factor) →
+  group-swap case FAILS (26,528 violations, max_abs 2.60). Restored + rebuilt
+  (md5 `ba34d5b8…`).
+- MEMCHECK: `compute-sanitizer --tool memcheck --leak-check full` on the swap cases =
+  **0 errors, 0 bytes leaked**.
+- #44 MXFP4 SMOKE (Yi30/Qwen3-8B-MXFP4, groups=4, DEFAULT async/graphed): swap-OFF PASS
+  (3/3 det token-exact + coherent); **swap-ON PASS (3/3 deterministic TOKEN-EXACT vs
+  golden AND byte-identical to swap-OFF** — capitals/arithmetic/fibonacci char-identical,
+  story identical). No token flip e2e → near-tie razor not needed; graphed capture-safety
+  token gate ON.
+
+NEXT (the flip campaign): `scripts/mxfp4-online-serving-grid.sh --snapshot <b3e7ab32>
+--build-dir ~/mxfp4-bench/build --configure-log <log>` with `VT_FA2_DECODE_GQA_SWAP=1`
+on OURS legs, vs the #45 numbers (tput c1 0.989 / c2 0.911 / c4 0.919 / c8 0.913; TPOT
+c2 28.27→25.45). #47 projects flash alone closes ~28%@c2 / ~55%@c8 (decode flash
+63.7→~41.7us c2 as the grid drops 192→96 CTAs); residual = marlin +7-9% + ~0.7ms host,
+so a single lever may not reach ≥1.0x — record the honest per-axis outcome. Then Qwen3
+0.6B/4B e2e SACRED swap-ON, then flip per parity-enablers. Box left clean (both locks
+free, GPU idle, worker down, disk 22G).
diff --git a/.agents/kernel-matrix.md b/.agents/kernel-matrix.md
@@ -85,6 +85,25 @@ trace-proven at **180.28 us/call**, within 1.1% of the matched vLLM FA2 kernel
 at 178.40 us/call. Evidence:
 [2026-07-25 4B repair](../docs/bench-evidence/qwen35-4b-main-repair-20260725.md).
 
+`KERNEL-ATTN-FA2` d128 varlen decode GQA group-swap (2026-08-06,
+`row/KERNEL-FA2-GQA-SWAP`, `VT_FA2_DECODE_GQA_SWAP`, default OFF): vLLM's
+`seqlenq_ngroups_swapped` decode grid ported into `LaunchDecodeVarlenFA2Bf16`
+(`cuda_flash_attn_fa2.cu`) so the Qwen3-dense d128 decode launches
+(batch, kv_heads) not (batch, hq) — the ngroups query heads pack into seqlen_q,
+KV read once per group, heuristic sees `batch*kv_heads` — halving the CTA count
+at batch>=2 (#47: ours over-waved 192 CTAs c2 / ~5 waves c8). Presented WITHOUT a
+materialized transpose via kv-major-group-minor strides, mirroring the shipped
+d256 `LaunchDecodeFA2Bf16` swap (the vendored `get_lse_tile`/combine already
+honor the flag). Gated so OFF is byte-identical to the shipped plain-varlen
+reduction; ON is non-byte-exact only when num_splits>1 (split reduction order
+→ near-tie, toward vLLM). Correctness-complete on GB10 (CUDA 13.0, sm_121a):
+op RED-first test **280/280 assn GREEN** (both ratios 16/8+32/8, batch 1/2/4/8,
+short+long context; `swap_launches==1` proves the grid engaged; wrong-stride RED
+= 26528 violations), compute-sanitizer **0-err/0-leak**, and the #44 MXFP4 e2e
+smoke swap-ON **3/3 deterministic TOKEN-EXACT + coherent**, byte-identical to
+swap-OFF (no token flip, near-tie razor unneeded). c1-c8 x3 binding re-bench +
+default flip is the recorded next step. Detail: state `KERNEL-FA2-GQA-SWAP`.
+
 | ID | Item | Upstream | Our code | Tests/evidence | Spike/spec | State | Owner |
 |---|---|---|---|---|---|---|---|
 | `KERNEL-ACCEL-PROVIDER-SELECT` | **WHICH implementation of an op runs, when more than one exists on a device** — the selection layer above every kernel family in this matrix. Distinct from `KERNEL-CUDA-DISPATCH-AOT`, which is about which ARCH a CUDA kernel is compiled/selected for; this is about which PROVIDER (ours, a vendor library, MLX, llama.cpp) serves the op at all | no single upstream file: this is the shape vLLM's runtime chain uses everywhere — flashinfer tactic registries, cuBLASLt/CUTLASS per-call heuristics, and torch's backend selection — rather than compile-time pinning | `vt::OpProvider` [op_provider.h](../include/vt/op_provider.h) + [op_provider.cpp](../src/vt/op_provider.cpp); the flat `[OpId][DeviceType]` `void*` table it replaces is gone from [ops.cpp](../src/vt/ops.cpp) with the ~70 op wrappers untouched. Providers registered today: `vt-native` (every backend kernel in the tree, priority 0, unconditional — behaviour preserved exactly), `mlx` (priority 100, Metal `kMatmul`/`kMatmulBT`, build-gated `VLLM_CPP_MLX`, [metal_mlx_provider.mm](../src/vt/metal/metal_mlx_provider.mm)), and — NEW 2026-07-23 (`CLAIM-BACKEND-SEAM-S5-1`, work row `S5`) — **`vt-cpu-ref` (priority −1000, the portable reference tier)**: the CPU kernel installed LAZILY as a negative-priority fallback on a UNIFIED-MEMORY device's first `GetOp` miss, mirroring `custom_op.py:138 forward_native`, so a partial backend runs an op it lacks natively instead of throwing. Native always wins (priority); gated on `Backend::UnifiedMemory()` (a discrete GPU never gets it — a CPU kernel on true device memory is corruption); observable via `GetReferenceTierHits()` + a one-time loud stderr line | [test_op_provider.cpp](../tests/vt/test_op_provider.cpp) 11 cases / 47 assertions — deterministic selection under REVERSED registration order, name tie-break, duplicate rejection, capability predicate, caps re-resolution, decline-and-fall-back, stats, runtime disable; [test_metal_backend.cpp](../tests/vt/test_metal_backend.cpp) 9 cases / 108 assertions on the M4 with MLX ON, including MLX-vs-MSL-vs-CPU NMSE per op at real shapes and an end-to-end DECLINE; **[test_reference_tier.cpp](../tests/vt/test_reference_tier.cpp) (S5): discrete-device refusal + unified-device zero-native-kernel fallback correctness + native-wins + observability, hardware-free via a fake backend on `kXPU`.** Linux CPU 156/156; dgx regression set ALL UNCHANGED — anchor `tests/vt/test_op_provider.cpp:64` | [Metal/MLX reuse study §6](specs/metal-mlx-reuse-study.md); [accelerator-seam-audit §10](specs/accelerator-seam-audit.md); [drop-in kernel ABI](specs/dropin-kernel-abi.md) (the complementary ARGUMENT half) | `ACTIVE` — mechanism landed and gated with THREE provider kinds (`vt-native`, `mlx`, `vt-cpu-ref`); the CUDA/CPU/Vulkan vendor provider rows it was designed for are not yet populated (so the row is deliberately left open) | `CLAIM-BACKEND-ACCEL-PROVIDER-1` |

diff --git a/.agents/parity-ledger.md b/.agents/parity-ledger.md
@@ -887,3 +887,4 @@ Columns:
 | 2026-07-31 (`CLAIM-CPU-GCC12-WERROR-PORTABILITY`; maintenance, rows `QUANT-GGUF-KEEPQ-LOADER` + `KV-OFFLOAD`; lifecycle unchanged) | Removes two GCC 12 production-library `-Werror` blockers without suppressions: the GGUF prefault keeps the same one-byte-per-page volatile XOR but uses simple assignment, and the KV filesystem tier builds the identical `.<pid>.<counter>.tmp` suffix with append operations inside its thread-local initializer. No API, algorithm, default, CUDA, fixture, or golden change. | Behavior remains grounded in the accepted loader and KV-persistence leaf specs: llama.cpp mmap prefault intent and vLLM `tiering/fs/io.py` unique temporary-file publication. This is compiler portability, not a parity-surface change. | RED: GCC 12 failed first at `qwen3_5_gguf_weights.cpp:49` (`-Wvolatile`), then at `fs_io.cpp:66` (`-Wrestrict`). GREEN: production `vllm` and focused test targets build clean; focused CTest 2/2 (`test_gguf_keep_quant`, `test_kv_offload_fs`). Full all-target build is PARTIAL at 42% on unrelated test-only GCC 12 `-Wrestrict` diagnostics in `test_deepseek_v2_paged_engine.cpp` and `test_glm4_moe_lite_paged_engine.cpp`; no full-CTest claim. Benchmark NOT APPLICABLE. |
 | 2026-08-01 (`SERVE-CLI-CHAT` W0 contract spike; `CLAIM-SERVE-CLI-CHAT-SPIKE`; CPU-only records/spec) | Accepts `.agents/specs/cli-chat-complete.md`, corrects the inventory from “no direct commands” to the actual pinned `chat`/`complete` surface, and decomposes a dual-mode port: exact remote OpenAI HTTP/SSE commands plus preservation of the existing in-process invocation as a compatibility alias. No production, test, CMake, model, kernel, fixture, or generated file changes. | Pinned vLLM `5559679229`: command registration `vllm/entrypoints/cli/main.py:17-37,73-98`; model/auth resolution and stream shaping `vllm/entrypoints/cli/openai.py:30-100`; chat `:155-234`; complete `:237-312`. The local compatibility baseline is `examples/cli/main.cpp:1-207`. | CPU record/doc gates only; benchmark `NOT APPLICABLE`, `benchmark_binding=false`. Implementation remains absent and the row moves `INVENTORIED` -> `SPIKE`. W1-W5 name parse, transport, complete, chat, and packaging gates, including fake-server request/SSE transcript parity, Release `-Werror`, ASan+UBSan, and TSan. |
 | 2026-08-04 (`HARDEN-DETECTOR-LANES` PR #28 CI repair; `CLAIM-HARDEN-SANITIZER-DISK`; Ordino task `t-e19dc73f`; CPU-only, lifecycle unchanged; closing commit: this checkpoint) | Repairs the hosted ASan+UBSan build's filesystem exhaustion without weakening detector coverage. Sanitizer tests share one internal fully instrumented engine image instead of force-linking another static copy into every executable, and `-g1` retains file/line traces without full type/local-variable DWARF. CI enables the existing `VT_POOL_BYPASS=1` exact-allocation/real-free detector mode. The newly reachable leak survey removes a real minja `MacroNode` ownership cycle by weakly capturing the context that owns the callable. It also closes the two Nix-only suite portability gaps: resolve `true` and the active Python executable instead of assuming `/usr/bin`, and remove inherited `PYTHONHASHSEED` only from the unconfigured control suite. Default build linkage and packaged C ABI exports remain unchanged. | No vLLM behavioral counterpart: this is local build/test infrastructure plus a vendored minja lifetime repair. Anchors: `CMakeLists.txt`, `tests/CMakeLists.txt`, `.github/workflows/ci.yml`, `third_party/minja/minja.hpp`, `tests/tools/test_gdn_packed_component.py`, and `tests/tools/test_online_gate_client.py`. Remote root-cause evidence: GitHub run `30819266647`, job `91704728276`, 99 MiB free then `ld: No space left on device`. | **PASS.** GCC 15.2.0 ASan+UBSan full suite **331/331** under leak detection and pool bypass; TSan full suite **331/331** under pool bypass; affected plain GCC 15 `-Werror` suites **3/3**. ASan+UBSan tree **93 GiB -> 5.6 GiB** (about 94% smaller); TSan tree **1.9 GiB**. `benchmark_binding=false`, performance **NOT APPLICABLE**. Hosted PR confirmation is the next external gate; `continue-on-error` stays until that confirmation. |
+| 2026-08-08 (`row/KERNEL-FA2-GQA-SWAP`; `CLAIM-KERNEL-FA2-GQA-SWAP`; kernel `KERNEL-ATTN-FA2`; gated default-OFF, lifecycle unchanged) | Ports vLLM's FA2 `seqlenq_ngroups_swapped` decode optimization into the d128 varlen decode launcher (`LaunchDecodeVarlenFA2Bf16`, gate `VT_FA2_DECODE_GQA_SWAP`): the Qwen3-dense decode grid becomes `(batch, kv_heads)` not `(batch, hq)` — the ngroups query heads pack into seqlen_q, KV read once/group, presented WITHOUT a materialized transpose via kv-major-group-minor strides (a 1:1 mirror of the already-shipped d256 `LaunchDecodeFA2Bf16` swap). OFF path byte-identical to the prior plain-varlen reduction; ON is non-byte-exact only when num_splits>1 (split reduction order → near-tie, toward vLLM's own numerics). | Mirrors `flash-attention @ 2c839c33` `mha_fwd_kvcache` seqlenq_ngroups_swapped + `set_params_splitkv` and vLLM v0.25.0 `flash_attn.py flash_attn_varlen_func` decode (#47 measured vLLM's swapped grid `(1,6,16)` = batch×kv_heads vs ours `(1,3,64)` = batch×query_heads). The vendored `flash_fwd_kernel.h` `get_lse_tile`/combine already honor the flag in both the num_splits==1 direct-write and >1 combine paths (the d256 arm is the proof). | GB10 sm_121a CUDA 13.0: op RED-first test 280/280 (both GQA ratios × batch{1,2,4,8} × short+long ctx; `swap_launches==1` proves the grid engaged; swap-vs-plain near-tie; MHA-inert) — RED proven (wrong swapped stride → 26,528 violations); full binary 28/28·454,679 no regression; compute-sanitizer 0-err/0-leak; #44 MXFP4 e2e smoke swap-ON 3/3 deterministic TOKEN-EXACT + coherent, byte-identical to swap-OFF. `benchmark_binding=false` (c1-c8 x3 re-bench + default flip = recorded next step; #47 projects flash ~28%@c2 / ~55%@c8 of the gap). |