fix(nvfp4): unlock decode fast-path for prequant MoE (Qwen3.6 + Gemma-4)

[kekzl]

Summary

Unlocks the NVFP4 decode fast-path for Qwen3.6-35B-A3B-NVFP4 and Gemma-4-26B-A4B-it-NVFP4 SafeTensors models. Both now produce coherent output with CUDA Graphs captured in decode.

What was broken

• Qwen3.6-NVFP4 (already shipped on this branch as 615758a): three bugs blocked the decode fast-path — can_decode_fast whitelist missing NVFP4, no cache_moe_native_nvfp4 for SafeTensors per-expert NVFP4 layout, and missing per-layer free of per-expert allocations to fit in 32 GiB. tg256 was 8.34 → 142 tok/s.

• Gemma-4-NVFP4 (this PR's new commit 3901600): engine.cpp was unconditionally setting use_nvfp4_decode = 0 for any Gemma-4 model to dodge unsupported NVFP4 attention CUTLASS paths under per-layer head_dim (256 SWA / 512 global). That over-broad disable also skipped Phase 3-MoE (cache_moe_native_nvfp4), forcing the legacy FP16 fallback (sm_80 WMMA + per-layer D2H expert-offsets sync). Consequences: CUDA Graph capture IMA cascade at decode step 1, decode tg ~42 tok/s, output stuck on the user's prompt instead of an answer.

What changed

• src/runtime/engine.cpp (+21/-2): when model.config().is_nvfp4_prequant is true for Gemma-4, keep use_nvfp4_decode at its auto value. Phase 3a (Q*_K → NVFP4) and Phase 3b (NVFP4 → CUTLASS sm_120) iterate wcache_.nvfp4, which is empty for prequant models — they are no-ops. Only Phase 3-MoE does load-bearing work, and that's the single thing the disable was preventing. The "per-layer head_dim" caveat applies to attention CUTLASS kernels, not MoE expert caching.

• CLAUDE.md: refresh the CUDA-Graphs note. NVFP4-prequant MoE models capture cleanly post-fix; only legacy GGUF MoE decode (which still does per-layer D2H expert-offsets sync) remains graph-incompatible. Prefill is never captured (variable n).

Result on Gemma-4-26B-A4B-it-NVFP4 (RTX 5090, default config)

Metric	Before	After
Phase-4 wcache nvfp4_moe	0	90 (30 layers × 3 projections)
CUDA Graph capture step 1	IMA cascade → fallback	152 PDL edges, AsyncGraphLoop on
MoE prefill: legacy fallback log	every iter	only n>1 prefill batch path
Decode tg	42.86 tok/s	166 tok/s (~3.9× speedup)
Output	<channel|>thought\nThe user is asking… (stuck)	<channel|>thought\n…The capital of France is **Paris**.

Cross-model regression check

• Qwen3.6-35B-A3B-NVFP4: tg = 142–155 tok/s, nvfp4_moe=120, output coherent — unchanged ✓
• Gemma-4 GGUF Q8_0 / Q4_K_M: untouched, runs through the non-prequant else branch — preserved ✓
• Mistral-Small-3.2-NVFP4 (dense, no MoE): not on this code path — unaffected ✓

Test plan

• Greedy decode "The capital of France is" → contains "Paris" (Qwen3.6, 16 tokens)
• Greedy decode "What is the capital of France?" → contains "Paris" (Gemma-4, 128 tokens; needs CoT budget)
• Sampled coherence T=0.7 top_p=0.9 256 tokens (both models, Rayleigh-scattering / sky-blue prompt) — coherent prose ✓
• Long-context smoke ~430 prefill tokens — no NVFP4 long-context drift ✓
• Perf sweep at 128 / 1024 / 4096 ctx — both models above plan baseline (Qwen3.6 ≥ 117 tok/s, Gemma-4 ≥ 34 tok/s) ✓
• CUDA Graph capture confirmed via logs (ConditionalRunner, AsyncGraphLoop) for both ✓

Pre-push verify-fast hook flagged a 3% decode regression on Qwen3-8B Q8_0 — unrelated dense GGUF model, no code path touched by this PR; almost certainly cuBLAS-autotune variance from a warm GPU after extended NVFP4 testing. Bypassed with --no-verify per author request after manual cross-check.

🤖 Generated with Claude Code