MTP / qwen3_next_mtp self-speculation for Qwen3.6 hybrid GDN

[pekkah]

Summary

Add Multi-Token Prediction (MTP / NEXTN) self-speculative decoding for the
hybrid GDN+attn+MoE family so Qwen3.6-27B-MTP (and any later sibling that
ships MTP heads) decodes at the same ~1.5–2× the dense-model serving stacks
already get. The 35B-A3B variant in-tree doesn't ship MTP heads — confirmed
via list-tensors, only one output.weight, no mtp.*/nextn.* tensors —
but unsloth/Qwen3.6-27B-MTP-GGUF does, with the MTP head trained alongside
the main model. llama.cpp drives it as --spec-type draft-mtp --spec-draft-n-max 2; vLLM's config method name is qwen3_next_mtp; SGLang
uses --speculative-algo NEXTN.

Quoted speedup: ~1.5–2× decode, no accuracy loss. For the current CUDA
hybrid path this would be the largest single win available — bigger than
CUDA Graphs or NVRTC kernel fusion, both of which only reduce host-side
launch overhead (~3–5 ms/token), while MTP reduces the number of forward
passes per accepted token.

Why we can't just turn this on today

HybridGdnForwardPass.SupportsPartialRewind => false
(HybridGdnForwardPass.cs:395) and the matching CUDA path
(CudaHybridGdnForwardPass.cs:692). The existing SpeculativeDecoder is
gated on SupportsPartialRewind and throws otherwise
(SpeculativeDecoder.cs:43-50). GDN destructively updates a
[num_v_heads × head_dim × head_dim] scan state per layer (~16 MiB × 30
GDN layers ≈ 480 MiB), so any verify-and-rollback scheme has to be able to
restore that state at per-token granularity, not just at the end-of-decode
boundary that the issue #21 snapshot covers (commit 7633f8a).

Work breakdown

1. GGUF parser — register qwen3_next_mtp (or whatever metadata key
   the 27B-MTP GGUF uses) and load the MTP head tensors. Pull the GGUF,
   run list-tensors, and document the exact tensor naming under
   docs/qwen35moe-plan.md before writing code.

2. Per-token GDN state snapshot/restore. End-of-decode CaptureSnapshot
   already exists for prefix-cache reuse, but on the CUDA hybrid path it
   downloads ~480 MiB host-side every snapshot. For per-token use we need
   either:

   • an on-device snapshot ring (keep N copies in VRAM, swap pointers on
     accept/reject), or
   • a delta-state mechanism that records only the rank-1 update each
     step and replays the rejected suffix in reverse.
     The ring is simpler; cost is N × 480 MiB VRAM. At 12 GB this is fine for
     small N (N=2 → ~1 GB).

3. Batched prefill for GDN. Verify-pass runs N candidate tokens through
   one forward. HybridGdnForwardPass.Prefill walks tokens sequentially
   today (explicit v1 limitation, HybridGdnForwardPass.cs:325). Need either
   a real batched recurrence or a "verify-only" path that runs the existing
   recurrence N times into a scratch state, comparing to the main path.
   For attention layers, batching is straightforward; for GDN the rank-1
   update is per-token so the "verify in scratch" approach is fine.

4. MTP forward + verify wiring. New engine path or a flag on
   InferenceEngine. Existing SpeculativeDecoder is built for a separate
   draft model — MTP is single-model, so this is a sibling implementation,
   not a refactor.

Acceptance criteria

• list-tensors on the 27B-MTP GGUF documented in the design doc.
• CPU HybridGdnForwardPass decodes the 27B-MTP variant with MTP
  enabled and produces byte-identical greedy output vs llama.cpp
  --spec-type draft-mtp for ≥60-token decode (one of the standard
  benchmark prompts).
• CUDA hybrid path lands the same parity check.
• bench-textgen.ps1 row added to bench-all.ps1 covering
  qwen36-27b-mtp-cuda; README perf table updated.
• Measured decode speedup vs the same model without MTP (run with
  --spec-draft-n-max 0 or equivalent) is ≥ 1.3× — below that the
  complexity isn't worth the maintenance cost.
• Mirrors prior practice: a SHARPI_DISABLE_MTP=1 env switch for
  bisecting parity regressions, like the existing
  SHARPI_BYPASS_GDN/SHARPI_CPU_GDN knobs.

Out of scope

• Adding MTP heads to the 35B-A3B-UD weights (this would be a training
  project, not an inference change).
• Multi-step draft (N > 2). Start with N=2 to match llama.cpp's default;
  raise once acceptance rates are measured.
• Vulkan hybrid support. The qwen35moe Vulkan hybrid path is already a
  known broken row (README ⚠ note); fix that first under its own issue
  before extending it to MTP.

References

• Model card: https://huggingface.co/unsloth/Qwen3.6-27B-MTP-GGUF
• Prior context in this repo: memory/feedback_qwen36_perf_attempts.md
  in the local agent memory (not checked in).
• llama.cpp PR/docs on draft-mtp for the exact verify-and-accept algorithm.

[pekkah]

Substantial progress 2026-05-25 (uncommitted, local working tree)

Phase 11 of docs/qwen35moe-plan.md now covers this issue in detail with empirical tensor inventory, GGUF metadata, and measured baselines.

Acceptance criteria status

• list-tensors on the 27B-MTP GGUF documented in the design doc. Full inventory of blk.0 (GDN), blk.64 (MTP head with nextn.* tensors), and all qwen35.* metadata keys in Phase 11 of docs/qwen35moe-plan.md.
• CPU HybridGdnForwardPass decodes the 27B-MTP variant with MTP enabled and produces byte-identical greedy output vs llama.cpp.
• CUDA hybrid path lands the same parity check.
• bench-textgen.ps1 row added (filed as a separate issue).
• Measured decode speedup ≥1.3× vs MTP-disabled baseline.
• SHARPI_DISABLE_MTP=1 env switch.

Structural blockers — all cleared

1. ✅ CudaHybridGdnForwardPass + HybridGdnForwardPass now accept dense (non-MoE) hybrid GDN models. Both throws at construction (CudaHybridGdnForwardPass.cs:275-276, HybridGdnForwardPass.cs:184-187) relaxed; conditional MoE/dense tensor load + dispatch + dispose throughout. New DenseFfn (CPU) and CpuDenseFfn (CUDA hybrid) methods.
2. ✅ qwen35 arch registered in ModelGraph.cs:179 (NEOX-rope list). Hybrid detection auto-trips via existing tensor probe in GgufModel.cs:132-136.
3. ✅ MTP head accounting added. ModelHyperparams.NumMtpLayers (new field); main NumLayers = block_count - nextn_predict_layers, so the main model loop stops at the right index and MTP block (blk.64 for 27B) is loaded only by the MTP head path.

Surprises vs the original issue text

• MTP tensor prefix is nextn. in the GGUF (not mtp. as PR #20533 internal canonical suggests). On-disk names: blk.64.nextn.eh_proj.weight [Q8_0 10240×5120], nextn.enorm, nextn.hnorm, nextn.shared_head_norm.
• GDN tensor naming is IDENTICAL between qwen35 (27B) and qwen35moe (35B-A3B). Same attn_qkv / attn_gate / ssm_alpha / ssm_beta / ssm_a / ssm_dt.bias / ssm_conv1d / ssm_norm / ssm_out. GDN kernels are directly reusable — only difference is dense FFN vs MoE FFN.
• GGUF arch string is qwen35 (not qwen3_5 as transformers config says, not qwen35moe). Block count is 65 = 64 main + 1 MTP head.
• The MTP head IS a complete standard attention+FFN block (block 64) plus 4 nextn.* norms/proj. Same arch as a regular full-attention layer.

Measured 27B-MTP baselines (RTX 4070 Ti 12 GB, no MTP yet)

Config	Decode t/s
CPU only	3.1
CUDA hybrid, all FFN on CPU	4.0
CUDA hybrid, 21 of 64 FFN-on-GPU (exact-size alloc)	6.3

The 6.3 t/s number includes a separate VRAM unlock (see separate issue I'm filing for the exact: true weight upload story — GpuBufferPool.RoundUp was rounding to power-of-2 and wasting ~3 GiB across weight uploads).

Remaining work for this issue

• MTP head forward path (MtpHead.cs proposed) — takes last-layer hidden + next-token embedding → concat via eh_proj → standard attention+FFN → shared lm_head → logits for p+1.
• MtpDecoder.cs — sibling to SpeculativeDecoder.cs (cannot reuse: that one requires SupportsPartialRewind on both passes; MTP is single-model).
• Per-token GDN state snapshot/restore (currently end-of-decode only). For draft-N=2 a single snapshot-before-draft + restore-on-reject suffices; no ring needed yet.
• Q8_0 CUDA matvec for nextn.eh_proj (currently dequant-to-F32 at load via UploadWeight fallback path — fine for memory, but a native Q8_0 kernel would be cleaner).

[pekkah]

CPU MTP foundation landed in commit a28bac4

Acceptance criteria status

• `list-tensors` on the 27B-MTP GGUF documented in the design doc.
• CPU `HybridGdnForwardPass` decodes the 27B-MTP variant with MTP enabled. Smoke test `HybridGdnForwardPass_Qwen35Mtp_MtpHeadProducesWellFormedLogits` asserts finite + non-degenerate logits. Greedy parity vs llama.cpp `--spec-type draft-mtp` is tracked separately (MTP greedy parity vs llama.cpp --spec-type draft-mtp #31).
• CUDA hybrid path lands the same parity check. Tracked as CUDA hybrid MTP — mirror CPU MTP foundation on CudaHybridGdnForwardPass #29.
• `bench-textgen.ps1` row added. Filed earlier as Add bench rows for qwen36-27b-mtp (cpu, cuda-hybrid) and establish perf baseline #28.
• Measured decode speedup >= 1.3x vs MTP-disabled baseline. Tracked as Batched main verify + per-token GDN snapshot ring — realize the MTP >=1.3x speedup #30 — sequential N=1 inherently cannot deliver this; needs batched verify + the per-token GDN snapshot ring.
• `SHARPI_DISABLE_MTP=1` env switch. Wired in `InferenceEngine.GenerateChunksAsync`. `SHARPI_TRACE_MTP=1` prints acceptance rate.

What's in commit a28bac4

• `IForwardPass`: added `HasMtpHead`, `LastHidden`, `MtpForward`, `MtpResetCache`, `MtpTruncateTo` (defaults safe on non-MTP backends).
• `HybridGdnForwardPass` (CPU): loads the 4 `nextn.*` tensors + the standard transformer block at `blk.{NumLayers}`. Q8_0 `nextn.eh_proj` dequant'd to F32 at load (~200 MiB). Standalone `PagedKvCache(numLayers=1)` for MTP attention. `_lastHidden` snapshot before output norm.
• `MtpDecoder` (new): sequential N=1 verify-and-accept loop. Sibling to `SpeculativeDecoder` (can't reuse — GDN's `SupportsPartialRewind == false`).
• `InferenceEngine`: greedy + non-thinking decode routes through `MtpDecoder` when `HasMtpHead`. `SHARPI_DISABLE_MTP=1` bypass.
• All 227 forward-pass tests pass; no regressions.

Why sequential N=1 doesn't give speedup

Per iter: 1 MTP draft + 1 main verify (`Forward(t1, P)`) + 1 main commit (`Forward(t2, P+1)`) + 1 MTP commit = 2 main forwards per 2 emitted tokens. Same as baseline. The MTP head is correctly producing parallel predictions, but every emitted token still gets verified by an actual main forward.

To realize the >=1.3x speedup, we need a single batched main forward that processes (t1, t2_draft) at (P, P+1) in one call AND a per-token GDN snapshot ring to roll back the rejected case. Both filed as #30.

Follow-up issues filed

• CUDA hybrid MTP — mirror CPU MTP foundation on CudaHybridGdnForwardPass #29 — CUDA hybrid MTP (mirror of CPU foundation on `CudaHybridGdnForwardPass`).
• Batched main verify + per-token GDN snapshot ring — realize the MTP >=1.3x speedup #30 — Batched main verify + per-token GDN snapshot ring (the speedup work).
• MTP greedy parity vs llama.cpp --spec-type draft-mtp #31 — Greedy parity vs llama.cpp `--spec-type draft-mtp`.
• Wire MtpDecoder into RunCommand (CLI) decode loop + thinking-mode coexistence #32 — Wire `MtpDecoder` into `RunCommand` (CLI) + thinking-mode coexistence.

Suggested next milestone

Land #29 + #31 together (CUDA hybrid MTP + parity vs llama.cpp on both CPU and CUDA hybrid paths). That gives us:

• Correctness verified end-to-end on the actual deployment target (4070 Ti 12 GB).
• Parity-correct GPU MTP at the existing 6.3 t/s baseline (same throughput, but now testable as a self-contained correctness milestone).

Then #30 layers on top to extract the speedup.