Skip to content

v0.2 — MTP spec-decode long-context fix

Choose a tag to compare

@avesed avesed released this 24 Jun 04:22
353be14

Headline — MTP speculative-decode long-context fix

The native-MTP spec-decode verify attention now routes through FA2 fwd_kvcache
(FlashDecoding split-KV), captured into the FULL cudagraph, instead of flash_attn_varlen_func.
FA2 varlen with q>1 doesn't KV-split → occupancy-starved at num_kv_heads=4 / head_dim=256
MTP decode went net-negative beyond ~10k context. The fix recovers it to net-positive, and
MTP now beats no-MTP at every context length (Qwen3.5/3.6 hybrid GatedDeltaNet+full-attn):

model 16k 32k
9B (tp1) +54% +76%
27B (tp2) +60% +106%

Lossless (accept-len preserved, output identical; the verify keeps the full target head).
Gated by VLLM_FA2_KVCACHE_VERIFY (default on); falls back to varlen for fp8-KV / SWA / alibi /
softcap / sinks / FA3 / mixed batches. Standalone tests + patch file in patches/fa2_kvcache_verify/.

Also since v0.1

  • Qwen3.6 27B / 35B-A3B quant recipes + runner — INT4 (AWQ+mse+g32) and INT8 (W8A8).
  • DiffusionGemma int8-act path (validation + fake-quant tooling).
  • Carried Ampere stack: W4A8 Marlin + int8-8row decode + int8-QK FlashInfer prefill.

Artifacts

  • Image: ghcr.io/avesed/vllm-ampere-optimized:v0.2 (multi-arch sm_75..sm_120, vLLM v0.23.0 from source).
  • Wheel: attached below.