v0.2 — MTP spec-decode long-context fix
Headline — MTP speculative-decode long-context fix
The native-MTP spec-decode verify attention now routes through FA2 fwd_kvcache
(FlashDecoding split-KV), captured into the FULL cudagraph, instead of flash_attn_varlen_func.
FA2 varlen with q>1 doesn't KV-split → occupancy-starved at num_kv_heads=4 / head_dim=256 →
MTP decode went net-negative beyond ~10k context. The fix recovers it to net-positive, and
MTP now beats no-MTP at every context length (Qwen3.5/3.6 hybrid GatedDeltaNet+full-attn):
| model | 16k | 32k |
|---|---|---|
| 9B (tp1) | +54% | +76% |
| 27B (tp2) | +60% | +106% |
Lossless (accept-len preserved, output identical; the verify keeps the full target head).
Gated by VLLM_FA2_KVCACHE_VERIFY (default on); falls back to varlen for fp8-KV / SWA / alibi /
softcap / sinks / FA3 / mixed batches. Standalone tests + patch file in patches/fa2_kvcache_verify/.
Also since v0.1
- Qwen3.6 27B / 35B-A3B quant recipes + runner — INT4 (AWQ+mse+g32) and INT8 (W8A8).
- DiffusionGemma int8-act path (validation + fake-quant tooling).
- Carried Ampere stack: W4A8 Marlin + int8-8row decode + int8-QK FlashInfer prefill.
Artifacts
- Image:
ghcr.io/avesed/vllm-ampere-optimized:v0.2(multi-arch sm_75..sm_120, vLLM v0.23.0 from source). - Wheel: attached below.