Releases: avesed/vllm-ampere-optimized
Release list
v0.3 — flashampere backend + famp Marlin W4A8/W4A16 + DSpark/DFlash + auto-dispatch
vLLM Ampere-Optimized — Release v0.3
Ampere (sm_80 / sm_86) optimized vLLM v0.23 fork. A major release consolidating the flashampere attention backend, the vendored famp Marlin W4A8/W4A16 kernel, DSpark/DFlash speculative decoding, and per-architecture auto-dispatch.
Highlights
DSpark / DFlash speculative-decoding serving
- Serve with
--speculative-config '{"method":"dspark","model":<head>,"num_speculative_tokens":7}'(dflashalso supported) — block-diffusion drafter + rank-256 Markov head. - Validated lossless on Qwen3.6-27B-W4A16: dspark-ON GSM8K == base (score-identical → verify correct); accept-len ~5.2 on GSM8K-style prompts.
famp Marlin W4A8 / W4A16 (FampMarlinKernel)
- Vendored, from-source-compiled Marlin kernel (sm_80+86); auto-engaged for compressed-tensors W4A16 (no flags).
- Bit-exact vs stock
_CMarlin (torch.equal) across W4A16 bf16, GPTQ-sym, W4A8-int8 g32/g128. - W4A8 int8-act via
VLLM_MARLIN_INPUT_DTYPE=int8on a W4A16 checkpoint.
flashampere — unified Ampere attention backend (Backend.CUSTOM, opt-in)
VLLM_FLASHAMPERE=1: own vendored FA2 prefill (fp16-PV legs), XQA MTP spec-verify kernel, and Gemma4 hd512 full-attn support (relaxed-IsInvalid hd512 prefill + own XQA hd512 decode).- Composes with the default FA2 (byte-identical / 0 e2e when off).
model-architecture auto-dispatch (new)
flashampere/model_profiles.py— a fork-owned registry mapping model architecture → famp routing profile (attention flags + a per-arch marlin-validation doc). Runs before worker spawn; respects explicit user env. Adding a validated model = one registry entry.
int8-QK attention backend removed
- Net-negative on decode (O(L²) quant tax); removed along with
VLLM_INT8QK. The fp16-PV prefill legs are kept.
Validated (2×RTX3090, sm_86, NVLink)
- Qwen3.6-27B-W4A16 (dense): famp bit-exact, GSM8K coherent, dspark lossless + accept ~5.2, 12k-ctx needle retrieval intact (runner KV-reshape correct).
- Qwen3.6-35B-A3B-W4A16 (MoE): coherent (GSM8K 85–95%); quantized experts on stock Marlin MoE, GDN hybrid attention.
- gemma-4-12B-it (tp2): default TRITON coherent 5/5; famp hd512 prefill +8% available opt-in.
Known / WIP
- gemma prefill-famp / decode-TRITON hybrid: famp hd512 prefill is +8% but its XQA decode is −14% vs TRITON, so the default is TRITON. The per-phase split (famp prefill + TRITON decode, same layer) needs FA↔TRITON metadata bridging — WIP.
- famp W4A8 asym-weight (AWQ uint4+zp): by design famp accepts it while stock Marlin rejects it; output is garbage for such checkpoints (quantization-recipe limitation, not a kernel bug).
Image
docker pull ghcr.io/avesed/vllm-ampere-optimized:0.3
# or :latest
Ampere sm_80 + sm_86, CUDA 13.0. Fully from-source: famp_marlin.so + flashampere backend + vLLM _C all compiled (not an overlay).
v0.2 — MTP spec-decode long-context fix
Headline — MTP speculative-decode long-context fix
The native-MTP spec-decode verify attention now routes through FA2 fwd_kvcache
(FlashDecoding split-KV), captured into the FULL cudagraph, instead of flash_attn_varlen_func.
FA2 varlen with q>1 doesn't KV-split → occupancy-starved at num_kv_heads=4 / head_dim=256 →
MTP decode went net-negative beyond ~10k context. The fix recovers it to net-positive, and
MTP now beats no-MTP at every context length (Qwen3.5/3.6 hybrid GatedDeltaNet+full-attn):
| model | 16k | 32k |
|---|---|---|
| 9B (tp1) | +54% | +76% |
| 27B (tp2) | +60% | +106% |
Lossless (accept-len preserved, output identical; the verify keeps the full target head).
Gated by VLLM_FA2_KVCACHE_VERIFY (default on); falls back to varlen for fp8-KV / SWA / alibi /
softcap / sinks / FA3 / mixed batches. Standalone tests + patch file in patches/fa2_kvcache_verify/.
Also since v0.1
- Qwen3.6 27B / 35B-A3B quant recipes + runner — INT4 (AWQ+mse+g32) and INT8 (W8A8).
- DiffusionGemma int8-act path (validation + fake-quant tooling).
- Carried Ampere stack: W4A8 Marlin + int8-8row decode + int8-QK FlashInfer prefill.
Artifacts
- Image:
ghcr.io/avesed/vllm-ampere-optimized:v0.2(multi-arch sm_75..sm_120, vLLM v0.23.0 from source). - Wheel: attached below.
v0.1 — Ampere W4A8 (vLLM 0.23.0)
First tagged release of the Ampere fork — vLLM 0.23.0 + FlashInfer 0.6.12, built from source for sm_80 (A100) and sm_86 (RTX 3090 / A40 / A6000 / A10).
Docker: ghcr.io/avesed/vllm-ampere-optimized:v0.1 (also :latest)
Wheel (no Docker): pip install the attached .whl — needs torch 2.11 + CUDA 13 (covers sm_80+sm_86).
W4A8 (int4 weights + int8 activations) on Ampere — dense and MoE (patches 0001–0006) + int8-QK prefill. Enable int8 activations with --marlin-input-dtype int8. Validated throughput + accuracy in the README.