Skip to content

v0.3 — flashampere backend + famp Marlin W4A8/W4A16 + DSpark/DFlash + auto-dispatch

Latest
Latest

Choose a tag to compare

View all tags
@avesed avesed released this 15 Jul 15:47
v0.3
dba50fe
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

vLLM Ampere-Optimized — Release v0.3

Ampere (sm_80 / sm_86) optimized vLLM v0.23 fork. A major release consolidating the flashampere attention backend, the vendored famp Marlin W4A8/W4A16 kernel, DSpark/DFlash speculative decoding, and per-architecture auto-dispatch.

Highlights

DSpark / DFlash speculative-decoding serving

  • Serve with --speculative-config '{"method":"dspark","model":<head>,"num_speculative_tokens":7}' (dflash also supported) — block-diffusion drafter + rank-256 Markov head.
  • Validated lossless on Qwen3.6-27B-W4A16: dspark-ON GSM8K == base (score-identical → verify correct); accept-len ~5.2 on GSM8K-style prompts.

famp Marlin W4A8 / W4A16 (FampMarlinKernel)

  • Vendored, from-source-compiled Marlin kernel (sm_80+86); auto-engaged for compressed-tensors W4A16 (no flags).
  • Bit-exact vs stock _C Marlin (torch.equal) across W4A16 bf16, GPTQ-sym, W4A8-int8 g32/g128.
  • W4A8 int8-act via VLLM_MARLIN_INPUT_DTYPE=int8 on a W4A16 checkpoint.

flashampere — unified Ampere attention backend (Backend.CUSTOM, opt-in)

  • VLLM_FLASHAMPERE=1: own vendored FA2 prefill (fp16-PV legs), XQA MTP spec-verify kernel, and Gemma4 hd512 full-attn support (relaxed-IsInvalid hd512 prefill + own XQA hd512 decode).
  • Composes with the default FA2 (byte-identical / 0 e2e when off).

model-architecture auto-dispatch (new)

  • flashampere/model_profiles.py — a fork-owned registry mapping model architecture → famp routing profile (attention flags + a per-arch marlin-validation doc). Runs before worker spawn; respects explicit user env. Adding a validated model = one registry entry.

int8-QK attention backend removed

  • Net-negative on decode (O(L²) quant tax); removed along with VLLM_INT8QK. The fp16-PV prefill legs are kept.

Validated (2×RTX3090, sm_86, NVLink)

  • Qwen3.6-27B-W4A16 (dense): famp bit-exact, GSM8K coherent, dspark lossless + accept ~5.2, 12k-ctx needle retrieval intact (runner KV-reshape correct).
  • Qwen3.6-35B-A3B-W4A16 (MoE): coherent (GSM8K 85–95%); quantized experts on stock Marlin MoE, GDN hybrid attention.
  • gemma-4-12B-it (tp2): default TRITON coherent 5/5; famp hd512 prefill +8% available opt-in.

Known / WIP

  • gemma prefill-famp / decode-TRITON hybrid: famp hd512 prefill is +8% but its XQA decode is −14% vs TRITON, so the default is TRITON. The per-phase split (famp prefill + TRITON decode, same layer) needs FA↔TRITON metadata bridging — WIP.
  • famp W4A8 asym-weight (AWQ uint4+zp): by design famp accepts it while stock Marlin rejects it; output is garbage for such checkpoints (quantization-recipe limitation, not a kernel bug).

Image

docker pull ghcr.io/avesed/vllm-ampere-optimized:0.3
# or :latest

Ampere sm_80 + sm_86, CUDA 13.0. Fully from-source: famp_marlin.so + flashampere backend + vLLM _C all compiled (not an overlay).

Assets 3
Loading