Skip to content

v0.1 — Ampere W4A8 (vLLM 0.23.0)

Choose a tag to compare

View all tags
@avesed avesed released this 20 Jun 09:51
v0.1
5ab1ee9
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

First tagged release of the Ampere fork — vLLM 0.23.0 + FlashInfer 0.6.12, built from source for sm_80 (A100) and sm_86 (RTX 3090 / A40 / A6000 / A10).

Docker: ghcr.io/avesed/vllm-ampere-optimized:v0.1 (also :latest)
Wheel (no Docker): pip install the attached .whl — needs torch 2.11 + CUDA 13 (covers sm_80+sm_86).

W4A8 (int4 weights + int8 activations) on Ampere — dense and MoE (patches 0001–0006) + int8-QK prefill. Enable int8 activations with --marlin-input-dtype int8. Validated throughput + accuracy in the README.

Assets 3
Loading