int8 activations (W4A8) are now a one-flag serve-time knob for the Qwen3.6-27B fast tier — ~+50% prefill, quality-tied #733
Replies: 1 comment 2 replies
-
|
@noonghunna Can I buy you a beer? (Or a beverage of your choice). The good work you're doing here, is lifting us local A.I folks up high. |
Beta Was this translation helpful? Give feedback.
-
|
thank you so much! coffee would nice if we ever meet up someday. 😸 |
Beta Was this translation helpful? Give feedback.
-
INT4 Weights / INT8 Activations / FP8 KV Cache (Hybrid)
The hybrid config nearly matches INT8's prefill speed (~94% of INT8 at 10K, ~97% at 90K) while reclaiming ~80% of INT4's decode advantage. It's the best overall configuration for speed (except for possibly the FP8 KV-Cache that all of them use). Comparison vs. Baselines
Custom Settings Applied for all Tests:
|
Beta Was this translation helpful? Give feedback.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
-
Short version: the fast-tier Qwen slugs can now run int8 activations on top of the int4 weights (the "W4A8" combo) via a single serve-time flag. On our 2×3090 reference rig it's ~+50% prefill, decode unchanged, and an 8-pack that ties the default on both reasoning legs — so it's a free prefill win for prefill-heavy work (big agent prompts, RAG), with no quality cost we can measure.
It ships off by default and inert — the enabling patches are mounted on the composes but do literally nothing until you set the flag, so your normal runs are byte-for-byte unchanged.
Measured (reference 2×3090,
vllm/dual, MTP n=3, single-variable A/B — only the env var changes)Single-card (
vllm/minimal, no spec-decode): +110% prefill. Decode is flat because it's batch-1 weight-bandwidth-bound — int8 activations buy prefill, full stop.Which slugs
Wired on the dense AutoRound slugs, where it's valid:
vllm/dual(=vllm/qwen-27b-dual-fast),vllm/minimal,vllm/qwen-27b-multi-fast.Not available on the others, and they'll tell you why if you try:
vllm/qwen-27b-dual-max) →Marlin W8A8 not supported(already 8-bit weights).NVFP4 weight + INT8/FP8 activation not supported(FP4 is a float format, can't widen losslessly to int8).# TODOin the kernel); tracked.How to use it — command line
That's it — the shipped AutoRound checkpoint works out of the box. Drop the env var for the normal W4A16 path.
How to use it — the c3 cockpit
Highlight a fast-tier slug, press ⏎ to open the serve card, then
[a]to toggle int8 acts ON, and Start. The toggle only appears for slugs where it's valid, and the card shows the ON/off state before you commit.Under the hood (for the curious)
VLLM_MARLIN_INPUT_DTYPE=int8is a stock vLLM toggle — the Marlin kernel dynamic-quantizes activations per-token to int8 and runs them on Ampere's INT8 tensor cores, while the int4 weights widen to int8 losslessly (so only the activations are quantized). Two upstream bugs blocked it on AutoRound checkpoints specifically — the toggle was silently dropped on one path, and negative group scales (which AutoRound emits ~50% of, harmlessly for W4A16) corrupted the int8 path. Both are now filed upstream (vllm#48904, vllm#48905); we ship the fixes as small, inert-until-enabled overlays until they land in a release.Credit
This one had a lot of hands before we typed a line: @chriskerley78910 called the shot and drove it through every dead-end in #609; @Ezgera's pointer to the avesed/vllm-ampere-optimized fork cracked the mechanism open (avesed's patch series + notes were the map); @3dluvr corrected the "AutoRound has no W4A8 scheme" dead-end (it's a runtime flag, not a scheme); and klement's NVFP4 forensics on #662 sharpened where the format lines actually fall. Thank you all. 🙏
Prefill-heavy 3090 owners: this is a genuinely free ~1.5× on your agent prompts. Give it a spin and report numbers from your rig.
Beta Was this translation helpful? Give feedback.
All reactions