-
Notifications
You must be signed in to change notification settings - Fork 0
Quantization
Quantization represents weights and activations in fewer bits than the format the model was trained in. It is the highest-leverage optimization available in LLM inference, because decode is bandwidth bound — halving the bytes per weight nearly halves decode time.
It is also the optimization most likely to silently break accuracy, which is why it needs its own verification discipline.
- Reduce memory traffic — fewer bytes per parameter read
- Reduce memory capacity — fit larger models in the same memory
- Increase effective throughput — narrower arithmetic means more operations per unit area and energy
- Preserve accuracy — within an agreed, measured tolerance
- Stay predictable — the same model must quantize the same way every time
flowchart LR
A[Model Frontend] --> B[Quantization]
B --> C[Graph Compiler]
C --> D[Kernel Library]
B -.->|accuracy regression| A
style B fill:#2d6a9f,color:#fff
Quantization maps a continuous range of real values onto a small set of integers:
q = round(x / s) + z # quantize
x̂ = s · (q − z) # dequantize
where s is the scale and z the zero point. The error x − x̂ is quantization noise, and everything in this field is about controlling where that noise lands.
| Choice | Options | Tradeoff |
|---|---|---|
| Symmetric vs. asymmetric |
z = 0 or z ≠ 0
|
Symmetric is cheaper in hardware; asymmetric fits skewed distributions better |
| Granularity | Per-tensor, per-channel, per-group, per-block | Finer = more accurate, more scale storage and overhead |
| Static vs. dynamic | Scales fixed at compile time or computed at runtime | Static is faster; dynamic adapts to actual activation ranges |
Why granularity matters so much: a single scale per tensor must cover the largest magnitude in it. One outlier forces a coarse scale, and every ordinary value loses precision. Per-channel or per-group scaling contains the damage to the group holding the outlier — which is why finer granularity buys accuracy so reliably.
| Format | Bits | Character |
|---|---|---|
| FP32 | 32 | Training reference; rarely used for inference |
| FP16 / BF16 | 16 | Wide dynamic range; BF16 trades mantissa for exponent |
| INT8 | 8 | Well-supported, usually near-lossless with care |
| FP8 | 8 | Better dynamic range than INT8 at the same width |
| INT4 | 4 | Big win, needs group-wise scaling and careful handling |
| Sub-4-bit | 2–3 | Aggressive; usually requires specialized methods |
Weights and activations need not share a format. A very common configuration is low-bit weights with higher-precision activations, because weights dominate memory traffic in decode while activations are comparatively few but sensitive.
Transformer activations contain systematic outliers — a small number of channels with magnitudes far larger than the rest, appearing consistently in the same channels across inputs. These wreck per-tensor scaling.
Established responses:
| Technique | Idea |
|---|---|
| Per-channel / group scaling | Isolate outlier channels so they don't set the scale for everyone |
| Outlier isolation | Keep a small set of outlier channels in high precision, quantize the rest |
| Equivalent transformation | Mathematically shift difficulty from activations into weights, which quantize more easily |
| Rotation | Apply an orthogonal transform that spreads outlier energy across channels |
| Clipping with search | Choose a clipping range that minimizes end-to-end error rather than max error |
The unifying insight: the problem is not precision itself, it's distribution shape. Techniques that reshape the distribution before quantizing outperform techniques that simply add bits.
| Approach | What it needs | When to use |
|---|---|---|
| Post-training quantization (PTQ) | A small calibration dataset | Default; fast, no retraining |
| Calibration-free | Nothing | Weight-only, coarse settings |
| Error-compensating PTQ | Calibration data + solve per layer | INT4 and below |
| Quantization-aware training (QAT) | Full training pipeline | When PTQ can't reach the accuracy target |
Calibration data matters more than people expect. Scales derived from a calibration set that doesn't resemble production traffic will be wrong in production. Use real, representative samples, and enough of them that the range estimates are stable.
Common calibration objectives: min/max (simple, outlier-sensitive), percentile clipping (robust), and error-minimizing search (best, slowest).
In long-context decode, the KV cache can exceed the model weights in size, and it is read in full every single step. Quantizing it is often a larger win than quantizing weights.
Considerations specific to the cache:
- Keys and values behave differently. Keys typically have stronger per-channel outlier structure than values, and often need finer granularity.
- It's written incrementally. Scales must be computable as tokens arrive, not over a complete tensor.
- Errors compound. A quantized cache entry is re-read for every subsequent token, so its error influences the whole remaining generation.
- Group along the right axis. Per-channel and per-token grouping give very different results; this is worth measuring rather than assuming.
Quantization is the stage most likely to be "working" while being wrong. Verification needs layers:
| Level | Check | Catches |
|---|---|---|
| Tensor | Per-layer error vs. reference | Which layer degraded |
| Logit | Output distribution divergence | Subtle distortion invisible in text |
| Task | Standard benchmark scores | Real capability loss |
| Generation | Long-output quality | Compounding errors that short tests miss |
Perplexity alone is not sufficient. A model can hold near-identical perplexity while losing specific capabilities — the damage concentrates in cases the average doesn't see. Always include task-level evaluation and long-generation testing.
| Metric | Why it matters |
|---|---|
| Compression ratio | Bytes vs. baseline; predicts decode speedup |
| Per-layer error | Localizes accuracy damage |
| Perplexity delta | Cheap regression signal, insufficient alone |
| Task accuracy delta | The number that actually matters |
| Outlier fraction | How much stays in high precision |
| Scale storage overhead | Fine granularity has a real cost |
| Dequantization overhead | Cycles spent unpacking |
| Problem | Root cause | Fix |
|---|---|---|
| Large accuracy drop at INT8 | Activation outliers, per-tensor scaling | Per-channel scaling; outlier isolation |
| Good perplexity, bad task scores | Averaged metric hides concentrated damage | Evaluate on tasks, not just perplexity |
| Fine on short outputs, degrades on long | Compounding KV cache error | Verify with long generations; finer cache granularity |
| Works in calibration, fails in production | Unrepresentative calibration data | Recalibrate on real traffic |
| No speedup despite fewer bits | Dequantization overhead eats the gain | Fuse dequantization into the consuming kernel |
| Nondeterministic results | Dynamic scales varying run to run | Pin to static scales where determinism is required |
| First and last layers degrade badly | These layers are unusually sensitive | Keep them at higher precision |
- Quantize weights first. It's the biggest win and the safest.
- Keep sensitive layers wide. Embeddings, the output projection, and normalization rarely justify aggressive quantization.
- Prefer finer granularity before more bits. Per-group INT4 often beats per-tensor INT8 in both size and accuracy.
- Fuse dequantization into the consumer. A separate dequantize pass reintroduces the traffic you just eliminated.
- Use real calibration data, and enough of it.
- Test long generations. Short-prompt testing hides cache error accumulation.
- Make quantization reproducible. Same model plus same config must give bit-identical output, or debugging becomes impossible.
- Measure, don't assume. Which technique wins is model-dependent and changes with architecture.
See also: Model Frontend · Graph Compiler · Memory Management · Kernel Library · Inference Software