Repository navigation
Replies: 2 comments
|
I don't think the cited note identifies this as SmoothQuant. SmoothQuant is a different operation: it applies an equivalent channel-wise rescaling to move activation outliers into the weights (paper). Here, Unsloth describes a representation mismatch: the QAT Q4_0 path uses BF16 scales, while llama.cpp Q4_0 stores F16 scales, so naive conversion does not preserve the QAT lattice. The Gemma 4 QAT docs say their Dynamic method makes the llama.cpp-compatible quant agree better with that lattice; they also note Q6_K wasn't needed for embeddings. They don't publish the exact scale-conversion trick behind the 99.96% figure, so I wouldn't call it SmoothQuant-style weight-scale smoothing without more detail from the authors. Unsloth's QAT analysis |
|
I wouldn't call it SmoothQuant. Nothing gets smoothed. The The hack is repacking, not re-quantizing: keep the codes, write the BF16 scale as the F16 scale (exact while it fits F16 range), fix nibble order/transposes, and for blocks where — |
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Unsloth's post says
In SmoothQuant, the problem was with representing the activations, so they "smooth" the activation outliers by offline migrating the quantization difficulty from activations to weights with a mathematically equivalent transformation.

However BF16's range exceeds F16, so the same big values cannot be stored.
Is the "hack" "smoothing" the weight scales?
All reactions