Repository navigation
Replies: 3 comments 1 reply
|
@max-krasnyansky @ORippler would love to hear your thoughts |
1 reply
|
@tdakhran The 2x prefill gain on the Qualcomm Hexagon v81 NPU stems from avoiding accumulator writes to apply per-block scales on systolic arrays. Storing scales as separate tensors drops usage to ~4.02 bpw, freeing bandwidth for larger contexts. Asymmetric offsets are required from the start; symmetric quantization forces outliers into a local optimum where the model optimizes for syntax rather than meaning. Int8 support should wait for fused GEMM ops to mature to avoid backend fragmentation. |
0 replies
|
On second thought, |
0 replies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Why?
Modern quality recovery methods (GPTQ, Hadamard rotation, QAT) make it possible to quantize model weights to 4 (and even 2) bits with a per-channel scale without significant quality loss. Calibrated per-channel checkpoints already exist: for example, llm-compressor / compressed-tensors W4A16 "channel" checkpoints that run natively in vLLM, and AngelSlim/HY-1.8B-2Bit. Today
convert_hf_to_gguf.pycan only dequantize such checkpoints and requantize them into block formats (#14810), which throws away the calibrated scales.A per-channel scale is also faster on hardware accelerators where applying per-block scales is expensive, e.g. Qualcomm NPU and Arm SME2: on a systolic array, copying accumulators out to apply each block's scale is expensive. For Arm SME2, see #19357.
It also saves memory: Q4_0 uses 4.5 bpw, while int4 with a per-channel f16 scale and offset uses ~4.02 bpw at K = 2048.
In our experiments with LFM3-1.7B on the Qualcomm Hexagon v81 NPU (Snapdragon 8 Elite Gen 5), per-channel int4 is on par in quality with Q4_0 QAD and more than 2x faster in prefill.
How?
#22042 is converging on GEMM ops taking optional scale tensors. Per-channel is the simplest case of that: a
[n_out]scale.The
*_s+ggml_mulpath already applies a scale after the matmul, andggml_mulbroadcasts a[n_out]tensor, so the scale needs no new graph op. The asymmetric offset can also be expressed with existing ops: forW = s·(q − z),y = s ⊙ (q·x) − (s·z) ⊗ sum(x).What is missing is a weight type that holds only the quantized levels, without per-block scales: e.g. plain int4 / int2 types. The scale and offset are stored as separate tensors next to the weight (
.scale,.offset). Backends that can fuse them into the matmul (NPU, SME2) do so; others fall back to the generic graph above.Open questions
MUL_MATwork from [GGML] Current NVFP4 support has risk of functional incorrectness due to unclear separation of concerns #22042?All reactions