allow 256-block quantization for tensors whose width isn't a multiple of 256 #27037
lucaspirola
started this conversation in
Ideas
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
llama-quantize silently falls back whenever a tensor's leading dim isn't a multiple of the block size, which means all the block-256 quants (the K/IQ family) just don't apply to those tensors. There's an existing issue about a --no-fallback flag (#26616), but I think teh underlying ask is a bit bigger than that.
Nemotron-3-Nano-30B-A3B experts are 1856/2688/3712 wide — none divisable by 256. The result is that every published sub-4-bit GGUF of these models is stuck at ~16.5–18 GB no matter what quant you ask for. Even IQ1_M, which should be tiny, lands around ~17.9 GB, because the expert tensors end up as ~4.5–5.2 bpw block-32 types instead. On a 16 GB card that's literally the diffrence between serving 262k+ context and not fitting at all.
draft PR 708-145#33 ("Allow 256-block quantization on unaligned tensors") pads the last partial block inside the ggml-quants kernels, drops the nx % qk_k fallback in llama-quant, and pads the activation in the graph (build_mul_mat_padded). It's general — any uneven tensor quantizes to its block ceiling (1856 → 8×256 blocks), the padded tail is zeros (d=0), so it's exact for any model
All reactions