Quantization

Quantization

On the NPU: 4-bit

The NPU multiplies int8 x int4 natively and has no sub-4-bit mode (see NPU). Its container format is Q4NX: a grid of 32 x 256-weight tiles, one chunk each. There are three chunk kinds, fully decoded (1bit-MONSTER npu-infer, the Qwen3.5-4B work; in the engine since PR #33, npu/q4nx.h):

Chunk	Kind	Weight	Seen in
5120 B	q4_1: bf16 scale and zero per 32 columns	code * scale + zero	Qwen3, Llama, Gemma, Phi
4736 B	Q4_K: u8 scale and min per 32 columns, bf16 S, M per row	S * scale * code + M * min	Qwen3.5-4B projections
8704 B	Q8: bf16 scale per 32 columns, int8 codes	d * code	Qwen3.5-4B lm_head, embedding

A tensor's chunk kind is its shape's last dimension. Q4_K repacks into q4_1 exactly apart from rounding two bf16 products (max error 0.00032 on a real tile, within the rounding bound), so the fast lane and dx run it unchanged; Q8 needs an 8-bit path.

• Ternary and 1-bit models convert to Q4NX exactly ({-1,0,+1} x s = nibbles 0/1/2 with min -s, step s). Bonsai-8B in Q4NX matches its CPU reference on the NPU (corr 0.9984).
• Per-model sweet spots are still being measured: pi goal mudeyyef-y97olm, paused. Candidates are Q4NX, q8, a q8 head, q8 attention, and a mix per tensor role taken from the model's Unsloth Dynamic file. Each is judged by KL <= 0.01 nats and perplexity within 3% of fp32.

On the GPU (llama.cpp: Vulkan, HRX): Unsloth Dynamic

An Unsloth Dynamic (UD) GGUF is an ordinary GGUF. "UD" names a recipe that picks a standard ggml type per tensor. For example, UD-Q4_K_XL keeps attn_v, ffn_down and the embedding at Q6_K, uses Q5_K/Q4_K elsewhere, and IQ4_XS on about 5 layers. Every UD file contains some IQ types, which llama.cpp decodes.

Qwen3-0.6B, KL against BF16 over 20 x 512 tokens (measured on the previous build, main b11258e), and llama-bench tg128 on the current build (llama.cpp f1a0aca, hrx-system 51b1739, 2026-09-23):

File	Size	PPL (BF16 10.47)	Mean KLD	Same top token	Vulkan0	HRX0
Q4_K_M	397 MB	10.82	0.063	89.1%	349 tok/s	314
UD-Q4_K_XL	405 MB	10.68	0.052	89.8%	324	293
UD-Q2_K_XL	302 MB	16.23	0.662	66.3%	388	fails
UD-IQ2_M	269 MB	20.44	0.857	64.0%	396	fails
UD-IQ1_S	215 MB	10633	7.41	6.0%	436	fails

The sub-4-bit UD files fail on HRX0 because ggml-hrx has no IQ3_XXS matmul (see GPU HRX and Vulkan).

• At 4 bits, UD-Q4_K_XL is better than Q4_K_M: its KL is 17% lower for 2% more bytes, at 7% lower Vulkan decode speed.
• The UD 1- and 2-bit recipes are for very large models. At 0.6B they lose too much.

Source: 1bit-MONSTER docs/wiki/unsloth-dynamic-ggufs.md (PR #2681).

The lean option: ROCmFP4 and ROCmI4 (2026-09-24)

ROCmFPX (MIT) is a llama.cpp fork with AMD-focused formats that upstream llama.cpp cannot read. The engine carries it as the lean option (1bit serve --lean, docs/lean.md, engine #37): ROCmFP4 on Vulkan, ROCmI4 on ROCm with the gfx1151 W4A4 path.

Qwen3.8-27B, every file quantized from the same BF16. KL against BF16 over wikitext-2 40 x 512 tokens (PPL(BF16) 5.92); llama-bench pp512 / tg128 tok/s; MTP chat is decode with the MTP head on three prompts (code / prose / short), Vulkan. ROCmFPX fb08d7c.

File	Size	PPL	Mean KLD	99% KLD	Same top token	Vulkan	ROCm	ROCm W4A4	MTP chat
UD-Q4_K_XL (default)	16.4 GiB	5.95	0.008	0.062	95.3%	353 / 11.9	307 / 10.8		22.6 / 20.5 / 19.4
ROCmFP4 STRIX_LEAN (lean)	13.8 GiB	6.08	0.055	0.482	88.7%	348 / 14.2	411 / 13.7		27.8 / 23.3 / 24.1
ROCmI4 (lean, ROCm)	13.9 GiB	6.12	0.051	0.466	89.7%	6 / 4.1	397 / 13.6	465 / 13.6

• ROCmFP4 decodes 19% faster on Vulkan (14-24% with MTP), from a 16% smaller file, at about 7x the KL divergence of UD-Q4_K_XL.
• ROCmI4 is ROCm-only (no Vulkan kernel). W4A4 lifts prompt processing 17% and leaves decode unchanged: ROCmFPX's MTP gains come from MTP verification, which is batched like prefill.
• Vulkan beats ROCm on standard GGUF: UD-Q4_K_XL is 353 / 11.9 on Vulkan against 307 / 10.8 on ROCm.
• The ROCmFPX files had no importance matrix; UD-Q4_K_XL was made with one.

Both lean routes pass serve_e2e on Strix Halo (ROCmFP4 on Vulkan0; ROCmI4 on ROCm0, ROCmI4 W4A4: enabled).

Reference: transformers runs GGUF quants natively (2026-09-22)

Hugging Face's transformers + llama.cpp quants post: AutoModelForCausalLM.from_pretrained(repo, gguf_file=...) now runs GGUF quants (Q4_K_M, Q5_K_M, Q6_K, ...) with ggml's own Metal kernels, packed rather than dequantized on load where a kernel exists. So far it is Apple Silicon only, starts with Qwen3.5 (dense and MoE), and needs transformers main plus a matching kernels.

For us this is an independent reference at the quantized level: on the rented M4 it can score a GGUF's logits (for example a ZINC or HRX run of the same file) against transformers using the same weights, not only against a BF16 original.