ggml: proposal to add ShiftAdd (binary-coded) quantization types sa2-sa6

[ranery]

Summary

This proposes ShiftAdd (SA) quantization types for ggml. SA stores each weight group as a sum of binary sign planes with per-group scales, so a mat-vec becomes sign-select-and-add per plane plus one scale multiply per group (ShiftAddLLM, NeurIPS 2024).

The types let llama.cpp run ShiftAdd-quantized models such as ShiftAddQwen-3.5-9B. The fitting pipeline is in the ShiftAddQwen repo.

The code is on two branches against current master: shiftadd-cpu (CPU only) and shiftadd-cuda (CPU + CUDA). I can submit it as one PR, or CPU first with CUDA as a follow-up, whichever you prefer.

Format and implementation

• Layout: each 256-weight block holds Q sign planes (Q*32 bytes), one u8 scale code per plane and group of G weights, and one fp16 scale d, so w_i = d * sum_p a[p][g] * s[p][i] and bpw = Q + 8Q/G + 1/16.
• Types: sa2 to sa6 at G = 64, 128 and 256: 15 types, ids 43-58 (46 unused), 2.1 to 6.8 bpw. I'm open to trimming the set.
• CPU: generic vec_dot, AVX2, AVX2+VNNI, AVX-512 BW+VNNI and NEON; quantize and dequantize.
• CUDA: MMVQ for decode, MMQ below batch 64, dequantize + cuBLAS from 64; get_rows; MUL_MAT_ID through the same paths. gguf-py types and test-backend-ops coverage.
• Producing: no LLAMA_FTYPE entries. llama-quantize --tensor-type makes the types with a greedy fit; the released files come from the ShiftAddQwen exporter, which gives much better quality.
• Limitations: on CUDA the sa types give a better quality-efficiency trade-off than the k-quants, which is their main strength, but on CPU they are still behind (below). MMQ reuses the IQ4_NL tile config and is not tuned; at batch 16-48 sa prefill is about 40% behind Q4_K_M.

Quality

Qwen3.5-9B on A100-SXM4-40GB. KLD against BF16 on held-out mixed-domain text; PPL on wikitext-2 (40 x 512). The sa files are mixed-precision: each linear uses one sa type, and the name gives the average bits/weight over decoder weights. Full comparison with the Unsloth quants: docs/results.md.

model	GGUF (GB)	KLD	PPL
BF16	18.41	0	7.069
Q4_K_M	5.87	0.0257	7.101
sa 4.4 bits	5.10	0.0303	7.201
Q3_K_M	4.83	0.0669	7.215
sa 4.0 bits	4.76	0.0384	7.263

KLD to BF16 (a) and mean accuracy on GSM8K, MMLU-Pro, HumanEval and MGSM (b) against GGUF size, for all sa files and the Unsloth quants. Task scores were measured on an earlier build, not re-run on this branch.

CUDA speed

A100-SXM4-40GB, llama-bench, tok/s, default dispatch.

model	pp512	tg128
Q4_K_M	4245	136.8
Q3_K_M	3569	143.5
sa 4.0 bits	5456	142.8
sa 4.4 bits	5575	131.9

Decode is level with the k-quants. For prefill, sa uses MMQ below batch 64 and dequantize + cuBLAS from 64, which a batch sweep showed is the faster split. At pp512 Q4_K_M runs MMQ by default; forced to cuBLAS it does 5953, 8% ahead of sa 4.0.

CPU speed

Xeon E5-2698 v4, 16 threads, llama-bench tok/s.

model	pp128	tg32
Q4_K_M	56.8	9.0
sa 4.0 bits	9.9	6.6

The CPU vec_dot makes one pass over the activations per sign plane, so its cost grows with the number of planes. The planned follow-up is a row-interleaved repack layout with table lookups.

Testing

• test-quantize-fns passes on generic, AVX2 and AVX-512. NEON: all 15 types pass under QEMU (AArch64 cross-build); not yet run on ARM hardware.
• test-backend-ops CUDA vs CPU: all sa cases pass on V100 and A100.
• ci/run.sh: ctest 52/52.
• NEON: passes on Apple silicon (fork CI, macos-latest-arm64) and under QEMU on AArch64.
• Fork CI: CI (cpu) all test jobs pass, and CI (apple) all jobs pass.

Environment: A100-SXM4-40GB (CUDA 12.6, gcc 11.5) and V100-SXM2-32GB (CUDA 12.3, gcc 9.3); CPU numbers on a Xeon E5-2698 v4.

Requirements

• I have read and agree with the contributing guidelines
• AI usage disclosure: YES. The kernels were drafted with AI assistance from my design; the rebase onto master and the test runs were AI-assisted. This description is my own.