Per-head adaptive KV cache quantization: lossless q4_0 on hybrid models, +8% quality on standard models

[SCJedi]

Summary

Two findings from production benchmarks of -ctk q4_0 -ctv q4_0:

1. q4_0 KV cache is completely lossless on hybrid models (Qwen3.5) -- BLEU 1.000 across 10 test configurations at 4x compression
2. For standard models (Llama, Mistral), per-head entropy-adaptive bit allocation improves quality by +8% BLEU at the same compression ratio

Feature request: per-head KV cache type support to enable adaptive allocation.

Finding 1: Lossless q4_0 on Hybrid Models

A/B benchmarks on RTX 3060 12GB, Qwen3.5-9B, 3-run averages:

Test	f16 gen/s	q4_0 gen/s	Speed Diff
Short (2-turn)	48.2	47.0	-2.4%
Medium (8-turn)	47.0	45.8	-2.6%
Long (20+ turn)	45.9	45.8	-0.1%

Quality: BLEU 1.000 (n=50 completions). Token-level identical output between f16 and q4_0 KV cache.

The 33K-token conversation test ran at 36.5 tok/s on q4_0 -- f16 would OOM on the same 12GB GPU.

Why Hybrid Models Tolerate q4_0

Qwen3.5 uses only 8 of 32 layers for full attention with KV cache. The other 24 layers use linear attention (Gated Delta Net) with no KV cache. Similarly, Gemma 4 uses 10 of 60 layers for global attention.

The linear/sliding attention layers act as error correction -- quantization noise in the few attention layers is absorbed by the surrounding layers. Standard models (Llama, Mistral) have no such correction because all layers use full attention.

Finding 2: Entropy-Adaptive Allocation for Standard Models

For standard architectures where q4_0 does degrade quality, per-head bit allocation based on attention entropy recovers most of the loss:

b_h = b_avg + 0.25 * (H_avg - H_2(h))

Where H_2(h) is the Renyi entropy of head h's attention distribution.

Results on GPT-2 (all at 4x compression):

Config	BLEU	Token Match	PPL Ratio
Uniform 4-bit	0.531	51.3%	1.101
Adaptive 4-bit	0.548	53.8%	1.089
Skip 3 sinks + adaptive	0.574	60.0%	1.062

The key insight: the bottom 2% of heads by entropy ("sink heads") contribute disproportionately to quantization error. Skipping quantization on just 3 out of 144 heads provides more benefit than optimal bit redistribution across all others.

Feature Request: Per-Head KV Cache Types

Currently -ctk and -ctv apply uniformly to all layers and heads. Per-head allocation would enable:

• Sink heads (lowest entropy, ~2% of heads): keep at f16 or q8_0
• Standard heads: q4_0 as today
• High-entropy heads (diffuse attention): could go to q3_0 or lower

This maintains the same average compression ratio but shifts bits to where they matter most. The implementation would need:

1. A calibration pass to compute per-head entropy (or accept a pre-computed entropy map)
2. Per-head type selection in the KV cache allocation
3. Appropriate dequantization dispatch per head during attention

The entropy computation is a one-time cost per model architecture (not per session). The per-head map could be distributed alongside GGUF files or computed on first load.

Production Deployment Data

The full 12x compression stack (q4_0 KV + application-level context eviction) is running in production on RTX 3060:

• 128K context window (vs 32K with f16 KV)
• 45-48 tok/s generation speed
• Infinite conversation length via automatic eviction
• Zero model or llama.cpp source changes required

References

• Discussion TurboQuant - Extreme KV Cache Quantization #20969 (TurboQuant for llama.cpp)
• Paper: https://github.com/SCJedi/entropy-adaptive-kv-cache/tree/main/paper
• GPT-2 entropy experiments: https://github.com/SCJedi/entropy-adaptive-kv-cache
• Production benchmarks + deployment guide: https://github.com/SCJedi/BitNet/tree/bitnet-tools

[jagmarques]

Your hybrid-tolerance finding shows up at the layer level too. On Qwen2.5-7B (not hybrid), K3V2 with the first and last layer left fp16 gives +0.84% PPL; dropping that protection blew PPL from 6.12 baseline to 3300. On Mistral-7B (also not hybrid), the same protection-off setting gave +0.31% PPL across 32 layers. Per-head adaptive plus per-layer protection look orthogonal; combining both axes would be a useful next experiment.

[SCJedi]

Thanks — interesting numbers. Heads-down on a separate workstream right now, not in position to discuss. If there's a paper or preprint, I'll bookmark it.

[jagmarques]

No preprint yet, will drop the arXiv link here when it goes up.

[SCJedi]

Keep making the world better brother.

[sztlink]

I ran TheTom's REFRACT v0.3.2.3 on the q4_0/q4_0 hybrid claim from #21385 — Qwen3.6-35B-A3B Q4_K_M, RTX 3090, llamacpp backend, 128-token Trajectory axis.

KLD stays close (98.81), consistent with the BLEU/token-match finding. Under REFRACT trajectory (path-preservation lens): q4_0/q4_0 is DEGRADED — path score 65.70 vs 100 for the q8_0/q8_0 reference; 53% of runs preserved the reference trajectory.

K-only and V-only isolation runs also DEGRADED (path 55.25 / 58.42). This is a metric-family question, not a contradiction: KLD and trajectory measure different fidelity layers.

Full artifacts: https://github.com/sztlink/turboquant-cuda-bench/tree/main/bench/q4-hybrid-refract
Chart: https://github.com/sztlink/turboquant-cuda-bench/blob/main/bench/q4-hybrid-refract/charts/q4-hybrid-refract.png

[github-actions]

This issue was closed because it has been inactive for 14 days since being marked as stale.

[dhawalc]

@jagmarques @SCJedi — your measurements in this thread appear to be independent
observations of a failure mode we have been characterizing in a different
codebase, and I'd like to invite you to check our numbers rather than take our
word for them.

What you observed, and what we see with a TurboQuant-family quantizer
(WHT rotation + Lloyd-Max + residual signs, no llama.cpp code shared):

• @jagmarques: Qwen2.5-7B, 3-bit keys, layer protection off → PPL 6.12 → ~3,300.
  We measure 7.52 → 9,410 (keys+values) / 10,655 (keys only) on wikitext-2 at
  3-bit, same model. Your Mistral-7B +0.31% under the identical setting matches
  our finding that the failure is family-specific: across 150 configurations on
  26 models, every cell above ×5 is Qwen2 or Qwen2.5. Ministral-8B measures
  immune here too (×1.07 worst), consistent with your Mistral-7B number.
• Your choice of which layers to protect looks non-accidental. Our per-layer
  profile on Qwen2.5-7B at 3 bits has its two worst layers at exactly L0 (0.54
  attention-logit correlation) and L27 (0.80), with everything between ≥0.91.
  Tracing it to the weights: Qwen2/Qwen2.5 carry a huge k_proj bias at those
  boundary layers (‖b_k‖ ≈ 605 at L0 and 921 at L27 on 7B) against a median of
  ~22 elsewhere. Qwen1.5 has no such spike (max 54) and does not fail. The
  1.5B models have the spike only at L0 — and their damage dips only at L0.
• The mechanism we measure: a per-head shared key mean carrying ~53% of key
  energy; subtracting the running mean before quantization and restoring it after
  moves 3-bit PPL from 9,410 to 7.90. Injecting a synthetic shared component into
  Llama-3.2-1B (which is otherwise immune) reproduces the full failure
  dose-dependently, so the mechanism is causal, not a Qwen quirk.
• @SCJedi: your q4_0-lossless result on Qwen3.5 matches our sweep — the Qwen3.5
  generation measures immune at every bit-width we tested (2/3/4-bit, ≤×1.01),
  and Qwen3.5-0.8B survives even 1-bit key indices at ×1.04.

We run a public registry for independent reproduction attempts — negative and
partial results are recorded in the same table as positive ones, and appearing
in it does not imply endorsement of our interpretation:

https://github.com/dhawalc/turboQuantDC/blob/HEAD/REPRODUCTIONS.md

The reference experiment is pinned to immutable commits (script, raw JSON,
report), runs on a single 24 GB GPU in ~30 minutes, and states explicitly what
counts as a match. If either of you has the time to run it — or to tell us why
the experimental design is wrong — both outcomes go in the registry.

One caveat on our own numbers, since it affects what a reproduction should
expect: the uncorrected arm is a diverged quantity and does not reproduce to a
specific value. We see ×1,416 on wikitext-2 and ×2,929 on a Gutenberg text for
the same configuration. Any value above ~100× baseline reproduces the
qualitative claim; the magnitude is not meaningful.

@jagmarques: when your preprint is up we will cite it. Our working manuscript
already records this thread's measurements as independent concurrent
observations (§2.6), and treats them as evidence we could not have produced
ourselves — a second codebase and a different quantizer reaching the same
place matters more than another run of ours.