Skip to content

TurboQuant v1.5.1 — Exceeds f16 Quality (4.2x compression)

Choose a tag to compare

View all tags
@AmesianX AmesianX released this 06 Apr 08:13
· 1029 commits to main since this release
v1.5.1
a6ab2a7

TurboQuant v1.5.1

Highlights

  • 3-bit KV cache EXCEEDS f16 quality — tbqp3/tbq3 avg 37.4/65 vs f16 36.6/65
  • SWA f16 bypass — auto-upgrade SWA K+V to f16 (the hidden quality bottleneck)
  • V 512-WHT — V cache now uses same 512-point WHT as K
  • QJL D=512 restored — works correctly with SWA f16 bypass
  • 4.2× compression — 5120 MiB → 1290 MiB

Benchmark (Gemma 4 26B-A4B MoE, DGX Spark GB10, 262K ctx, temp=0)

Config K V Global KV SWA KV Math Accuracy (10 runs) Avg Compression
tbqp3/tbq3 tbqp3 tbq3 990 MiB 300 MiB(f16) 37,38,40,38,38,36,37,36,37,37 37.4 4.2x
tbq3/tbq3 tbq3 tbq3 980 MiB 300 MiB(f16) 39,39,37,37,38,35,35,39,35,36 37.0 4.2x
f16/f16 f16 f16 5120 MiB 300 MiB(f16) 37,36,36,36,36,38,36,38,37,36 36.6 1.0x

Key Techniques

  1. SWA KV f16 Bypass: SWA cache is small (~300 MiB) but has 25 layers dominating quality. Auto-upgrade eliminates SWA quantization noise.
  2. V 512-WHT + 512-IWHT: V cache uses same encode path as K (512-point WHT + global norm).
  3. QJL D=512: Previously removed as "ineffective" — SWA noise was masking the improvement. Now restored with attn_rot auto-disabled for TBQP.
  4. Recommended: --cache-type-k tbqp3 --cache-type-v tbq3 (SWA auto-upgraded to f16)

Compatibility

  • All v1.5.0 features preserved
  • Upstream llama.cpp synced
  • Existing models work without changes
Assets 2
Loading