E8 lattice KV cache: calibration-free 2-bit that preserves retrieval #30279
jagmarques
started this conversation in
Show and tell
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
-
E8 lattice KV cache compression. Calibration-free, Hadamard rotation + E8 lattice VQ. Same design family as KVarN but uses the lattice constraint to remove worst-case quantization error at 2-bit.
On Llama-3.1-8B-Instruct 4K (n=30 NIAH): our K2V2 at 2.0 bpe scores 30/30 where TurboQuant 2-bit collapses to 0/30 (McNemar b=29/c=0).
Validated on 9 architectures. pip install nexusquant-kv.
Live demo: huggingface.co/spaces/jmarquex/nexusquant-demo
GitHub: github.com/jagmarques/nexusquant
Beta Was this translation helpful? Give feedback.
All reactions