Implement KVarN variance-normalized KV-cache quantization (2-bit values)

[pekkah]

Summary

Track implementation of KVarN (Müller et al., Huawei CSL, arXiv:2606.03458, repo) — a calibration-free KV-cache quantizer that reaches 4-bit keys / 2-bit values at FP16-level accuracy and ≥FP16 throughput, targeting long-context / reasoning / agentic decoding.

Feasibility research note: docs/kvarn-feasibility-research.md (see linked PR).

Why

• 2-bit KV at FP16 accuracy → 3–5× context, directly on-target for our VRAM-bound single-user desktop use case.
• Explicitly a Pareto improvement over our existing SharpInference.TurboQuant; ~70% of the supporting infra already exists (Hadamard rotation, tile-staged hybrid FP32+compressed cache, SnapKV eviction, fused dequant-dot attention dispatch).
• Calibration-free — no offline codebook generation, works at any head dim (vs. TurboQuant's hardcoded Lloyd-Max tables for dim 128/256).

Algorithm (per 128-token tile)

1. Hadamard rotation along the channel dim (orthonormal → preserves attention scores). Reuse WalshHadamard.cs.
2. Dual-axis "Sinkhorn" variance normalization (log-space alternation of per-channel / per-token std normalization) — the novel piece.
3. Asymmetric RTN quant: per-channel keys, per-token values, group 128.
4. Scales folded at read time. Preset kvarn_k4v2_g128 = 4-bit K / 2-bit V.

Phased plan

• P0 — CPU reference + accuracy gate (~1.5–2.5 wks). KVarNCompressor (Hadamard reuse + Sinkhorn norm + asymmetric RTN K4V2/g128); plug into a TurboQuant-style hybrid cache; scalar dequant-dot; TqAttention-sibling branch behind an env flag. Validate 2-bit accuracy vs FP32 and vs TurboQuant 3–4 bit on MATH500/GSM8K-style runs. ← go/no-go milestone.
• P1 — CPU AVX2 fused kernels (~1–2 wks). Score + V-aggregate for the RTN+scale layout; 2-bit unpack.
• P2 — CUDA kernels (~3–4 wks). NVRTC quantize / dequant-dot / V-aggregate for K4V2; integrate with GpuBufferPool + existing TQ GPU cache state in CudaForwardPass. This is where "throughput ≥ FP16" is won.
• P3 — Vulkan kernels (optional) (~2–3 wks). SPIR-V ports.

Scope notes

• Build as a selectable quantizer inside the existing TurboQuant cache machinery (not a second parallel KV-quant stack); position to eventually supersede the codebook path if it wins.
• KVarN's Triton/CUDA reference is a vLLM fork — algorithm reference only, implement clean-room from the paper.
• MLA support is out of scope (we don't target DeepSeek/GLM latent-attention models).

See the feasibility note for the full infra mapping and for/against analysis.