Problem
FreeToken currently uses only basic safetensors/NVFP4/FP8 weights + MoE offload.
Long-context support (50k+) is limited on MoE models (Qwen3.6-35B-A3B, DeepSeek-V4, GLM-5.2 etc.).
Proposed Solution (drop-in, hardware-agnostic)
Add a unified KV cache backend that supports all SOTA 2026 methods:
Quantization layer (inference-only, no retraining):
- TurboQuant (Google, ICLR 2026) — PolarQuant rotation + QJL residual, 3–4 bit, 5–6x compression, near-zero loss (LongBench, needle recall)
- RotorQuant family (Scrya, IsoQuant/PlanarQuant) — Clifford/quaternion rotations, 3-bit, beats TurboQuant on PPL (+4–6% better), 5–10x faster prefill, 28% faster decode
- KIVI (Meta) — 2-bit per-channel
- KVQuant — mixed-precision + dense-sparse
- Fallback: standard FP8 / int4
Eviction layer (for hybrid):
- H2O (Heavy Hitter Oracle)
- SnapKV, PyramidKV, CAKE, DynamicKV, PyramidKV, CAKE, DynamicKV
Architectural:
- MLA (DeepSeek-V3/V4) — native 5–10x KV reduction in MoE
- GQA / MQA — 4–16x
Implementation:
- Auto-detect MoE expert masks
- Per-expert or per-layer compression
- LRU expert cache compatibility
- Hybrid mode: eviction + quantization (up to 10–20x total)
Full reference implementation + Triton/CUDA kernels available on request (based on existing TurboQuant / RotorQuant / vLLM / llama.cpp codebases).
Expected impact
50k–100k+ context on MoE models without quality loss
Works with any model (Qwen3.6, DeepSeek-V4, GLM etc.) and any offload strategy
Higher decode/pre-fill throughput, lower memory for concurrent requests
This would make FreeToken the first edge-native MoE engine to support the full 2026 KV compression stack.
Problem
FreeToken currently uses only basic safetensors/NVFP4/FP8 weights + MoE offload.
Long-context support (50k+) is limited on MoE models (Qwen3.6-35B-A3B, DeepSeek-V4, GLM-5.2 etc.).
Proposed Solution (drop-in, hardware-agnostic)
Add a unified KV cache backend that supports all SOTA 2026 methods:
Quantization layer (inference-only, no retraining):
Eviction layer (for hybrid):
Architectural:
Implementation:
Full reference implementation + Triton/CUDA kernels available on request (based on existing TurboQuant / RotorQuant / vLLM / llama.cpp codebases).
Expected impact
50k–100k+ context on MoE models without quality loss
Works with any model (Qwen3.6, DeepSeek-V4, GLM etc.) and any offload strategy
Higher decode/pre-fill throughput, lower memory for concurrent requests
This would make FreeToken the first edge-native MoE engine to support the full 2026 KV compression stack.