• [Upcoming] 🔧 vLLM & SGLang backend integration — under active development, official support will be announced in future releases.

• [2026-05-20] 🎉 Our paper "OScaR: The Occam's Razor for Extreme KV Cache Quantization in LLMs and Beyond" is now available on arXiv! [Link]

• [2026-05-19] 🚀 Codebase and evaluation suite publicly released.

• Latest News
• Overview
  • TNI in X-LLMs
• Key Features
• Main Results
  • Text-Only LLMs: LongBench-E
  • Multi-Modal LLMs: OCRBench
  • Omni-Modal LLMs: MMAU-Pro
• Installation
• Quick Start
  • Smoke Test
  • Full Benchmark
  • Accuracy Evaluation
  • Single Example
• Citation
• Acknowledgement

The rapid advancement toward long-context reasoning and multi-modal intelligence has made KV cache memory footprint a dominant bottleneck. We revisit the inherent limitations of the established per-channel quantization paradigm and identify Token Norm Imbalance (TNI) as the primary bottleneck to quantization fidelity.

Rather than relying on intricate pipelines, we follow the principle of Occam's Razor. We propose OScaR (Omni-Scaled Canalized Rotation) , an accurate and lightweight KV cache compression framework for X-LLMs (text-only, multi-modal, and omni-modal LLMs).

Text-Only LLMs Low-norm outlier tokens (Attention Sink tokens)

Multi-Modal LLMs Large-norm outliers

Multi-Modal LLMs Inter-modality disparities

TNI is pervasive across X-LLMs. In text-only models, it manifests as low-norm outlier tokens, also known as Attention Sink tokens. In multi-modal settings, TNI exhibits more diverse forms, including large-norm outliers, significant inter-modality disparities, and broader norm variations. Additional visualizations and detailed experimental configurations are provided in the paper.

• 🔍 Unveils TNI as the structural bottleneck of per-channel quantization through both empirical and theoretical analysis.

• 🪒 Streamlined OScaR framework guided by Occam's Razor — requiring only two essential operations, Canalized Rotation and Omni-Token Scaling, with no training or calibration overhead.

• 📈 Redefines the Pareto front for X-LLMs KV quantization, delivering near-lossless INT2 quantization across diverse benchmarks while maintaining low computational complexity.

• ⚡ Optimized System Design and CUDA kernels built on BitDecoding and HadaCore with Tensor Core acceleration, achieving 3.0× decoding speedup, 5.3× memory reduction, and 4.1× throughput increase vs. BF16 FlashDecoding-v2.

OScaR achieves the highest average accuracy among all 2-bit methods on LongBench-E, outperforming KIVI, OTT, QuaRot, and TurboQuant+ across both Llama-3.1-8B and Qwen3-8B.

On OCRBench, OScaR consistently outperforms other 2-bit methods across LLaVA-v1.6-vicuna-7B, Qwen3-VL-8B, and Qwen3-VL-4B.

On the challenging MMAU-Pro benchmark for omni-modal understanding, OScaR surpasses both the 16-bit baseline and all quantized methods across open-ended QA, Good Rate, and Audio Instruction Following (AIF).

Note: Detailed experimental setups and TurboQuant+ implementation details are available in the original paper.

git clone https://github.com/ZunhaiSu/OScaR-KV-Quant.git OScaR
cd OScaR

# Prerequisite: install `uv` and ensure it is available on PATH.
uv venv --python 3.10 --seed oscar-env
source oscar-env/bin/activate

# Required for CUTLASS headers used by oscar_cuda.
git submodule update --init --recursive

# flash-attn imports torch and psutil during its build, so they must exist first.
uv pip install "torch==2.6.0+cu124" psutil --index https://download.pytorch.org/whl/cu124

# Install dependencies declared in pyproject.toml, then install the project itself.
uv sync --active --no-install-project
uv pip install --no-build-isolation -e .

If you clone with --recursive, you should still run git submodule update --init --recursive before building to ensure libs/cutlass is present.

The Python dependency source of truth is pyproject.toml. tool.uv.sources pins torch==2.6.0 to the cu124 PyTorch index, and tool.uv.no-build-isolation-package disables build isolation for flash-attn. The explicit torch/psutil bootstrap step is still required because flash-attn imports them while building but does not declare them as build dependencies. The editable install uses --no-build-isolation because this repository's CUDA extension build imports PyTorch from the active environment.

Tested Environment:

• Python 3.10.17
• PyTorch 2.6.0+cu124
• flash-attn 2.8.3
• transformers 5.8.1 for a fresh installation from the current pyproject.toml

Set the model path:

export MODEL_PATH=/path/to/Qwen3-8B

Quick end-to-end accuracy validation using the Qasper-E benchmark:

CUDA_VISIBLE_DEVICES=0 $(which python) eval_longbench.py \
  --model_path "$MODEL_PATH" \
  --datasets qasper_e \
  --max_input_len 32768 \
  --dtype bfloat16 \
  --device cuda:0 \
  --offline_v_hadamard \
  --output_dir pred_e/oscar-qasper \
  --log_every 1 \
  --resume

Note: This requires the following data files:

• longbench_data/data/qasper_e.jsonl
• longbench_config/dataset2prompt.json
• longbench_config/dataset2maxlen.json

The metric helper longbench_metrics.py is part of this repository, and its Python dependencies are included in pyproject.toml.

Run a single inference example with explicit configuration:

MODEL_PATH="${MODEL_PATH}" \
DTYPE=bfloat16 \
NUM_BITS=2 \
QUANT_MODE=k-channel \
GROUP_SIZE=32 \
KV_ROTATION=hadamard \
KV_NORM=1 \
ATTN_BACKEND=oscar \
bash evaluation/scripts/example.sh

If you find OScaR useful for your research or production, please cite our paper:

@article{su2026oscar,
  title={OScaR: The Occam's Razor for Extreme KV Cache Quantization in LLMs and Beyond},
  author={Su, Zunhai and Yang, Rui and Zhang, Chao and Liu, Yaxiu and Zhang, Yifan and Wu, Wei and Xiong, Jing and Du, Dayou and Zhuang, Xialie and Qian, Yulei and Xie, Yuchen and Wu, Yik-Chung and Yang, Hongxia and Wong, Ngai},
  journal={arXiv preprint arXiv:2605.19660},
  year={2026}
}

OScaR is inspired by many open-source libraries, including but not limited to BitDecoding, HadaCore, KIVI, and SGLang-FluentLLM.