Extreme weight and KV cache compression for LLMs on Apple Silicon. MLX implementation of Google's TurboQuant (Zandieh et al., 2025) — Hadamard rotation + Lloyd-Max codebooks applied both to weights (compile time) and the KV cache (run time).
Supports dense models (LLaMA, Qwen, Mistral), Mixture-of-Experts (Qwen-MoE, GPT-OSS, Qwen3.5-MoE, Qwen3.6-35B-A3B, Qwen3-235B-A22B, DeepSeek-V2/V3), Mamba/attention hybrids (Nemotron-3-Nano-4B, Nemotron-3-Super-120B), and multimodal models (Muse Glimmer, Qwen3.8-27B). Compatible with hybrid attention architectures, attention sinks, sliding-window attention, and linear attention layers (including Gated DeltaNet).
With both weight and KV cache compression at 3-bit, GPT-OSS-120B fits its full 131K context window in 50 GB on a 64 GB MacBook — and KV cache compression actually makes generation faster on the 120B (8.7 vs 6.4 tok/s) because the smaller cache cuts memory bandwidth more than dequant costs.
Expert streaming (v0.4.0) runs MoE models whose weights exceed available RAM by paging only the router-selected experts from disk per token — e.g. the 35B-parameter Qwen3.6-35B-A3B runs on a 16 GB Mac mini in under 4 GB of RAM, with output bit-identical to the fully-resident model. See Qwen3.6-35B-A3B on a 16 GB Mac mini.
Local coding — Qwen3.6-27B, a dense SWE-bench-grade coder, runs fully resident on a 48 GB Mac at 3-bit (~13 GB on disk, ~17.5 GB at runtime) and serves to Cursor / VS Code over an OpenAI-compatible endpoint. See Qwen3.6-27B.
| Model | Method | Bits | PPL | Size | Gen Speed (M4 Max) |
|---|---|---|---|---|---|
| Qwen2.5-7B | TurboQuant | 3 | 8.92 | 3.5 GB | — |
| Qwen2.5-7B | Affine | 3 | 13.37 | 3.3 GB | — |
| GPT-OSS-20B | Affine (mlx-lm) | 4 | — | 11.2 GB | 148 tok/s |
| GPT-OSS-20B | MXFP4 (original) | 4 | 83.04 | 12.8 GB | — |
| GPT-OSS-20B | TurboQuant | 4 | 72.63 | 11.2 GB | — |
| GPT-OSS-20B | TurboQuant | 3 | 78.60 | 9.3 GB | 73 tok/s |
| GPT-OSS-120B | Affine 4-bit (mlx-community) | 4 | — | 65.8 GB | Doesn't fit 64GB |
| GPT-OSS-120B | MXFP4 (original) | 4 | — | 63.5 GB | Doesn't fit 64GB |
| GPT-OSS-120B | TurboQuant | 3 | — | 48 GB | 44 tok/s |
| GPT-OSS-120B (hybrid for 48GB) | TQ 3-attn / 2-experts, gs=32 | 2/3 mix | — | ~35 GB | 42–50 tok/s |
| GPT-OSS-120B | TurboQuant | 2 | — | 32 GB | 51 tok/s (poor quality) |
| Qwen3.5-122B-A10B | BF16 (original) | 16 | — | ~240 GB | Doesn't fit 64GB |
| Qwen3.5-122B-A10B | TurboQuant | 3 | — | ~50 GB | 26.5 tok/s (64 GB) · streams on a 16 GB Mac mini |
| Qwen3.6-35B-A3B | TurboQuant, gs=32 | 3 | — | ~16 GB | ~60 tok/s (resident) · runs in <4 GB via streaming |
| Qwen3.6-27B (dense coder) | TurboQuant, gs=32 | 3 | — | ~13 GB | ~14 tok/s (resident) · fits 48 GB, SWE-bench coder |
| Qwen3-235B-A22B-Instruct-2507 (hybrid) | TQ 3-attn / 2-experts, gs=32 | 2/3 mix | — | 70.5 GB | ~4–6 tok/s (64 GB, 40 GB cache) · converts + streams on a 16 GB Mac mini |
| Qwen3-235B-A22B-Instruct-2507 (full 3-bit) | TurboQuant, gs=32 | 3 | — | 103 GB | ~1.3 tok/s (64 GB, 40 GB cache) · recall-critical sibling, passes 6/6 stress |
| Qwen3-235B-A22B-Instruct-2507 (ternary experts, 1.58-bit) | TQ 3-attn / ternary trit-packed experts, gs=64 | 1.6/3 mix | — | 53 GB | 5.6 tok/s (64 GB) · fully resident, no streaming |
| Nemotron-3-Nano-4B | TurboQuant | 3 | — | ~2.2 GB | 75.6 tok/s |
| Nemotron-3-Super-120B-A12B | BF16 (original) | 16 | — | ~240 GB | Doesn't fit 64GB |
| Nemotron-3-Super-120B-A12B | TurboQuant | 3 | — | ~50 GB | 18.7 tok/s |
| Nemotron-3-Super-120B-A12B (hybrid for 48GB) | TQ 3-attn / 2-experts, gs=32 | 2/3 mix | — | ~36 GB | ~27.2 tok/s |
| Laguna-XS.2 (33B agentic coder) | TurboQuant, gs=64 | 3 | — | 13.8 GB | 46.8 tok/s · 32 GB Mac · Opencode pass |
| Laguna-S-2.1 (118B) | BF16 (original) | 16 | — | ~219 GB | Doesn't fit 64GB |
| Laguna-S-2.1 (118B) | Affine 4-bit | 4 | — | ~64 GB | Exceeds Metal's ~56 GB working set — won't load |
| Laguna-S-2.1 (118B, ternary experts) | TQ 3-attn / ternary trit-packed experts, gs=64 | 1.6/3 mix | — | ~27 GB | ~12.5 tok/s · fully resident on 64 GB with ~25 GB spare · Opencode pass |
| Kimi K3 (2.8T MoE) | TQ 3-attn / ternary experts / 4-bit expert-down, gs=64 | 1.6/3/4 mix | — | 931 GB | 2.3 tok/s (Mac Studio 512 GB, top-8) · 1.2 tok/s at native top-16 · streams experts from disk |
| Muse Glimmer (30B dense VLM) | Affine 4-bit | 4 | 4.3798 | 19.88 GiB | 26.1 tok/s · leaves embedding + vision tower at bf16 |
| Muse Glimmer (30B dense VLM) | TurboQuant, gs=64 | 4 | 4.3315 | 14.88 GiB | 9.4 tok/s · better PPL than affine 4-bit in 25% less space |
| Muse Glimmer (30B dense VLM) | TurboQuant, gs=64 | 3 | 5.0454 | 12.53 GiB | 10.9 tok/s · smallest coherent build |
| Qwen3.8-27B (dense 27.8B hybrid-GDN VLM) | BF16 (original) | 16 | — | 55.6 GB | Needs a 64 GB Mac just to load |
| Qwen3.8-27B | Affine 4-bit (mlx-community) | 4 | 7.2685 | 14.95 GiB | 29.7 tok/s — fastest 4-bit, but peaks 16.16 GiB: won't fit 16 GB |
| Qwen3.8-27B | TurboQuant, gs=64 | 4 | 7.2496 | 15.15 GiB | 11.3 tok/s · best PPL of the three · Opencode agentic pass · vision 4/4 |
| Qwen3.8-27B (16 GB build) | TurboQuant, gs=64 | 3 | 8.0052 | 11.55 GiB | 12–13.6 tok/s (M4 Max) · verified on a real 16 GB Mac mini: 13.61 GiB peak, 3.7 tok/s, vision passes |
| Model | KV cache config | KV size | Speed | Notes |
|---|---|---|---|---|
| GPT-OSS-20B (FP16 weights) | FP16 KV | 27.0 MB | 90.6 tok/s | baseline |
| GPT-OSS-20B (FP16 weights) | TQ 3-bit KV | 7.79 MB | 29.9 tok/s | 3.5x cache savings |
| GPT-OSS-120B (TQ 3-bit weights) | FP16 KV | 45.0 MB | 6.4 tok/s | baseline |
| GPT-OSS-120B (TQ 3-bit weights) | TQ 3-bit KV | 11.83 MB | 8.7 tok/s | 3.8x cache savings — and faster than FP16 |
| GPT-OSS-120B (TQ 3-bit weights) | TQ 4-bit KV | 12.21 MB | 16.0 tok/s | also clean |
| Qwen3.5-122B (TQ 3-bit weights) | FP16 KV | 161.06 MB | 5.4 tok/s | baseline |
| Qwen3.5-122B (TQ 3-bit weights) | TQ 3-bit KV | 150.17 MB | 5.7 tok/s | output identical to FP16 |
KV cache compression projects to ~7 GB RAM saved at 131K context on GPT-OSS-120B and ~5 GB at 262K on Qwen3.5-122B. Roundtrip cosine similarity vs FP16: 0.983 at 3-bit, 0.995 at 4-bit.
On the Qwen3.5-122B KV rows: these were measured with symmetric 3-bit KV (
demo_kv.py --tq-bits 3, short prompt), and the rates sit well below the model's fully-resident decode. With the Metal wired cap raised so the ~50 GB model stays fully resident, decode with the recommended mixed K8/V3 cache is ~24–25 t/s — and there fp16 edges out compression at short-to-moderate context, the gap widening as context grows. On this model KV compression is a memory win, not a speed win; the genuine decode speed-up is GPT-OSS-120B-specific. See the resident long-context sweep in #19.
First Metal4 / MTLGPUFamilyApple10 data point, contributed by @sbayer2 (#14, #16) — a new 48 GB tier between the 16 GB Mac mini and the 64 GB M4 Max. Reproduce with benchmarks/bench_m5_pro.py.
| Model | Mode | Gen t/s | Peak Memory | Notes |
|---|---|---|---|---|
| Qwen3.6-35B-A3B tq3-g32 | resident | 52.0 | 18.1 GB | fp16 KV |
| Nemotron-3-Super-120B-A12B tq3a/tq2e-g32 | resident | 20.2 | 41.1 GB | needs sudo sysctl iogpu.wired_limit_mb=49152 |
| Qwen3.5-122B-A10B tq3 | streaming (30 GB cache) | 9.4 (7.3 e2e) | 34.2 GB | 89.9% expert hit-rate |
KV cache sweep on the 35B (resident, 3-run averages):
| KV config | Prompt t/s | Gen t/s | Peak |
|---|---|---|---|
| fp16 baseline | 47.8 | 52.0 | 18.131 GB |
| K8 / V3 | 75.5 | 45.7 | 18.123 GB |
| K8 / V3 + sink128 | 76.1 | 45.7 | 18.124 GB |
| K3 / V3 | 75.6 | 45.2 | 18.122 GB |
122B expert streaming — parallel prefetch (--cache-budget-gb 30, 256 tokens):
--prefetch-workers |
Gen t/s | E2E t/s | Disk read | Hit rate |
|---|---|---|---|---|
| 1 (serial) | 7.4 | 5.7 | 44.3 GB | 89.5% |
| 8 (parallel) | 9.1 | 7.6 | 41.9 GB | 90.1% |
| Speedup | 1.23× | 1.33× |
A 1.23× decode speedup from --prefetch-workers 8, landing between the Mac mini (1.3×) and the M4 Max (1.67×). The M5 Pro MacBook Pro SSD is the limiter — parallel prefetch helps but doesn't saturate the way the M4 Max's higher-bandwidth SSD does.
122B expert streaming — cache-budget sweep (--prefetch-workers 8, 256 tokens):
| Budget | Hit rate | Gen t/s | E2E t/s | Peak Metal | Disk read |
|---|---|---|---|---|---|
| 20 GB | 80.9% | 7.1 | 6.1 | 24.2 GB | 80.7 GB |
| 30 GB | 90.3% | 9.1 | 7.6 | 34.2 GB | 40.9 GB |
| 38 GB | 91.0% | 8.9 | 7.5 | 42.2 GB | 38.0 GB |
The hit-rate curve flattens hard past 30 GB (only +0.7% for +8 GB), and throughput actually dips at 38 GB as peak Metal (42.2 GB) crowds the wired cap. 30 GB is the sweet spot on the 48 GB tier — 90%+ hit rate with ~14 GB of headroom for OS stability; pushing to 38 GB gives negligible gain while peaking uncomfortably close to the wired limit.
KV compression gives a consistent ~1.6× prompt-processing speedup for a ~12% decode cost (long-context decode behavior is in The speed flip). Expert-streaming hit-rate scales with the cache budget — 44.6% at 4 GB (16 GB mini) → 89.9% at 30 GB (48 GB), a ~7× throughput jump that fills the gap between the 16 GB and 64 GB tiers.
Stability near the memory ceiling: long-context prompt prefill close to the wired cap can starve the kernel watchdog (a
watchdogd/AppleARMWatchdogTimerpanic). A rapidly-growing KV cache makes Metal commit pages continuously (AGXG17XFamilyResidencySet _commitAddedAllocations), so the binding limit is allocation rate, not peak — a static 41 GB resident model (Nemotron) is stable, while a growing ~30 GB KV during a 63K-token prefill can panic. Practical limits on the 48 GB tier (#14): contexts up to ~14.5K are safe on the 35B with either KV config; 63K is feasible with K8/V3 (28.7 GB peak) but not fp16 (30.8 GB → panic). Keep headroom and close other apps for long-context runs near the cap.
pip install turboquant-mlx-fullThe package is published as turboquant-mlx-full on PyPI, but importable as
turboquant_mlx (without the -full suffix) — this matches the original
project name and the examples in the Medium articles.
import turboquant_mlx
from turboquant_mlx.layers import TurboQuantKVCache, convert_cache_to_turboquant- macOS with Apple Silicon (M1/M2/M3/M4)
- Python 3.10+
- 64 GB unified memory recommended for 20B+ models
The Metal kernels are JIT-compiled by MLX at first use, so no Xcode / CMake toolchain is required to install the package.
git clone https://github.com/manjunathshiva/turboquant-mlx.git
cd turboquant-mlx
pip install -e .Optional extras:
pip install "turboquant-mlx-full[eval]" # perplexity benchmarking (datasets, transformers)
pip install "turboquant-mlx-full[vlm]" # multimodal / diffusion models via mlx-vlm
pip install "turboquant-mlx-full[kimi]" # Kimi K3's tiktoken-based tokenizerMuse Glimmer needs mlx-vlm >= 0.6.12, the first release carrying its MLX model classes (Blaizzy/mlx-vlm#1838, merged 2026-08-10).
[vlm]pins that floor, so the one-liner above is all you need — the git pin previously documented here is obsolete.
Before a multi-GB download, ask:
turboquant-plan --model manjunathshiva/Qwen3.6-35B-A3B-tq3a-tqTe-down4-g64Projection at 16,384 tokens of context
weights 12.59 GB
KV cache 0.34 GB (20.0 KB/token, hybrid: 10/40 full-attention layers)
prefill workspace 1.07 GB (estimate, at --prefill-step-size 2048)
runtime reserve 1.00 GB (buffer cache, activations, fragmentation)
----------------------------------
peak 15.00 GB of 61.85 GB usable 46.85 GB headroom
Verdict: ✅ RESIDENT — fits fully in memory
It reads only safetensors headers — no tensors are loaded, no engine starts. A HuggingFace repo id is planned over the network, from range requests against the shard headers: the 12.6 GB repo above answers in ~2.5 s and pulls 232 KB into a cold cache, so "will it fit?" is answered before the download, not after. Weights are exact; KV is derived from the config's attention geometry (hybrid models only grow KV on their full-attention layers); the prefill workspace is an estimate that scales with both chunk size and context.
Planning for a machine you're not sitting at, and asking for the flags:
turboquant-plan --model <repo> --wired-gb 10.5 --ram-gb 16 --context 21000 --kv-bits 8Verdict: ⚠️ RESIDENT — fits, but only after raising the Metal wired cap
Recommended:
sudo sysctl -w iogpu.wired_limit_mb=13721 (raises the 10.50 GB Metal cap —
the binding limit here, not your 16 GB of RAM; resets on reboot)
--prefill-step-size 128 (the default 2048 needs 1.38 GB of transient
workspace at this context)
--kv-bits 8
That is the 16 GB Mac mini recipe, derived rather than found by trial and error — the projection is calibrated against measurements on that machine (it predicts 10.44 GB where the 9.4 GB ternary build measures 10.42, and recommends a wired limit within 1% of the one that actually works).
Changing the prefill chunk size can change a greedy answer by a token — measured
on a ~3.9K-token prompt, --prefill-step-size 2048 produced "...which explains
why the sky reads blue" where 512/256/128 produced "...which is why the sky
reads blue". Both are valid argmax continuations; the chunk size only changes the
order the GPU reduces in, and fp16 rounding occasionally lands on the other side
of a near-tie.
This is a property of chunked prefill on Metal, not of TurboQuant. The same
test on stock mlx-community/Llama-3.2-1B-Instruct-4bit — plain mlx-lm, plain
affine 4-bit, none of our kernels — forks the same way. Our own decode and
prefill MoE kernels are bit-identical to each other
(tests/test_kernel_determinism.py), which is what keeps --disk-cache
checkpoint restores consistent.
Practical upshot: for byte-reproducible greedy output, hold --prefill-step-size
fixed across runs. Everything else — quality, correctness, coherence — is
unaffected.
turboquant-doctor runs the same projection plus a read-only readiness check —
model files, tokenizer, quantization block, mlx/mlx-lm imports, machine limits —
with stable check ids and exit codes (0 ok/warn, 1 won't fit or files missing,
2 bad usage). Both take --json for automation.
# Dense model (e.g., LLaMA 3.2 1B at 3-bit)
python -m turboquant_mlx.convert \
--hf-path meta-llama/Llama-3.2-1B \
--mlx-path ./llama-3.2-1b-tq3 \
--bits 3 --group-size 64
# MoE model (e.g., GPT-OSS-20B at 2-bit)
python -m turboquant_mlx.convert \
--hf-path openai/gpt-oss-20b \
--mlx-path ./gpt-oss-20b-tq2 \
--bits 2 --group-size 64
# Very large model whose quantized form won't fit in RAM (200B+): --streaming
# writes each layer to a shard and frees it, so peak memory stays ~one shard
# (5 GB) + one layer — letting 235B/671B-class MoEs convert on a 64 GB Mac.
python -m turboquant_mlx.convert \
--hf-path Qwen/Qwen3-235B-A22B-Instruct-2507 \
--mlx-path ./qwen3-235b-tq3 \
--bits 3 --group-size 64 --streaming
# Ternary (1.58-bit) experts — sub-2-bit MoE experts on the data-free {-c, 0, +c}
# codebook, packed as genuine base-3 trits (20 per uint32, ~1.6 bpw vs 2.0 for the
# 2-bit slot). Attention + lm_head stay at --bits; only the routed experts go
# ternary. On a 128-expert model this shrinks Qwen3-235B to 53 GB (from 70.5 GB),
# so it runs FULLY RESIDENT on a 64 GB Mac at ~5.6 tok/s (vs ~2 tok/s streaming).
# Needs enough expert redundancy — great on 128 experts, breaks below ~64.
python -m turboquant_mlx.convert \
--hf-path Qwen/Qwen3-235B-A22B-Instruct-2507 \
--mlx-path ./qwen3-235b-tq3a-tqTe-g64 \
--bits 3 --group-size 64 --ternary-experts --streaming
# Resident load of a ~53 GB model needs a wired-memory bump on a 64 GB Mac:
# sudo sysctl -w iogpu.wired_limit_mb=60416
# Asymmetric experts — ternary up/gate + higher-bit down_proj. The down
# projection is the SwiGLU write-back into the residual stream, and spending
# 4 bits on just that one expert matrix is what turns a sub-2-bit build
# agent-capable: on Qwen3.6-35B-A3B, pure ternary fails an Opencode coding
# task 0/4 while ternary+down4 passes 3/3 — for +3.2 GB (9.4 -> 12.6 GB),
# still resident on a 16 GB Mac mini.
python -m turboquant_mlx.convert \
--hf-path Qwen/Qwen3.6-35B-A3B \
--mlx-path ./Qwen3.6-35B-A3B-tq3a-tqTe-down4-g64 \
--bits 3 --group-size 64 --ternary-experts --expert-down-bits 4turboquant-generate \\
--model ./gpt-oss-20b-tq2 \
--prompt "Why is the sky blue? Explain in simple terms." \
--max-tokens 200python -m turboquant_mlx.evaluate \
--hf-path openai/gpt-oss-20b \
--bits 2 3 4 \
--num-samples 256 --seq-len 512The production turboquant-generate CLI accepts KV-cache flags directly (v0.2+).
Use mixed K/V precision (--kv-k-bits 8 --kv-v-bits 3) — required for
TurboQuant-quantized weights, and lossless on stock fp16 weights:
# v0.2 recommended default: mixed K8/V3 + 128-token fp16 sink
turboquant-generate \
--model ./gpt-oss-120b-tq3 \
--prompt "Why is the sky blue?" \
--max-tokens 1024 --temp 0.7 \
--kv-k-bits 8 --kv-v-bits 3 --kv-min-tokens 128
# Symmetric (legacy) — only safe on fp16 weights
turboquant-generate \
--model openai/gpt-oss-20b \
--prompt "Why is the sky blue?" \
--max-tokens 200 --kv-bits 3
# Side-by-side comparison harness (4 configs in one run)
python -m turboquant_mlx.benchmarks.demo_kv_v02 \
--model ./gpt-oss-120b-tq3 \
--prompt "Why is the sky blue?" \
--max-tokens 1024 --temp 0.7 --top-p 0.9 --repetition-penalty 1.1turboquant-serve wraps mlx_lm.server and patches its loader so any
TurboQuant model (quantization.mode = "turboquant" in config.json)
loads through the PolarQuant path. Non-TurboQuant models pass through
unchanged, so this is a drop-in replacement for mlx_lm.server.
# Serve a local TQ model
turboquant-serve \
--model ./NVIDIA-Nemotron-3-Super-120B-A12B-BF16-tq3 \
--port 8080
# Or serve directly from the Hugging Face Hub
turboquant-serve \
--model manjunathshiva/Nemotron-3-Super-120B-A12B-tq3 \
--port 8080Then call it like any OpenAI-compatible endpoint. The model field in
the request must match the string passed to --model:
curl http://localhost:8080/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "./NVIDIA-Nemotron-3-Super-120B-A12B-BF16-tq3",
"messages": [{"role": "user", "content": "Why is the sky blue?"}],
"max_tokens": 4096,
"temperature": 0.7
}'For Nemotron-3 reasoning models, prefer max_tokens >= 2048 so the
<think> trace and the final answer both fit. mlx-lm splits them into
message.reasoning (the thinking) and message.content (the answer).
From Python via the OpenAI SDK:
from openai import OpenAI
client = OpenAI(base_url="http://localhost:8080/v1", api_key="not-needed")
resp = client.chat.completions.create(
model="./NVIDIA-Nemotron-3-Super-120B-A12B-BF16-tq3",
messages=[{"role": "user", "content": "Why is the sky blue?"}],
max_tokens=4096,
temperature=0.7,
stream=True,
)
for chunk in resp:
print(chunk.choices[0].delta.content or "", end="", flush=True)All mlx_lm.server flags forward unchanged — see turboquant-serve --help
for --host, --temp, --top-p, --prompt-cache-size, etc.
turboquant-serve wraps mlx_lm.server, which knows nothing about multimodal
architectures. For a VLM — Muse Glimmer, DiffusionGemma — use
turboquant-serve-vlm, which drives mlx-vlm's server instead (OpenAI
and Anthropic routes, per-model tool parsers, continuous batching):
turboquant-serve-vlm \
--model manjunathshiva/Muse-Glimmer-30B-tq4-g64 \
--reasoning-strength low --port 8080All mlx_vlm.server flags forward unchanged. Two things are handled for you:
- The reasoning channel is split out of
content. Muse Glimmer answers in a harmony-style channel format, and mlx-vlm's ATEM parser only strips that envelope when a tool call was parsed — so an ordinary turn would hand your agent the model's private deliberation as its reply. The right delimiters are supplied per architecture; the thinking lands inreasoning_contentwhere it belongs. An explicit--thinking-start-token/--thinking-end-tokenstill wins. --reasoning-strengthactually reaches the template. OpenAI clients sendreasoning_effort, but Muse Glimmer's template readsreasoning_strengthand otherwise deliberates at itshighdefault. Requests are translated, and this flag sets the default for clients that ask for nothing — which is most agent harnesses. Measured ontq4-g64: a tool-result turn cost 54 completion tokens atlowversus 106 at the defaulthigh, for the same answer.
Passing --cache-budget-gb routes the loader through the streaming path, so a
MoE whose weights exceed RAM can be served over the OpenAI API — only the
router-selected experts are paged from disk per token. This is how you put a
~50 GB 122B on a 16 GB Mac mini behind Claude Code / Aider:
turboquant-serve \
--model manjunathshiva/qwen3.5-122b-tq3 \
--cache-budget-gb 4 \
--kv-k-bits 8 --kv-v-bits 3 --kv-min-tokens 128 \
--prompt-concurrency 1 --port 8080The Flash-MoE streaming levers ride along:
--max-active-experts (K-reduction, default 4 → ~2× less disk I/O) and
--use-page-cache / --no-page-cache (auto by model-size-vs-RAM — trust-OS is
~2.4× faster decode when the model fits free RAM, F_NOCACHE otherwise). Pair
with --kv-* to compress the growing KV cache of long agentic loops, and
--prompt-concurrency 1 since streaming is a single-user path.
--cache-budget-gb auto sizes the expert cache from the machine (80% of
Metal's recommended working set, minus resident weights and a KV reserve),
and a hot_experts.json shipped in the model directory (the pin output of
calibrate_experts.py) is auto-discovered: those experts are pinned and
preloaded at startup — measured on the 35B ternary, a 1.1 GB / 1,500-expert
profile preloads in 0.3 s and lifts the cache hit rate 74 → 82%, cutting
critical-path disk reads 30%. --no-hotlist opts out; --wire-memory
(opt-in) wires weights + expert cache against memory pressure.
That 35B lift is the roomy case (a large cache budget relative to the experts). The gain shrinks when the cache is a small fraction of a much larger expert set — on a genuinely disk-bound 122B it falls to ~4% fewer disk reads and no speed change; see the measured note under Tuning the streaming reader. Pinning helps hit-rate and disk, not throughput; disk bandwidth is the wall.
Note:
mlx_lm.serveris intended for development and local use, not production. It does not implement authentication or rate limiting.
Serving a 50 GB model on a 64 GB Mac (or any TQ model that fills most of RAM on a 48 GB / 96 GB Mac) leaves very little headroom for Metal command buffers and accumulating prompt caches. After 3-4 multi-turn requests the server can crash with:
libc++abi: terminating due to uncaught exception of type std::runtime_error:
[METAL] Command buffer execution failed: Insufficient Memory
(00000008:kIOGPUCommandBufferCallbackErrorOutOfMemory)
mlx_lm.server keeps a persistent prompt cache per role/conversation
to speed up follow-up turns. Each new prompt grows that pool, and once
caches + model weights + decode workspace exceed Metal's wired-memory
budget, the next allocation aborts the process.
Since 0.13, turboquant-serve applies two auto-guards on tight machines
(both no-ops when working-set headroom is ≥ 8 GB):
--metal-cache-limit-gb auto(default) caps MLX's GPU buffer reuse cache after the model loads. During a long chunked prefill the attention workspace has a new, larger shape every chunk, so the unbounded cache grows ~80 MB per 1K prompt tokens — measured on a 16 GB mini serving a resident 12.6 GB 35B, that alone OOM-crashed a 21K-token prefill at 14K while active memory stayed flat. With the cap the same prefill runs flat end to end.--prompt-cache-max-gb auto(default) puts a byte budget on the LRU prompt cache, enforced on every insert. Upstream's--prompt-cache-bytesexists but is only applied on the batched serving path — TurboQuant KV-quant serving is sequential, where an agent conversation retained 8 KV states (1.14 GB) on the mini, paged the system, and got the server killed by the GPU watchdog. Evictions are cheap when--disk-cacheis on (states restore from disk).
On 16 GB machines also pass --prefill-step-size 256 (or 128 near the
context ceiling): the per-chunk prefill workspace scales with chunk size,
and mlx-lm's default of 2048 OOMs tight boxes on the first chunk.
For bigger machines serving near the ceiling, the two manual levers:
1. Raise Metal's wired-memory ceiling (biggest lever, requires sudo, resets on reboot):
# 64 GB Mac → leave ~7 GB for macOS
sudo sysctl iogpu.wired_limit_mb=57344
# 48 GB Mac → leave ~5 GB for macOS
sudo sysctl iogpu.wired_limit_mb=43008To make it permanent, append iogpu.wired_limit_mb=57344 to
/etc/sysctl.conf.
2. Cap the prompt cache (works without sudo, evicts oldest cached prompts to stay under the cap):
turboquant-serve \
--model ./NVIDIA-Nemotron-3-Super-120B-A12B-BF16-tq3 \
--port 8080 \
--prompt-cache-bytes 2147483648 # 2 GB hard capTighter caps (536870912 = 512 MB, or --prompt-cache-size 1 to keep
only one sequence) trade follow-up prefix-cache speedup for stability.
Recommended combo for Nemotron-3-Super-120B-A12B-tq3 on a 64 GB Mac:
sudo sysctl iogpu.wired_limit_mb=57344
turboquant-serve \
--model ./NVIDIA-Nemotron-3-Super-120B-A12B-BF16-tq3 \
--port 8080 \
--prompt-cache-bytes 2147483648Also close any other GPU users (Chrome/Electron apps, Final Cut, Xcode simulators) before launching — even an idle Chrome can be holding 1-2 GB of unified memory.
--prompt-cache-bytes caps how much the reuse pool across requests can
grow, but each in-flight request still holds an fp16 KV cache that
scales with context length. On a memory-constrained box — e.g. a streaming
120B on 16 GB driven by an agentic loop (Aider, Claude Code) whose prompts
grow every turn — that per-request cache is usually what runs you out of
memory first. turboquant-serve adds the same KV-quant flags as
turboquant-generate to shrink it ~4x in place:
turboquant-serve \
--model manjunathshiva/Nemotron-3-Super-120B-A12B-tq3 \
--port 8080 \
--kv-k-bits 8 --kv-v-bits 3 \ # mixed-precision KV (recommended default)
--kv-min-tokens 128 \ # keep first 128 tokens fp16 (attention sinks)
--prompt-concurrency 1| Flag | Meaning |
|---|---|
--kv-k-bits N / --kv-v-bits M |
Mixed-precision KV; K8/V3 is the recommended default |
--kv-bits N |
Symmetric K=V=N (legacy; not recommended below 3) |
--kv-min-tokens N |
Keep first N cached tokens in fp16 (sink protection) |
--kv-group-size G |
Hadamard rotation group size (default 64) |
--kv-fused |
Use the fused decode+attend Metal kernel (experimental; needs --kv-min-tokens 0; see Fused decode+attend) |
These are processed by turboquant-serve and stripped before the remaining
flags forward to mlx_lm.server. See KV Cache Compression
below for how to choose bit-widths and the speed/quality trade-offs.
Single-stream when KV-quant is on. TurboQuant KV caches don't support the cross-request
mergethatmlx_lm.server's batch generator needs, so enabling any--kv-*flag makes the server serve requests sequentially. That's the right trade-off for a single-user setup; for a multi-client server it means concurrent requests queue rather than batch.
--disk-cache [DIR] makes the prompt-prefix cache a disk citizen: KV-cache
checkpoints are written per conversation (background writer, LRU byte-budget
eviction, default 10 GB) and any later request — including the first one
after a server restart or crash — resumes from the longest saved token
prefix instead of re-prefilling from scratch. Measured: a 16.3K-token
conversation's first turn after restart drops 21.1 s → 3.0 s on a 64 GB
Mac (6.9×, output byte-identical), and on a 16 GB mini a crashed 21K-token
prefill resumed from token 18,432 (249 MB checkpoint loaded in 2.3 s).
Checkpoints are also taken mid-prefill every --disk-cache-save-every
tokens (default 1024). That ladder is what makes persistence work for
hybrid GDN/Mamba models whose recurrent state can't be trimmed: chat
templates re-render the assistant turn differently in history (e.g. Qwen's
empty <think> block), so the end-of-request checkpoint is never an exact
prefix of the next turn — the ladder restores from the newest checkpoint
below the divergence instead. Measured on the mini: turn 2 of a 21K
conversation reused 20,480 of 21,250 tokens on a non-trimmable cache
that previously got zero.
Agent harnesses need temperature (greedy decoding perseverates across turns
on low-bit builds), but sampled structure is where those builds fabricate
tool calls. --tool-syntax-greedy masks logits to argmax only while the
generation is inside a <tool_call>...</tool_call> block — braces, keys,
tags — and leaves the sampler in charge of JSON value strings and of the
decision to call a tool at all. Tags are configurable via
--tool-syntax-tags "<open>,</close>".
The asymmetric-expert build Qwen3.6-35B-A3B-tq3a-tqTe-down4-g64 (ternary up/gate + 4-bit down_proj, 12.6 GB) is the smallest 35B that survives an agent loop — validated end-to-end on a base M-series 16 GB Mac mini:
sudo sysctl -w iogpu.wired_limit_mb=14336 # 14 GiB; 13824 is enough below ~16K context
turboquant-serve \
--model manjunathshiva/Qwen3.6-35B-A3B-tq3a-tqTe-down4-g64 \
--host 0.0.0.0 --port 8080 \
--kv-bits 8 --tool-syntax-greedy --disk-cache \
--prefill-step-size 128 \
--temp 0.7 --top-p 0.8 --top-k 20 \
--chat-template-args '{"enable_thinking": false}' \
--prompt-concurrency 1Measured on the mini (model on an external SSD): 15.2–15.6 tok/s decode
at 13.6 GB peak (flat to 1800 generated tokens), 21K-token cold prefill
~234 s (~90 tok/s), and the Opencode fix-the-failing-test task passes
3/3 (3:37 / 2:25 / 1:54 — faster each run as the disk cache warms; the
identical task fails 0/4 on the pure-ternary build). Practical context
ceiling is ~16–18K at the default wired cap; the 128-token prefill step +
14 GiB wired configuration above is what carries 21K+. Point Opencode (or
any OpenAI-compatible harness) at http://<host>:8080/v1 with model id
default_model, keep temperature ~0.7, and disable auto-discovered skills.
TurboQuant KV cache compression applies the same Hadamard rotation + Lloyd-Max codebook pipeline to KV vectors at runtime. The compressed cache is dequantized to float16 only when attention needs it, so it routes through MLX's standard scaled_dot_product_attention and is compatible with attention sinks, sliding windows, and linear attention layers.
from turboquant_mlx.layers import convert_cache_to_turboquant
from mlx_lm.models.cache import make_prompt_cache
# 1. Build per-layer cache (correct types for hybrid models)
cache = make_prompt_cache(model)
# 2. Convert to TurboQuant KV cache (v0.2 mixed K/V + sink protection)
cache = convert_cache_to_turboquant(
cache,
k_bits=8, v_bits=3, # K-precision-critical, V tolerates 3-bit
min_tokens_before_quant=128, # keep first 128 tokens fp16 (attention sinks)
group_size=64,
)
# 3. Process the prompt and generate — cache is compressed from token 128+
model(prompt_tokens, cache=cache)
for token in generate_loop(model, cache):
...v0.1 → v0.2 migration:
tq_bits=3still works (symmetric K=V=3) but is not recommended on TurboQuant-quantized weights. Passk_bits=8, v_bits=3instead. Pre-existing checkpoints and code paths are fully backward compatible.
K precision matters far more than V precision: softmax amplifies any K error, while V tolerates aggressive quantization. Mixed K8/V3 is the new default.
| Weights | K bits | V bits | sink | When to use |
|---|---|---|---|---|
| FP16 / BF16 | 8 | 3 | 128 | Default — lossless quality, ~4× smaller cache |
| FP16 / BF16 | 4 | 3 | 128 | More aggressive; small quality dip on dense attention |
| TurboQuant-quantized | 8 | 3 | 128 | Required on tq3 weights — symmetric K3 collapses past ~1k generated tokens |
| Any | 8 | 4 | 128 | Highest fidelity TQ KV setting |
Why K8 specifically on TurboQuant weights: stacking 3-bit K cache on top of
already-3-bit weight quantization compounds the noise enough to break long-form
generation on GPT-OSS-20B (we observed total output collapse past ~800 tokens
with K3_V3 on tq3 weights, while K8_V3 is clean). The same K3_V3 cache is
fine on stock fp16 weights — the failure mode is co-compression, not the cache
alone.
turboquant-generate exposes the same controls:
turboquant-generate --model ./model-tq3 --prompt "..." \
--kv-k-bits 8 --kv-v-bits 3 \
--kv-min-tokens 128 \
--kv-group-size 64| Flag | Purpose |
|---|---|
--kv-bits N |
Symmetric K=V=N (legacy v0.1) |
--kv-k-bits / --kv-v-bits |
Mixed precision (v0.2 recommended) |
--kv-min-tokens N |
Keep the first N cached tokens in fp16 (sink protection) |
--kv-group-size N |
Hadamard rotation group size (default 64) |
Whether KV compression speeds up or slows down decode depends on the per-token KV cache size, not the parameter count. When the per-token KV is large (many KV heads and/or long context), its 4x smaller footprint cuts memory bandwidth more than dequant adds, and decode is faster than FP16. When it is small (few active params, short context), dequant overhead dominates and compression is slower — a pure memory optimization.
| Model | FP16 KV | TQ 3-bit KV | Direction |
|---|---|---|---|
| GPT-OSS-20B | 90.6 tok/s | 29.9 tok/s | TQ is 3x slower |
| Qwen3.6-35B-A3B (3B active) | 52.0 tok/s | 45.7 tok/s | TQ is 1.1x slower |
| GPT-OSS-120B | 6.4 tok/s | 8.7 tok/s | TQ is 1.4x faster |
The penalty also grows with context on small-KV models. A community long-context benchmark on M5 Pro (#14) measured Qwen3.6-35B-A3B-tq3-g32 decode at four context lengths (256 gen tokens, except 63K which used 128):
| Context | KV Config | Prompt t/s | Decode t/s | Peak Metal | Memory Saved |
|---|---|---|---|---|---|
| ~65 tok | fp16 | 47.8 | 52.0 | 18.13 GB | — |
| ~65 tok | K8/V3 | 76.1 | 45.7 | 18.12 GB | 0.01 GB |
| ~2.5K tok | fp16 | 131.9 | 51.5 | 20.50 GB | — |
| ~2.5K tok | K8/V3 | 133.4 | 38.2 | 20.55 GB | -0.05 GB |
| ~14.5K tok | fp16 | 132.8 | 46.0 | 22.50 GB | — |
| ~14.5K tok | K8/V3 | 132.6 | 16.7 | 21.94 GB | 0.56 GB |
| ~63K tok | fp16 | 124.5 | 34.5 | 30.79 GB | — |
| ~63K tok | K8/V3 (no sink) | 123.6 | 5.2 | 28.71 GB | 2.08 GB |
The fp16 decode advantage widens with context — 1.14× at 65 tokens → 1.35× at 2.5K → 2.75× at 14.5K → 6.6× at 63K. So on small-active MoEs, use KV compression to fit longer contexts in less RAM (2.08 GB saved at 63K) — not to speed them up. The flip to faster shows up only on GPT-OSS-120B (8.7 vs 6.4 t/s). The similarly-sized Qwen3.5-122B does not flip — run resident on a 64 GB M4 Max, fp16 KV beats mixed K8/V3 at every context (1.07× at 256 → 1.20× at 4096; #19) — so the speed-up isn't a general 120B-class property; it's specific to GPT-OSS-120B's KV geometry.
The "speed flip" penalty above is almost entirely the dequantize step: at
each decode token the compressed cache is expanded to fp16 before
scaled_dot_product_attention runs, and at long context that materialization —
not the attention math — is the wall. The fused kernel removes it.
turboquant_mlx.kernels.kv_decode_attend is a hand-written Metal kernel that
reads the packed cache directly and runs a FlashAttention-style
online-softmax pass, decoding K/V on the fly and never building the fp16
tensors. The inverse-Hadamard rotation is kept outside the kernel via an
orthonormal identity (rotate the query once, un-rotate the output once), so the
per-token rotation cost collapses too. It is numerically equivalent to the
dequantize+SDPA path (cosine 1.0 vs it, same fp16-rounding error as always).
Opt in per prompt-cache — it patches the model's SDPA seam and flips a flag on
each TurboQuantKVCache:
from turboquant_mlx.layers import convert_cache_to_turboquant, enable_fused_attend
from mlx_lm.models.cache import make_prompt_cache
cache = convert_cache_to_turboquant(make_prompt_cache(model),
k_bits=8, v_bits=3, group_size=64)
enable_fused_attend(cache) # decode steps now use the fused kernelMeasured decode speedup over the standard TQ-KV (dequant+SDPA) path, greedy, byte-identical output:
| Model | Context | TQ-KV shipped | TQ-KV fused | Speedup |
|---|---|---|---|---|
| Llama-3.2-1B (D=64, full-attn) | 4.2K | 70.5 t/s | 105.2 t/s | 1.49× |
| Qwen3.6-35B-A3B (hybrid, 10/40 full-attn) | 2.1K | 40.2 t/s | 43.9 t/s | 1.09× |
The isolated attention-layer speedup grows with context (≈10× over the shipped dequant path at 32K in a per-layer microbenchmark); the end-to-end gain scales with the full-attention-layer fraction × context length, so hybrid MoEs (Qwen3.5's GatedDeltaNet layers are untouched) see less than dense models.
Or enable it while serving with --kv-fused (see the
serve KV flags):
turboquant-serve --model ./model-tq3 --kv-k-bits 8 --kv-v-bits 3 \
--kv-min-tokens 0 --kv-fusedScope / limits. Decode only (S_q=1), batch 1, single-tier
(min_tokens_before_quant=0, no fp16 sink window), bit-packed K/V, and equal
K/V head dims that are a multiple of 32. Prefill and any non-applicable step
transparently fall back to the standard path. The kernel performs plain causal
attention, so on a layer that needs attention sinks or a sliding-window mask it
gracefully falls back to the standard dequantize+SDPA path for that step
(correct output, just without the fused speedup) — so it is safe to enable on
any model, including GPT-OSS. (Sink/sliding layers mostly use RotatingKVCache
and never convert to TurboQuantKVCache; the fallback covers the GPT-OSS
full-attention-with-sinks case.)
| Feature | Supported | Notes |
|---|---|---|
| Attention sinks | Yes | GPT-OSS sink vectors flow through standard SDPA |
| Sliding window attention | Yes | RotatingKVCache layers are left untouched |
| Linear attention | Yes | ArraysCache (Qwen3.5 GatedDeltaNet) is left untouched |
| Hybrid architectures | Yes | Per-layer cache type is preserved |
| Prompt-first conversion | Yes | Process prompt with FP16, convert before generation |
| Fused decode+attend | Experimental | Opt-in (enable_fused_attend / serve --kv-fused); standard causal decode, graceful dequant fallback on sink/sliding layers (see above) |
Hardware: Apple M4 Max 64GB (or any Apple Silicon with 16GB+ unified memory at 3-bit)
python -m turboquant_mlx.convert \
--hf-path openai/gpt-oss-20b \
--mlx-path ./gpt-oss-20b-tq3 \
--bits 3 --group-size 32Model size: 9.3 GB (vs 12.8 GB MXFP4 original — 28% smaller, lower perplexity)
The converter automatically:
- Detects MoE architecture (SwitchLinear / QuantizedSwitchLinear layers)
- Dequantizes MXFP4 expert weights to float
- Applies Hadamard rotation + Lloyd-Max codebook quantization
- Keeps router weights and attention at full precision
- Handles blockwise Hadamard for 2880-dim experts (2880 = 9 x 320)
turboquant-generate \\
--model ./gpt-oss-20b-tq3 \
--prompt "Explain quantum entanglement to a 10-year-old." \
--max-tokens 256Expected: ~73 tok/s generation, ~85 tok/s prefill on M4 Max
python -m turboquant_mlx.evaluate \
--hf-path openai/gpt-oss-20b \
--bits 3 \
--no-affine --no-qjl \
--num-samples 64 --seq-len 512| Method | Bits | Size | Peak RAM | Gen Speed | Quality |
|---|---|---|---|---|---|
| Affine (mlx-lm) | 4 | 11.2 GB | ~14 GB | 148 tok/s | Coherent (but see note below) |
| TurboQuant | 4 | 11.2 GB | ~14 GB | — | Best (PPL 72.63, beats MXFP4) |
| TurboQuant | 3 | 9.3 GB | ~12 GB | 73 tok/s | Recommended (PPL 78.60, beats MXFP4, coherent) |
| TurboQuant | 2 | 7.5 GB | ~10 GB | — | Poor (incoherent generation on pre-quantized models) |
Speed vs quality tradeoff: Affine 4-bit is ~2x faster on the 20B model due to simpler dequantization, but TurboQuant 3-bit is 28% smaller with lower perplexity than both affine 4-bit and OpenAI's own MXFP4. Crucially, affine 4-bit cannot scale to 120B on 64GB hardware — TurboQuant 3-bit is the only option there.
# 4-bit (best quality, beats OpenAI's MXFP4)
python -m turboquant_mlx.convert \
--hf-path openai/gpt-oss-20b \
--mlx-path ./gpt-oss-20b-tq4 \
--bits 4 --group-size 32Hardware: Apple M4 Max 64GB — neither the original MXFP4 (63.5 GB) nor the mlx-community 4-bit affine (65.8 GB) fit on a 64GB machine. TurboQuant 3-bit is the only way to run this model on consumer hardware.
python -m turboquant_mlx.convert \
--hf-path openai/gpt-oss-120b \
--mlx-path ./gpt-oss-120b-tq3 \
--bits 3 --group-size 64Model size: 48 GB
Note: The default converter materializes the full quantized model in RAM before saving, so peak memory ≈ the quantized model size (~50–55 GB for a 120B). On a 64 GB machine that caps conversion at ~130B params. For anything larger, add
--streaming: it writes each quantized layer to a shard and frees it, keeping peak memory to ~one 5 GB shard plus the layer being processed — so 200B+ models (Qwen3-235B, DeepSeek-V3) convert on a 64 GB Mac. Output is byte-identical to the in-memory path.
turboquant-generate \\
--model ./gpt-oss-120b-tq3 \
--prompt "Explain quantum computing in simple terms." \
--max-tokens 200Expected: ~44 tok/s generation, ~9.5 tok/s prefill, 52 GB peak memory on M4 Max 64GB
python -m turboquant_mlx.evaluate \
--hf-path openai/gpt-oss-120b \
--bits 3 \
--no-affine --no-qjl \
--num-samples 32 --seq-len 512| Method | Bits | Size | Peak RAM | Gen Speed | Fits 64 GB? | Quality |
|---|---|---|---|---|---|---|
| mlx-community 4-bit | 4 (affine) | 65.8 GB | — | — | No | — |
| MXFP4 (original) | 4 (mxfp) | 63.5 GB | ~70 GB | — | No | — |
| TurboQuant | 3 | 48 GB | 52.3 GB | 44 tok/s | Yes | Coherent, well-structured |
| TurboQuant | 2 | 32 GB | 34.9 GB | 51 tok/s | Yes | Incoherent after ~20 tokens |
Neither the original MXFP4 format (63.5 GB) nor the mlx-community affine 4-bit re-quantization (65.8 GB) fit on a 64GB Mac. TurboQuant 3-bit (48 GB) is the only way to run GPT-OSS-120B on consumer hardware — and at 44 tok/s, it's interactive speed. At 2-bit, the model fits easily but generation quality degrades rapidly — 3-bit is the minimum for coherent output on pre-quantized MoE models.
Hardware: Apple M4 Max 64GB — the original BF16 model is ~240 GB. TurboQuant 3-bit compresses it to ~50 GB, fitting on a 64GB machine.
This is a brand-new architecture featuring 256 MoE experts (the most of any model we've tested), hybrid attention (GatedDeltaNet linear attention + standard softmax attention), and thinking/reasoning capability. The model also has a shared expert per layer alongside the routed experts.
python -m turboquant_mlx.convert \
--hf-path Qwen/Qwen3.5-122B-A10B \
--mlx-path ./qwen3.5-122b-tq3 \
--bits 3 --group-size 64Model size: ~50 GB | Conversion time: ~90 seconds
Note: Conversion requires ~55 GB peak memory. Close all other applications before running. The converter uses memory-efficient processing — each expert layer is replaced immediately after quantization with aggressive garbage collection to handle the 256 experts per layer.
turboquant-generate \\
--model ./qwen3.5-122b-tq3 \
--prompt "Why is the sky blue? Explain in simple terms." \
--max-tokens 200Expected: ~26.5 tok/s generation, 55 GB peak memory on M4 Max 64GB
| Method | Bits | Size | Peak RAM | Gen Speed | Fits 64 GB? | Quality |
|---|---|---|---|---|---|---|
| BF16 (original) | 16 | ~240 GB | — | — | No | — |
| TurboQuant | 3 | ~50 GB | 54.9 GB | 26.5 tok/s | Yes | Coherent reasoning with structured thinking |
Qwen3.5-122B-A10B is the largest and most complex model TurboQuant has been tested on: 122B parameters, 256 experts (8 active per token), hybrid GatedDeltaNet + softmax attention, and a shared expert per MoE layer. At 3-bit, the model produces structured reasoning with proper analysis steps — demonstrating that TurboQuant preserves thinking capability at extreme compression.
This 122B model — ~54 GB on disk — also runs on a 16 GB Mac mini via expert streaming (the same mechanism as Qwen3.6-35B-A3B). Only the router-selected experts are paged from disk per token (LRU-cached), so the resident footprint stays well under the machine's GPU wired-memory cap, and output is bit-identical to the fully-resident model. Requires turboquant-mlx-full>=0.4.1.
python -m turboquant_mlx.stream.stream_generate \
--model manjunathshiva/qwen3.5-122b-tq3 \
--prompt "Explain why the sky is blue." \
--max-tokens 128 --cache-budget-gb 4Measured on a base Apple M4 Mac mini, 16 GB:
| Cache budget | Expert hit-rate | Disk read / token | Decode | Peak (mlx) |
|---|---|---|---|---|
--cache-budget-gb 1 |
0% | ~1.78 GB | ~0.6 tok/s | 6.0 GB |
--cache-budget-gb 4 (recommended) |
44.6% | ~0.93 GB | ~1.1 tok/s | 9.0 GB |
On a 16 GB machine the binding limit is the Metal GPU wired-memory cap (~10.5 GB), not total RAM — and the expert cache counts against it, so mlx_peak ≈ 5 GB + cache_budget. --cache-budget-gb 4 is the sweet spot (~9 GB peak, safe margin); higher budgets risk a Metal out-of-memory error. Throughput is disk-bandwidth-bound (~10B active params/token) → ~1 tok/s on a single mini SSD. Slow, but a 122B model running on a 16 GB Mac is the result.
Hardware: Apple M4 Max 64GB to convert; runs fully resident on 64 GB or on a 16 GB Mac mini via expert streaming. Qwen3.6-35B-A3B is a hybrid linear-attention (qwen3_5_moe, qwen3_next-style) + MoE model — 256 routed experts (top-8) + 1 shared, ~35B total / ~3B active. The text-only language model is extracted (the vision tower is dropped during conversion).
A pre-converted 3-bit (group-size 32) model is on the Hub:
→ manjunathshiva/Qwen3.6-35B-A3B-tq3-g32 — ~16 GB on disk; ~60 tok/s at ~18 GB peak when fully resident on a 64 GB Mac.
turboquant-generate \
--model manjunathshiva/Qwen3.6-35B-A3B-tq3-g32 \
--prompt "Explain why the sky is blue." \
--max-tokens 512The model is ~16 GB on disk, so it won't fit fully resident in 16 GB alongside the OS (resident decode peaks ~18 GB). Expert streaming pages only the router-selected experts from disk per token (LRU-cached), keeping resident memory to a few GB. Output is bit-identical to the fully-resident model. (os.pread + macOS F_NOCACHE keep the OS page cache from ballooning while streaming.)
Since v0.5.0 the missing experts for each layer are read in parallel on a thread pool (--prefetch-workers, default 8), hiding SSD latency behind compute — ~1.9× faster decode at a tight cache budget, still bit-identical. Pass --prefetch-workers 1 for the serial baseline.
python -m turboquant_mlx.stream.stream_generate \
--model manjunathshiva/Qwen3.6-35B-A3B-tq3-g32 \
--prompt "Explain why the sky is blue." \
--max-tokens 512 --cache-budget-gb 8| Config | Expert hit-rate | Disk read / token | Decode | Peak RSS |
|---|---|---|---|---|
--cache-budget-gb 2 |
~60% | ~175 MB | ~3.0 tok/s | 3.9 GB |
--cache-budget-gb 8 (recommended) |
91% | ~41 MB | ~4.5 tok/s | 9.4 GB |
A larger cache keeps more experts resident, raising the hit-rate and cutting SSD reads — the throughput limiter when streaming. --cache-budget-gb 8 is the sweet spot on a 16 GB machine; drop to 2 if RAM is tight. Streaming targets the SwitchGLU expert layout used by qwen3_5_moe and the DeepSeek MLA+MoE family (deepseek_v2/v3); the loader auto-detects the model's layer-key prefix.
Note: Qwen3.6 is a thinking-mode model — it emits a reasoning trace before the final answer, so give it a generous
--max-tokens(512+) for tasks that need a concluding answer.
Once the cache policy is reasonable, disk bandwidth is the wall — for MoE decode the LRU + 8-worker parallel-read pool is already near-optimal, so the big levers are faster storage (Thunderbolt/NVMe) and fewer bytes/token (a hybrid build, a bigger --cache-budget-gb), not the read algorithm. A few knobs squeeze the rest:
How much can a smarter cache policy actually buy? (measured, 122B on a 16 GB mini.) The 0.15.0 learning expert cache (
learn_experts=True, default) records which experts your traffic routes to and pins from that profile once mature — the adaptive cousin of a shippedhot_experts.json. On the genuinely disk-bound case it was built for —Qwen3.5-122B-A10B-tq3a-tqTe-g64(30.9 GB,F_NOCACHE, auto budget 4.3 GB), learned pins vs pure LRU over identical prompts — it lifted the cache hit rate 54.0 → 55.9% and cut expert bytes read 84.4 → 81.0 GB (−4.0%), but decode speed stayed inside run-to-run noise (two byte-identical unpinned runs themselves varied ±7%). The reason is structural, and it is the same wall: 4.3 GB of cache against 27 GB of experts is 16%, and 256-expert top-8 routing is too spread for a small pin to beat recency. At ~0.47 GB/token off the SSD, trimming 4% is nothing. The pin keeps better experts and reads a little less disk; it does not make streaming faster. Faster storage and fewer bytes/token do — the learned profile's real value is warm-starting the cache (skipping the cold-record run), not throughput.
| Knob | Default | What it does |
|---|---|---|
--max-active-experts K |
4 |
K-reduction — caps router top_k to min(native, K) per MoE block, so the switch streams fewer experts/token. argpartition selects fewer and norm_topk_prob renormalizes the gates, so it stays a clean reduced-K MoE. On Qwen3.6-35B-A3B (native top-8) K=8→4 is byte-identical on the 6-test stress harness and cuts streamed disk reads ~2.09× (1.4× faster decode in the disk-bound regime); K=2 collapses (broken JSON). 4 = safe floor; 0 = native routing. |
--use-page-cache / --no-page-cache |
auto | Trust-OS — whether expert reads use the OS page cache (vs F_NOCACHE). On a roomy machine where the model fits in free RAM, leaving the page cache on returns LRU-eviction re-reads from warm RAM instead of disk: 2.44× faster decode on the 35B at a small budget (7.58 → 18.50 tok/s), same hit-rate/RSS. Auto-enables only when model files are < 0.6× total RAM, so a 16 GB mini on a 70 GB MoE keeps F_NOCACHE and never thrashes. |
| read-coalescing | on | Merges contiguous missed experts into one os.pread. Bit-identical, free, ~5% faster when disk-bound. No flag. |
--prefetch-ahead N |
0 (off) |
Speculatively prefetch the next N layers' experts (predicted from the previous token's routing) on a background thread. ~+6% on fast NVMe with spare bandwidth; self-disables if the drive proves bandwidth-bound (e.g. a saturated USB bus), so it's safe to set 1. |
--pin-file pin.json |
none | Keep a calibrated hot-expert set permanently resident. Experimental — measured net-negative vs pure LRU on a 122B (static pinning costs LRU's adaptivity). For experimentation only. |
The --max-active-experts and page-cache levers are ports of Flash-MoE's K-reduction and "trust the OS" findings (blueprint: Apple LLM in a Flash), measured on the TurboQuant streaming path.
Generate pin.json (and a co-activation perm.json for the optional stream/repack_experts.py relayout) with python -m turboquant_mlx.stream.calibrate_experts.
Hardware: converts on a 64 GB Mac — or off slow USB storage with v0.6.2+, which forces the disk read ahead of GPU compute so conversion doesn't trip the Metal GPU watchdog — and runs fully resident on a 48 GB Mac with headroom to spare. Qwen/Qwen3.6-27B is a dense (qwen3_5) long-context coder — 64 layers, hybrid attention (48 GatedDeltaNet linear-attention + 16 full-attention layers), head_dim 256, 262K context — that Qwen positions as competitive on SWE-bench Verified / SWE-bench Pro. Being dense, it has no experts to stream: it loads once and stays in RAM, so storage only matters for load time.
A pre-converted 3-bit (group-size 32) build is on the Hub:
→ manjunathshiva/Qwen3.6-27B-tq3-g32 — ~13 GB on disk, ~17.5 GB peak at runtime (fits 48 GB with ~30 GB free for KV), ~14 tok/s decode.
turboquant-generate \
--model manjunathshiva/Qwen3.6-27B-tq3-g32 \
--prompt "Write a Python function that merges overlapping intervals." \
--max-tokens 512 --temp 0.7turboquant-serve --model manjunathshiva/Qwen3.6-27B-tq3-g32 --port 8080Point the IDE's custom OpenAI base URL at http://localhost:8080/v1. Stock mlx_lm.server can't load a TurboQuant model (KeyError: 'turboquant') — turboquant-serve patches the loader so the weights load through the PolarQuant path.
python -m turboquant_mlx.convert \
--hf-path Qwen/Qwen3.6-27B \
--mlx-path ./Qwen3.6-27B-tq3-g32 \
--bits 3 --group-size 32 --streamingNote: Qwen3.6 is a thinking-mode model — it emits a reasoning trace before the answer, so give it a generous
--max-tokens(512+). Only the 16 full-attention layers keep a growing KV cache (the 48 linear-attention layers use a fixed-size state), so it stays KV-light for long coding context; compress further with--kv-k-bits 8 --kv-v-bits 3.
Hardware: converts on a 16 GB Mac mini via --streaming; runs on a 64 GB Mac (expert streaming) or fully resident on 96 GB+. Qwen3-235B-A22B is a qwen3_moe Mixture-of-Experts — 94 layers, 128 routed experts (top-8), ~235B total / ~22B active.
This is a hybrid tq3a-tq2e build: 3-bit attention (the always-on path, kept safer) + 2-bit experts (where the parameters — and the savings — live), routers full precision. The 128-expert / top-8 routing carries enough redundancy to absorb 2-bit experts cleanly — the same reason gpt-oss-120b holds at 2-bit while gpt-oss-20b (32 experts) collapses. Result: ~470 GB BF16 → 70.5 GB (15 shards, 6.7×).
A pre-converted build is on the Hub:
→ manjunathshiva/Qwen3-235B-A22B-Instruct-2507-tq3a-tq2e-g32
python -m turboquant_mlx.convert \
--hf-path Qwen/Qwen3-235B-A22B-Instruct-2507 \
--mlx-path /Volumes/SSD/qwen3-235b-tq3a-tq2e-g32 \
--bits 3 --mlp-bits 2 -g 32 --streaming--mlp-bits 2 drops experts to 2-bit while --bits 3 keeps attention at 3-bit; --streaming writes each quantized layer to a shard and frees it, so the full 235B converts in ~8–12 GB of RAM — it was produced on a 16 GB Mac mini in ~18 minutes. Point --mlx-path at a drive with ≥70 GB free.
python -m turboquant_mlx.stream.stream_generate \
--model manjunathshiva/Qwen3-235B-A22B-Instruct-2507-tq3a-tq2e-g32 \
--prompt "Explain why the sky is blue." \
--max-tokens 512 --cache-budget-gb 40A 6-probe stress run passes 5/6: coherent long-form essay, correct multi-step math ($142.80 with a 15% bulk discount), correct memoized Fibonacci, strict-JSON formatting, and clean 1–15 enumeration. The one miss was exact factual recall — an in-context password came back with a single flipped digit (RAVEN-stone-91 → -51). Math/reasoning held; verify outputs where an exact literal value matters.
Need exact recall? The full-3-bit sibling Qwen3-235B-A22B-Instruct-2507-tq3-g32 (3-bit experts, 103 GB) fixes the needle flip and passes 6/6 — at ~1.3 tok/s / 86.3% hit-rate on a 64 GB Mac (slower and bigger than this hybrid; the cost of full 3-bit experts). Pick the hybrid for the smallest footprint, the tq3 build when exact literal recall matters.
| Machine | Cache budget | Expert hit-rate | Disk read / token | Decode | Peak memory |
|---|---|---|---|---|---|
| M4 Mac mini, 16 GB | --cache-budget-gb 6 |
~38% | ~3.2 GB | ~0.2 tok/s | 10.1 GB |
| 64 GB Mac | --cache-budget-gb 40 |
94.1% | ~0.28 GB | ~4–6 tok/s (warm) | 46 GB |
On 64 GB a 40 GB cache holds ~60% of the ~67 GB of experts, but temporal locality lifts the hit-rate to 94.1%, so warm decode runs at the compute-bound ~4–6 tok/s. Throughput is bursty: the first generation and tasks that route into a colder slice of experts stall on the SSD until their experts page in. Bump sudo sysctl iogpu.wired_limit_mb=57344 to raise the cache past the ~48 GB default Metal wired cap.
Nemotron-3 is NVIDIA's hybrid Mamba2 + attention architecture. Two variants are tested:
- Nano-4B — dense (Mamba + MLP + attention), 42 layers
- Super-120B-A12B — hybrid MoE (Mamba + 512-expert latent-MoE + attention), 88 layers, ~12B active per token
Both require mlx-lm ≥ 0.31.3 for upstream Nemotron-H support (installed automatically).
# Nano-4B
python -m turboquant_mlx.convert \
--hf-path nvidia/NVIDIA-Nemotron-3-Nano-4B-BF16 \
--mlx-path ./nemotron-3-nano-4b-tq3 \
--bits 3 --group-size 64
# Super-120B
python -m turboquant_mlx.convert \
--hf-path nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-BF16 \
--mlx-path ./nemotron-3-super-120b-tq3 \
--bits 3 --group-size 64Nemotron-3's chat template ends in a <think>\n scaffold that primes EOS as the top-1 logit at the start of the assistant turn. Pass --min-tokens to mask EOS for the first N tokens so the model enters the think phase:
turboquant-generate \\
--model ./nemotron-3-super-120b-tq3 \
--prompt "Why is the sky blue?" \
--max-tokens 200 --min-tokens 50| Model | Bits | Size | Peak RAM | Gen Speed | Quality |
|---|---|---|---|---|---|
| Nemotron-3-Nano-4B | 3 | ~2.2 GB | 4.3 GB | 75.6 tok/s | Coherent |
| Nemotron-3-Super-120B-A12B | 3 | ~50 GB | 54.7 GB | 18.7 tok/s | Coherent with structured <think> reasoning (974-token answer w/ self-correction, formulas, formatted structure) |
| Nemotron-3-Super-120B-A12B (hybrid) | 3-attn / 2-experts, gs=32 | ~36 GB | ~40.8 GB | ~27.2 tok/s | Coherent prose, code, format, and long-context recall; math accuracy degraded — see Phase-1 note below |
The standard 3-bit Super-120B (~50 GB) needs ~55 GB peak and only fits a 64 GB
Mac after raising iogpu.wired_limit_mb. For users on a 64 GB Mac who want
headroom for other applications — or for users on 48 GB Macs — there is
a hybrid quantization that keeps attention at 3-bit (where precision
matters most) and pushes experts to 2-bit (where the bulk of the weights
live), at a smaller group size (g=32) that improves per-group fit.
Pre-converted model on Hugging Face:
hf download manjunathshiva/Nemotron-3-Super-120B-A12B-tq3a-tq2e-g32 \
--local-dir ~/models/nemotron-3-super-120b-tq3a-tq2e-g32→ manjunathshiva/Nemotron-3-Super-120B-A12B-tq3a-tq2e-g32
on the Hub: ~36 GB on disk, ~40.8 GB peak memory, ~27.2 tok/s decode, fits the
default 48 GB iogpu.wired_limit_mb cap.
Or convert from BF16 source yourself:
python -m turboquant_mlx.convert \
--hf-path nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-BF16 \
--mlx-path ./nemotron-3-super-120b-tq3a-tq2e-g32 \
--bits 2 --attn-bits 3 --mlp-bits 2 --group-size 32For long-form generation, the model needs a small repetition penalty to avoid degenerate tail loops at >1500 tokens. The recommended decode config (empirically validated to keep essay, code, format, and long-context recall clean):
turboquant-generate \\
--model ./nemotron-3-super-120b-tq3a-tq2e-g32 \
--prompt "Why is the sky blue?" \
--max-tokens 4096 --min-tokens 50 \
--temp 0.7 --rep-penalty 1.04 --rep-ctx 256Phase-1 known limitation: math accuracy. Step-by-step arithmetic on the hybrid degrades under any non-zero
--rep-penalty. For numeric/math prompts in Phase 1, omit--rep-penalty(you may see long-gen tail loops on long prompts, but the arithmetic will land correctly more often). A permanent fix is planned for Phase 2 — likely first/last-layer bit protection, a calibration-data codebook, or a fused QJL Metal kernel. Until Phase 2, use the hybrid for prose, coding, format, and long-context tasks; use the standard 3-bit model for serious numeric work.
The fused MoE decode kernel transparently chunks expert routings on long prompts, so this hybrid handles long-context retrieval (e.g. password- recall over 4000+ tokens of context) without the kernel argument-validation crash that affected earlier builds.
Two models share this architecture: XS.2 (33B) and S-2.1 (118B). The port covers both; the licence does not — XS.2 is Apache-2.0, S-2.1 is OpenMDW-1.1. Check per repo before redistributing derived weights.
Laguna-XS.2 is Poolside's 33B
Mixture-of-Experts (256 experts, top-8, 3B active) built for agentic coding
and long-horizon work on a local machine. It is a custom architecture —
mixed sliding-window / global attention in a 3:1 ratio, per-head attention
gating, QK-norm, partial+YaRN RoPE on the global layers, and a sigmoid router
with a selection-only correction bias. mlx-lm has no native laguna
model; the MLX port and loader ship here (registered via compat.py), with
logit-parity of 2e-7 against the transformers reference.
# Convert bf16 -> TurboQuant 3-bit. bf16 (~67 GB) exceeds 64 GB RAM, so use --streaming.
python -m turboquant_mlx.convert \
--hf-path poolside/Laguna-XS.2 \
--mlx-path ./laguna-xs2-tq3-g64 \
--bits 3 --group-size 64 --streaming
# Generate. Laguna's generation_config ships top_p 1.0; ALWAYS pass --top-p 0.9
# or the untruncated 100k-vocab tail injects rare junk tokens.
python -m turboquant_mlx.generate \
--model ./laguna-xs2-tq3-g64 \
--prompt "Write a mergesort in Python." --temp 0.7 --top-p 0.9
# Serve (agentic use). Do NOT pass --kv-bits: 30/40 layers are sliding-window(512)
# so KV is already tiny (peak 16.8 GB @2K -> 19.4 GB @31K); KV-quant saves nothing
# here and can cost up to ~4x decode.
turboquant-serve --model ./laguna-xs2-tq3-g64 --port 8080 \
--temp 0.7 --top-p 0.9 --prompt-concurrency 1 \
--chat-template-args '{"enable_thinking": false}'- Runs on a 32 GB Mac at 13.8 GB on disk / ~16 GB peak; MMLU-Redux 76.9 %, GSM8K 79.0 % (3-bit, no-think), and passes the Opencode observe→read→edit→verify loop.
- Tool calls are GLM-XML, not Hermes-JSON;
mlx-lm'sglm47parser is auto-selected and patched here for a trailing-newline function-name bug. - 3-bit vs 4-bit: this data-free codebook 3-bit build is 4 GB smaller than MLX-native affine 4-bit but ~2× slower per token (codebook decode + online Hadamard rotation, not bandwidth). Prefer it when memory-bound; prefer affine 4-bit for the fastest interactive loop.
Laguna-S-2.1 is the 118B sibling
(256 experts, top-10 + 1 shared, moe_intermediate 1024, 48 layers). At bf16 it
is ~219 GB. Ternary (1.58-bit) experts bring it to ~27 GB, which is the only
build of this model that runs resident on a 64 GB Mac — affine 4-bit (~64 GB)
exceeds Metal's ~56 GB working set and won't load at all, and TurboQuant's own
3-bit (48 GB) peaks at 52.7 GB and starves the OS.
python -m turboquant_mlx.convert --hf-path poolside/Laguna-S-2.1 \
--mlx-path ./laguna-s21-tqTe-g64 \
--attn-bits 3 --ternary-experts --group-size 64 --streaming
# Resident on 64 GB. --no-think matters: Laguna reasons out loud by default, so
# a plain generate call can spend its whole budget deliberating and never answer.
python -m turboquant_mlx.generate --model ./laguna-s21-tqTe-g64 \
--prompt "Write a mergesort in Python." \
--temp 0.7 --top-p 0.9 --max-tokens 1024 --no-think- ~30 GB peak resident, ~12.5 tok/s on an M4 Max, leaving ~25 GB for the OS.
- MMLU-Redux 79.5 %, GSM8K 82.5 % — at 1.58-bit experts this 118B still edges the 33B XS.2 at 3-bit (76.9 / 79.0). The larger, wider expert pool absorbs the deeper quantization.
- Passes Opencode agentic coding at 1.58-bit, where the 512-wide-expert 35B ternary build fails the same harness. Expert width (1024 vs 512), not just count, is what sustains sub-2-bit — see Sub-2-bit ternary experts.
Expert streaming works on Laguna too, so 27 GB resident is not the floor — only 2.28 GB of this model has to stay resident, the other 26.42 GB pages from disk on demand:
python -m turboquant_mlx.stream.stream_generate --model ./laguna-s21-tqTe-g64 \
--cache-budget-gb 4 --max-active-experts 0 --prompt "..." --max-tokens 256Measured on an M4 Max with a 4 GB expert cache: peak 7.3 GB (RSS 6.6 GB),
141 of 144 expert projections streamed, ~5 tok/s, 56.8 % cache hit rate — a
footprint that fits a 16 GB Mac mini. Use --max-active-experts 0 to keep
native top-10 routing (lowering it reads less from disk but changes which experts
run, and breaks agentic tool-calling).
Requires
turboquant-mlx-full >= 0.18.0. Earlier versions matched expert tensors by theswitch_mlpname only, so Laguna'smlp.expertscontainer counted as resident:turboquant-planreported "won't run" on small Macs and--cache-budget-gb autosized itself from a bad figure. Explicit--cache-budget-gbwas unaffected.
Kimi K3 is Moonshot's 2.8-trillion-
parameter MoE: 93 layers (69 KDA linear-attention + 24 gated-NoPE MLA with
q-LoRA), 896 routed experts at top-16 plus 2 shared, and a Stable LatentMoE
where experts run in a 3584-d latent behind shared per-layer down/up projections
while the router reads the full 7168-d stream. mlx-lm has no native kimi_k3,
so the MLX port ships here — contributed by
@anders94 in
#71, verified at
≤ 2.7e-6 fp32 forward parity against transformers.
The published source is 1.56 TB of mxfp4 compressed-tensors, which TurboQuant reads directly (bit-exact unpack, no fp16 round-trip):
pip install "turboquant-mlx-full[kimi]" # tiktoken + blobfile for K3's tokenizer
python -m turboquant_mlx.convert --hf-path ./Kimi-K3 \
--mlx-path ./Kimi-K3-tq3a-tqTe-down4-g64 \
--bits 3 -g 64 --ternary-experts --expert-down-bits 4 --streamingThis model does not fit resident on any Mac you can buy, at any bit width. 2.8T parameters want roughly a 768 GB machine; the cheapest recipe here is 617 GB and 512 GB is the current ceiling. It runs by streaming experts from disk — each token pages in only its router-selected experts:
sudo sysctl iogpu.wired_limit_mb=512000 # 512 GB box; resets on reboot
python -m turboquant_mlx.stream.stream_generate \
--model ./Kimi-K3-tq3a-tqTe-down4-g64 \
--prompt "Explain how Rayleigh scattering works." \
--max-tokens 500 --cache-budget-gb auto \
--max-active-experts 8 --prefetch-workers 16 --fanoutMeasured on a Mac Studio (512 GB) with a 506 GB expert cache and the shipped
hot-expert list: 2.3 tok/s at --max-active-experts 8, 1.2 tok/s at
native top-16. Full numbers and the recipe breakdown are in
model_card_kimi_k3.md.
--fanoutmatches how those numbers were measured — it parallelizes each layer's expert reads across the pool instead of reading them serially. It is off by default because it trades away the coalesced serial read path, which wins on bandwidth-bound external storage. It does not self-disable, so measure it on your own disk.
--max-active-experts 8halves K3's native top-16. That is a 2× cut, the ratio validated elsewhere in this repo — but K-reduction changes which experts run and has broken agentic tool-calling on other models. Use--max-active-experts 0for native routing whenever quality matters.
sarvam-30b and later models of that
type convert and run through the sarvam_moe port shipped here (mlx-lm has no
native module). The port is verified against the fp32 reference three ways: full
key/shape coverage of all 7122 source tensors, layer-1 parity at 5.4e-7, and
whole-model logit agreement.
No quantized sarvam build is published, deliberately. At 4-bit this model degenerates on long generations — measured as a rate over repeated trials at temp 0.7 / top_p 0.9, TurboQuant 4-bit degenerated 8/15 (53%) and mlx-lm affine 4-bit 15/15 (100%). TurboQuant is clearly the better of the two and neither is good enough to ship. Use a higher bit width and validate long-form output on your own prompts before relying on it.
Muse Glimmer is Meta Superintelligence Lab's 29.8B-parameter dense multimodal model, built for agentic work on consumer hardware. Architecturally it is Gemma2-style sandwich norms plus gated attention (a sigmoid output gate alongside Q/K/V), a 3:1 sliding(2048)/full attention pattern with NoPE on the full-attention layers, final logit softcapping, and a 1.9B ViT-G/14 perception encoder.
This is the one case where TurboQuant beats MLX's affine quantizer outright on quality, at matched bit width — and does it while quantizing more of the model:
| build | size | PPL ↓ | decode | notes |
|---|---|---|---|---|
mlx-community/Muse-Glimmer-30B-4bit |
19.88 GiB | 4.3798 | 26.1 tok/s | embedding + vision tower stay bf16 |
TurboQuant tq4-g64 |
14.88 GiB | 4.3315 | 9.4 tok/s | everything quantized |
TurboQuant tq3-g64 |
12.53 GiB | 5.0454 | 10.9 tok/s | smallest coherent build |
800-token fixed-passage perplexity, greedy, M4 Max. The affine build is 2.4–2.8× faster to decode — the usual codebook-vs-affine trade. Do not read the tq3 row as a TurboQuant deficiency: the gap there is the 3-bit step, which is exactly what the matched-4-bit control isolates.
Most of the affine build's extra 5 GB is the embedding and the vision tower,
which mlx-vlm declines to quantize. embed_tokens is a NormedEmbedding whose
inherited to_quantized would silently drop its RMS normalization; TurboQuant
quantizes it through a norm-preserving subclass instead.
# mlx-vlm 0.6.12 shipped the Muse Glimmer classes, so this is a plain install.
pip install "turboquant-mlx-full[vlm]"
python -m turboquant_mlx.convert_vlm \
--hf-path meta-models/Muse-Glimmer-30B \
--mlx-path ./Muse-Glimmer-30B-tq4-g64 \
--bits 4 --group-size 64 --quantize-extras --extras-bits 4
python -m turboquant_mlx.generate_vlm \
--model ./Muse-Glimmer-30B-tq4-g64 --prompt "..." --max-tokens 256--extras-bits 4 matters here: the embedding and vision tower are 3.3B
parameters between them, ~3.9 GB at the 8-bit default versus ~1.8 GB at 4-bit.
16 GB Mac mini: yes — measured.
tq3-g64runs resident on a Mac16,10 (macOS 26.5.2) withiogpu.wired_limit_mb=14336, at every prompt length tried up to 5068 tokens:
prompt peak prefill decode 268 tok 13.44 GiB 19.9 tok/s 3.70 tok/s 868 tok 14.13 GiB 20.2 tok/s 3.44 tok/s 2068 tok 13.91 GiB 17.0 tok/s 3.38 tok/s 5068 tok 14.21 GiB 6.0 tok/s 3.56 tok/s Meta's own smallest Apple Silicon artifact is 17.95 GB, text-only and without the drafter — larger than the machine's whole RAM. Two caveats: headroom at 5068 tokens is 0.20 GB, and prefill collapses past ~2000 tokens (20 → 6 tok/s) while decode stays flat, so long prompts are batch work. The smaller alternative does not work — a
tq3a-tq2e-g64hybrid fits at 9.63 GiB but measures PPL 7.5547 (vs 5.0454) and visibly corrupts text, the same way 2-bit experts failed on 32-expert GPT-OSS-20B. 2-bit on a dense MLP is not a usable tier.
Muse Glimmer also ships a DFlash block-diffusion drafter (5.11 GB, 5 layers, block 16, tapping target layers {1,13,25,37,49}). It is not wired up here. Worth noting it is a dense target, which is the regime where DFlash speculation paid off in earlier testing rather than the MoE regime where it did not.
Qwen3.8-27B reports
model_type: qwen3_5 despite the name — a dense 27.8B multimodal model whose
64 decoder layers are 48 Gated-DeltaNet (linear attention) + 16 full
attention, one full layer in every four. Hidden 5120, intermediate 17408,
vocabulary 248,320, plus a 27-block vision tower. mlx-lm 0.31.3 and mlx-vlm
0.6.3 already carry the architecture, so nothing needed porting; conversion
takes about 18 seconds.
Two builds are published, and they target different machines.
| build | size | peak (1.3K prompt) | decode | Opencode agentic |
|---|---|---|---|---|
tq4-g64 |
15.15 GiB | 22.85 GB | 11.4 tok/s | pass (6m12s) |
tq3-mini-g64 |
11.55 GiB | 13.30 GiB | 12.2 tok/s | pass (6m32s)¹ |
tq3-g64, 8-bit extras (unpublished) |
12.88 GiB | 20.60 GB | 9.2 tok/s | pass (6m32s) |
¹ the agentic run was done on the 8-bit-extras tq3; the published tq3-mini
differs only in the extras tier, at an identical 8.0052-vs-8.0551 perplexity.
Both fix a planted off-by-one end to end: run the given pytest command, read the
file, make the minimal edit, re-run to green, explain with file:line. Vision
works on both — shapes, colours, positions and grounded bounding boxes.
Take tq4 if it fits; take tq3-mini if it does not. At 4-bit, tq4 is
faster as well as better than the 8-bit-extras tq3 for 2.3 GiB more — the
same result the speed flip predicts, since the codebook path
only earns its keep at 2-bit and 3-bit's 10-indices-per-uint32 packing costs
more to unpack than 4-bit's clean 8.
The exception is the 16 GB tier, where nothing else reaches.
tq3-mini-g64
is verified on a real 16 GB M4 Mac mini — not projected down from a bigger
machine — at iogpu.wired_limit_mb=14336 and --prefill-step-size 256:
| prompt | peak, 16 GB mini | peak, 64 GB M4 Max* | prefill | decode |
|---|---|---|---|---|
| 220 tok | 12.95 GiB | 13.22 GiB | 20.7 tok/s | 3.9 tok/s |
| 818 tok | 12.96 GiB | 13.05 GiB | 19.0 tok/s | 3.9 tok/s |
| 2,014 tok | 13.14 GiB | 13.30 GiB | 21.0 tok/s | 3.6 tok/s |
| 5,017 tok | 13.61 GiB | 13.81 GiB | 20.6 tok/s | 3.7 tok/s |
* The M4 Max sweep realized 246 / 844 / 2,040 / 5,043 prompt tokens — a constant 26 more than the mini's, worth 1.62 MiB of KV at 64 KiB/token. Read the comparison as close, not as an exact A/B.
Vision passes there too — OCR correct at 350×100 (12.44 GiB) and 700×200
(12.68 GiB). Peaks reproduced identically across two independent sweeps, and
they run 0.09–0.27 GiB below the comparable prompts on the M4 Max — 55–170×
more than that 26-token gap can account for — so the larger machine is the
pessimistic estimate. Reproduce with
benchmarks/bench_mini_qwen38.py; the full
record is in
bench_mini_qwen38_results.json.
Neither 4-bit build reaches that tier: affine 4-bit peaks at 16.16 GiB, more than a 16 GB machine has in total. Decode there is 3.7 tok/s — fine for document Q&A, summarization and vision calls, too slow for interactive chat.
The hybrid stack makes KV very cheap — only 16 of 64 layers cache anything, so 64 KiB/token, or 2.0 GiB at 32K context.
pip install "turboquant-mlx-full[vlm]"
python -m turboquant_mlx.convert_vlm \
--hf-path Qwen/Qwen3.8-27B \
--mlx-path ./Qwen3.8-27B-tq4-g64 \
--bits 4 --group-size 64 --quantize-extras --extras-bits 8
python -m turboquant_mlx.generate_vlm \
--model ./Qwen3.8-27B-tq4-g64 --prompt "..." --max-tokens 256Measured on mlx-lm 0.31.3 with both mlx-vlm 0.6.3 and 0.6.13 — identical
peaks and identical vision results on each, so the [vlm] floor of 0.6.12
reproduces these numbers.
--prefill-step-size is the memory knob for long prompts, and it is a large
one. mlx-vlm only chunks prefill above the step size, so a 1342-token prompt
runs unchunked at the 2048 default:
--prefill-step-size |
peak | prefill |
|---|---|---|
| default (2048 — unchunked here) | 21.28 GB | 162 tok/s |
| 256 | 16.77 GB | 77 tok/s |
−4.5 GB for 2.1× slower prefill. 256 also keeps every TurboQuant layer on the fused Metal kernel, which materializes nothing. This is what makes the 24 GB target work.
With a 248,320-token vocabulary lm_head is 248320×5120 = 1.271B
parameters, the largest matrix in the model. PolarQuantizedLinear has a
fused Metal kernel only up to _QMM_MAX_TOKENS (256) tokens; past that it
dequantizes the weight through full-size intermediates. Measured on exactly that
shape at 3-bit/g64:
| tokens | transient peak |
|---|---|
| 256 | +0.116 GiB |
| 257 | +13.141 GiB |
| 2048 | +13.953 GiB |
The same shape as an 8-bit affine layer costs +1.895 GiB at 2048 tokens — that
is the logits array itself, with no weight-dequant term at all. So qwen3_5
joins muse_glimmer in VLM_SKIP_PATTERNS, and lm_head is left to the
--quantize-extras affine path. End to end on a 1342-token prompt:
| build | size | peak |
|---|---|---|
lm_head affine |
15.15 GiB | 22.85 GB |
lm_head polar |
14.52 GiB | 33.34 GB |
Polar buys 0.63 GB on disk and costs 10.5 GB at runtime — and that 33 GB peak is the difference between fitting a 36 GB Mac and not.
This is easy to miss because mlx-vlm's chunked prefill discards the chunk
forward's return value, so MLX never evaluates the matmul at all. But chunking
only engages above prefill_step_size (default 2048); a prompt of 257–2048
tokens takes the single-shot branch in generate/ar.py, which slices
logits[:, -1, :] and therefore does evaluate lm_head across the whole
sequence. That is an ordinary chat turn, not an edge case.
mlx-community/Qwen3.8-27B-4bit
is the same architecture at the same 4-bit / group-size-64 setting, with no MTP
weights on either side — so this is a like-for-like comparison, run through the
same harnesses and the same server.
The two TurboQuant columns differ only in what the non-polar remainder
(embeddings, lm_head, vision tower) is stored at — --extras-bits 8 versus
4. They are separate builds with separate numbers, so they get separate
columns.
| affine 4-bit | tq4-g64 extras 8 |
tq4-g64 extras 4 |
|
|---|---|---|---|
| on disk | 14.95 GiB | 15.15 GiB | 13.81 GiB |
| bits/weight (polar tier) | 4.500 | 4.251 | 4.251 |
| PPL, 16K tok | 7.2685 | 7.2496 | 7.3043 |
| PPL, 32K tok | 8.3593 | 8.3321 | 8.4061 |
| decode | 29.7 tok/s | 11.3 tok/s | 11.5 tok/s |
| peak, same harness | 16.16 GiB | 17.80 GiB | 16.47 GiB |
| vision battery | 3/4 | 4/4 | — |
| Opencode agentic | pass (6m42s) | pass (6m12s) | — |
Only the extras 8 build is published, and the vision/agentic runs were done
on it; the extras 4 column is the size-matched control.
Take the affine build if you want 4-bit Qwen3.8-27B. It is 2.6× faster to decode at a lower peak, and the perplexity difference is 0.3% — real and stable in ordering across both corpus sizes, but far too small to trade that speed for.
The one place TurboQuant is structurally ahead is storage. At identical bit width and group size, on the same 17408×5120 projection:
| weight | scales | biases | bits/weight | |
|---|---|---|---|---|
| affine 4-bit | 42.50 MiB | 2.66 | 2.66 | 4.500 |
| polar 4-bit | 42.50 MiB | 2.66 | — | 4.251 |
The Hadamard rotation symmetrizes each group, so there is no zero-point to store — 5.5% leaner for free. That margin is a rounding error at 4-bit and the reason the codebook only earns its keep at 2-bit and below (see the speed flip).
Two cautions on reading the table. The vision split is not a capability
gap: affine's single miss is one string (VOLTAGE 47 → VOLTAGE), and a
five-string follow-up probe put it at 4/5, so treat 3/4-vs-4/4 as a tie. And
the agentic wall-clock is not a speed metric — affine decodes 2.6× faster
yet finished slower, because it spent more tool turns.
Reproduce both halves:
# 16K-token column (--chunks 32) and 32K-token column (--chunks 64);
# both records are checked in under benchmarks/
python -m turboquant_mlx.benchmarks.eval_vlm_perplexity \
./Qwen3.8-27B-tq4-g64 ./Qwen3.8-27B-tq4-g64-x4 \
mlx-community/Qwen3.8-27B-4bit --chunks 32
python -m turboquant_mlx.benchmarks.eval_vlm_perplexity \
./Qwen3.8-27B-tq4-g64 ./Qwen3.8-27B-tq4-g64-x4 \
mlx-community/Qwen3.8-27B-4bit --chunks 64
python -m turboquant_mlx.benchmarks.eval_vlm_vision \
./Qwen3.8-27B-tq4-g64 mlx-community/Qwen3.8-27B-4bitturboquant-plan --model <path> at 16K context, per machine:
| Mac | verdict |
|---|---|
| 16 GB | ❌ Will not run. Peak 18.44 GB against 15.46 GB usable — over by 2.98 GB. A dense model has no expert tier to stream, so the 16 GB recipe does not apply |
| 24 GB | --prefill-step-size 256 — that knob measures 16.77 GB peak on a 1342-token prompt, against 21.28 GB at the default |
| 36 GB | ✅ Resident, 12.34 GB headroom |
| 64 GB | ✅ Resident, 39.40 GB headroom |
Speed is the caveat. At ~11 tok/s decode and ~110–160 tok/s prefill an agentic turn takes about six minutes, against 33 s for Laguna-XS.2 at 3-bit. It passes, but this is a model to serve for quality rather than latency.
TurboQuant is a two-stage, calibration-free quantization pipeline:
-
Hadamard Rotation — Multiply weights by a randomized Hadamard matrix, transforming any weight distribution into a near-Gaussian shape. This is data-oblivious (no calibration data needed).
-
Lloyd-Max Codebook — Apply information-theoretically optimal quantization for Gaussian distributions. The codebook is a mathematical constant, precomputed once.
The result: near-zero quality loss at 3-bit, and usable 2-bit quantization where standard affine completely breaks down.
For MoE models, all experts within a layer share the same rotation signs and codebook, keeping storage efficient.
python -m turboquant_mlx.convert --help
Options:
--hf-path TEXT HuggingFace model path or local path (required)
--mlx-path TEXT Output directory (default: mlx_model)
--bits {2,3,4} Quantization bit-width (default: 3)
--group-size {32,64,128} Elements per quantization group (default: 64)
--rotation TEXT Rotation method: hadamard, blockwise_hadamard, none
--use-qjl Enable 1-bit QJL residual correction (+1 bit overhead)
--dtype TEXT Model dtype before quantization: float16, bfloat16
| Architecture | Model Type | MoE | Status |
|---|---|---|---|
| LLaMA / Llama 3 | llama |
No | Tested |
| Qwen2 / Qwen2.5 | qwen2 |
No | Tested |
| Qwen3.5 / Qwen3.8 (text) | qwen3_5 |
No | Tested |
| Qwen3.5-family VLM (hybrid Gated-DeltaNet + full attention, via mlx-vlm) | qwen3_5 |
No | Tested (Qwen3.8-27B, 27.8B dense: tq4-g64 = 15.15 GiB, tq3-mini-g64 = 11.55 GiB verified on a real 16 GB Mac mini; both pass an Opencode agentic task and the vision path). lm_head is kept off the polar path — see Qwen3.8-27B |
| Mistral | mistral |
No | Tested |
| Qwen1.5-MoE | qwen2_moe |
Yes | Tested |
| GPT-OSS | gpt_oss |
Yes | Tested |
| Qwen3.5-MoE / Qwen3.6-35B-A3B | qwen3_5_moe |
Yes (256 experts) | Tested (122B, 35B-A3B); 35B streams on a 16 GB Mac mini |
| Qwen3-MoE | qwen3_moe |
Yes (128 experts, top-8) | Tested — Qwen3-235B-A22B converted to a hybrid tq3a-tq2e build (70.5 GB) on a 16 GB Mac mini via --streaming; streams and passes 5/6 quality probes on a 64 GB Mac |
| Nemotron-H (Mamba/attention hybrid) | nemotron_h |
Yes (512 experts w/ latent MoE on Super-120B) | Tested (Nano-4B, Super-120B) — requires mlx-lm ≥ 0.31.3 |
| Poolside Laguna (SWA + global attn, per-head gating) | laguna |
Yes (256 experts, top-8/top-10) | Tested (XS.2 33B at 3-bit; S-2.1 118B at ternary 1.58-bit — both convert + generate + Opencode agentic pass; expert streaming wired for both). Custom arch — MLX port + loader shipped here (mlx-lm has no native laguna); logit-parity 2e-7 vs transformers |
| DeepSeek-V2 / V3 (MLA + MoE) | deepseek_v2 / deepseek_v3 / deepseek_v32 |
Yes (SwitchGLU experts) | Tested (V2-Lite: convert + resident + streaming, coherent at 3-bit); V3/V3.2 share the MLA+MoE layout and reuse the config (untested — need ~250 GB disk) |
| Kimi K3 (KDA linear-attn + gated-NoPE MLA, Stable LatentMoE) | kimi_k3 |
Yes (896 experts, top-16 + 2 shared) | Tested (2.8T: mxfp4 source → 931 GB tq3a-tqTe-down4-g64, streams from disk on a 512 GB Mac Studio; does not fit resident on any Mac). Custom arch — MLX port shipped here (mlx-lm has no native kimi_k3); fp32 forward parity ≤ 2.7e-6 vs transformers. Needs [kimi] extra |
| Sarvam MoE (Megatron-style fused QKV, DeepSeek-V3 routing) | sarvam_moe |
Yes | Port tested against fp32 (7122/7122 tensors, layer-1 parity 5.4e-7). Custom arch — MLX port shipped here. No quantized build published: 4-bit degenerates on long generations (53% of trials; mlx-lm affine 4-bit 100%) |
| Muse Glimmer (dense VLM: gated attention, sandwich norms, sliding/NoPE mix, ViT-G/14) | muse_glimmer |
No | Tested (29.8B: tq4-g64 beats affine 4-bit on PPL — 4.3315 vs 4.3798 — in 25% less space; tq3-g64 = 12.53 GiB). Structural match verified against the checkpoint, 1436/1436 tensors. Needs mlx-vlm >= 0.6.12 (#1838, merged) — see Muse Glimmer |
| DiffusionGemma (block-diffusion MoE, via mlx-vlm) | diffusion_gemma |
Yes (128 experts, top-8) | Tested (26B-A4B: convert + block-diffusion sampler, coherent at 3-bit — HF). Experimental: decode is much slower than native 4-bit until a batched codebook gather-GEMM kernel lands |
Architectures that live in mlx-vlm rather than mlx-lm convert and run through dedicated entry points (v0.7.0+):
pip install "turboquant-mlx-full[vlm]" # adds mlx-vlm >= 0.6.12
# Convert (vision towers, routers, and known quant-sensitive blocks stay full precision)
python -m turboquant_mlx.convert_vlm \
--hf-path google/diffusiongemma-26B-A4B-it \
--mlx-path ./diffusiongemma-26B-A4B-it-tq3-g32 --bits 3 -g 32
# Generate (runs mlx-vlm's sampler — block-diffusion denoising for DiffusionGemma)
python -m turboquant_mlx.generate_vlm \
--model ./diffusiongemma-26B-A4B-it-tq3-g32 \
--prompt "Write a short paragraph about the ocean." --max-tokens 256turboquant_mlx/
config.py # TurboQuantConfig
convert.py # CLI: HF model -> TurboQuant MLX
generate.py # Text generation with TurboQuant models
evaluate.py # Perplexity evaluation
quantize_model.py # Model traversal & layer replacement
demo_kv.py # Streaming generation demo with KV cache compression
test_kv_cache.py # KV cache roundtrip + integration tests
core/
codebook.py # Lloyd-Max codebooks for Gaussian
rotation.py # Randomized Hadamard rotation
polar_quantize.py # Rotate + codebook quantize
packing.py # Bit-packing into uint32
qjl.py # QJL residual correction
layers/
polar_linear.py # PolarQuantizedLinear (dense)
polar_switch_linear.py # PolarQuantizedSwitchLinear (MoE)
polar_kv_cache.py # TurboQuantKVCache (runtime KV compression)
kernels/
polar_qmv.py # Fused Metal kernel (dense decode)
polar_gather_qmv.py # Fused Metal kernel (MoE shared input)
polar_multi_gather_qmv.py # Fused Metal kernel (MoE per-expert input)
integration/
rotation_configs.py # Per-architecture rotation configs
stream/ # Expert streaming — run MoE models beyond RAM (v0.4.0)
safetensors_reader.py # Per-expert disk slice reads (os.pread + F_NOCACHE; coalesced ranges)
streaming_switch.py # StreamingSwitchLinear + byte-budgeted LRU ExpertCache (+ prefetch/pin)
loader.py # load_streaming(): swap experts to streaming after lazy load
stream_generate.py # CLI: stream-generate (--cache-budget-gb, --prefetch-ahead, --pin-file)
calibrate_experts.py # Routing trace → pin.json (hot experts) + perm.json (co-activation)
repack_experts.py # Optional co-activation on-disk relayout (byte-identical)
@misc{turboquant_mlx,
title={TurboQuant-MLX: Extreme Weight and KV Cache Compression for Apple Silicon},
year={2025},
note={MLX implementation of TurboQuant (Zandieh et al., 2025) for both weight quantization and runtime KV cache compression}
}MIT
- TurboQuant — Zandieh, Daliri, Hadian, Mirrokni (2025)
- MLX — Apple Machine Learning Research
- mlx-lm — MLX language model utilities