16 GB bench-in-a-box: independent quality anchors for heavily quantized LLMs

Reproducible benchmark harness + the first independent numbers for PrismML Ternary Bonsai 27B (1.71 bpw, 7.17 GB) vs Qwen3.6-27B UD-IQ2_XXS (9.39 GB) on a single RTX 5060 Ti 16 GB.

This exists because when Bonsai 27B shipped, nobody outside the vendor had published a single independent number. If you have a 16 GB (or 8, 12, 24 GB) card, this repo lets you reproduce the whole suite on your hardware in one night and contribute a results row. The goal is a community table nobody has to take on faith.

Abstract

We benchmark PrismML's Ternary Bonsai 27B against the strongest same-size-class baseline that fits beside it on one 16 GB card, Qwen3.6-27B UD-IQ2_XXS. Both derive from the same base model, so deltas are about the compression, not the corpus. The contribution we care most about is methodological: on AIME26 the winner's margin is almost entirely a function of the thinking budget (28k cap: 0.48 vs 0.28; 60k: 0.87 vs 0.63), yet accuracy on problems that converge is budget-invariant and near ceiling for both models, and knowledge is intact (MMLU-Redux statistical tie, p=0.74). What aggressive quantization damages is not knowing, and mostly not correctness. It is convergence: the IQ2_XXS quant thinks 15-22% longer at the median with a much fatter tail, and hits the budget ceiling 1.5-3.7x more often, a pattern that replicates on all three datasets we ran (AIME26, MMLU-Redux, LiveCodeBench) and is individually significant on each (Fisher p = 0.030 / 0.020 / 0.043) even where the accuracy gap alone is not. A single accuracy number at an unstated budget therefore measures the budget as much as the model, and can manufacture either a "collapse" or a "win" from the same weights. We propose that benchmark rows for thinking-mode quants always report accuracy and cap-rate together at a stated budget ("score@budget"); the harness in this repo does exactly that, and also logs the third axis home labs actually pay for: energy per solved problem (~2.5x apart here, hardware-counter-verified).

The one methodological point this repo exists to make

Report score@budget, not score. Thinking-mode models fail benchmarks two different ways: they don't know, or they don't converge within the token budget. Vendor tables collapse both into one number. The distinction flips the story:

model	budget	AIME26 acc	cap rate	acc when converged
Bonsai 27B ternary	28k	0.480	0.48	0.923
Qwen3.6-27B IQ2_XXS	28k	0.278	0.72	1.000
Bonsai 27B ternary	60k	0.867	0.10	0.963
Qwen3.6-27B IQ2_XXS	60k	0.633	0.37	1.000

The IQ2_XXS quant is not "collapsed": it is token-inefficient. When it converges it is at ceiling (24/24 pooled across budgets on AIME; on MMLU its converged-only accuracy actually beats Bonsai's, 0.907 vs 0.887); it just often does not converge in budget. scripts/ measures both.

Results (RTX 5060 Ti 16 GB, CUDA, Linux; raw JSONs in results/)

metric	Bonsai 27B Q2_0 (ternary, fork)	Qwen3.6-27B UD-IQ2_XXS (mainline)
AIME26 (60k budget, np=1, n=30)	0.867 (3 capped, 1 wrong)	0.633 (11 capped, 0 wrong)
MMLU-Redux (n=342, 8k)	0.871 (6 capped)	0.860 (18 capped)
LiveCodeBench (n=50, release_latest, 14k)	0.520 (23 capped)	0.300 (34 capped)
decode t/s (tg128, best engine each)	44.4	35.8
decode t/s + DSpark Q4_1 drafter	79.1 (single-stream only)	n/a
VRAM @ 4k ctx, f16 KV	7,627 MiB (7.4 GiB)	9,999 MiB (9.8 GiB)
max ctx in 16 GB	262k @ 12.9 GiB (q4 KV)	~100k does not fit, est. (f16 KV)
GPU energy, full AIME26 run	~1,177 Wh (7.2 h)	~2,146 Wh (12.3 h)
energy per solved problem (AIME / LCB)	~45 / 13.8 Wh	~113 / 35.3 Wh
thinking tokens per kWh	~0.86M	~0.61M

Statistics (Wilson 95% CIs, two-sided Fisher exact; scripts/stats_tests.py): knowledge is a statistical tie (MMLU p=0.737); the cap-rate difference is significant on all three datasets (p=0.030 / 0.020 / 0.043); the AIME accuracy gap alone is marginal at n=30 (p=0.072); LCB accuracy p=0.041. The defensible claim is the replicated cap-driven mechanism, not any single scoreline. Capped generations are scored "did not converge within budget", never "wrong".

Energy: NVML board power, single-stream, measured two independent ways (1 s sampler integration

• the GPU's cumulative hardware energy counter); where both instruments covered a run end-to-end they agreed within 0.1%. AIME totals include a flagged 82-minute estimate (telemetry started late; measured floors 954 / 1,908 Wh, same conclusion).

Serving note (concurrency)

llama-server with a shared 32k context pool sustains 8 concurrent streams without errors, but TTFT p95 degrades to 12.7 s (4 streams) / 25.1 s (8 streams) on cold ~3.4k-token prefills, and the drafter cannot help (it forces one slot). Single-stream, agents, batch/offline: excellent. Multi-user latency-bound serving: wrong tool today; no public ternary vLLM path exists yet.

Rig stability

17 h of sustained eval load on a 2013 desktop (i5-4670K, DDR3-1333, PCIe gen3 x8 per card measured under load): zero throttle events across 123,562 telemetry samples, max 78 °C, run-to-run replication within ±0.4% hours apart. The platform is a floor, not a confound: decode is GPU-bandwidth-bound; prefill and cold load reflect the old host and the 5400-rpm disk.

Engines (read before building)

• Ternary Q2_0 CUDA currently requires the PrismML llama.cpp fork, commit 62061f9 (b9591). Build in a CUDA devel container with the GPU attached at configure time: a GPU-less configure caches CUDA_DRIVER=NOTFOUND and later builds keep failing until you wipe CMakeCache.
• Mainline llama.cpp runs the g64 file on CPU/Metal only; CUDA Q2_0 is an open PR (ggml-org/llama.cpp#25707). PQ2_0 loads nowhere at time of writing (invalid ggml type 142).
• The fork runs standard quants ~4.5% slower than mainline (measured on the same IQ2_XXS file: 34.2 vs 35.8 t/s), so each model gets its best engine in headline numbers; same-engine rows are in results/. Mainline pin used here: 12127de.
• Drafter: --spec-type draft-dspark needs --spec-draft-n-max 4 (block size), forces a single slot, and the Q4_1 drafter is both faster and 5.3 GB smaller than bf16 (outputs byte-identical at temp 0, 5/5 prompts).
• EvalScope traps the harness already handles: --limit is per subset (MMLU-Redux = 57 subsets, LiveCodeBench defaults to 29), and the modelscope cache belongs on a big disk (LiveCodeBench pulls tens of GB with extraction debris; a full root filesystem cost us a rerun).

Quickstart

1. Serve both models as OpenAI-compatible endpoints (any engine). For AIME use a single slot (np=1-style) and ctx >= 64k so a 60k generation fits; for MMLU/LCB 2 slots @ 32k works.
2. pip install evalscope (we used 1.9) into a venv.
3. Put your dataset cache on a big disk: export MODELSCOPE_CACHE=/big/disk/modelscope-cache
4. scripts/env_manifest.sh > my-manifest.txt to capture hardware/engine/model identity first.
5. nohup python3 scripts/nvml_sampler.py telemetry.csv & for telemetry from t=0, not after.
6. scripts/run_anchors.sh modelA http://localhost:8092/v1 modelB http://localhost:8093/v1 ./results
7. python3 scripts/digest_anchors.py ./results for the accuracy + cap-rate table.
8. python3 scripts/energy_analysis.py telemetry.csv ... for energy per problem/token.
9. Open a PR with your row (see CONTRIBUTING.md).

Honesty rules (also the contribution bar)

• Budgets are part of the result: never report accuracy without the max_tokens it was measured at, and never without cap-rate. A capped run is "did not converge within budget", not "wrong".
• Raw eval JSONs (reports + reviews) go in the PR; scores nobody can re-derive don't count.
• Same sampling for both models (we used the vendor-recommended 0.7 / 0.95 / 20); any deviation from a model's recommended sampling must be stated.
• Per-card numbers, never averaged across different cards; one manifest per card.
• Negative results and engine gotchas are contributions.

What's in scripts/

script	what
run_anchors.sh	full anchor suite against two endpoints; serialized dataset warmup; evalscope --limit per-subset trap already handled; logs the energy counter at every stage boundary
digest_anchors.py	score + cap-rate + median-tokens table from review JSONLs
cap_verify.py	per-problem transparency table (verify any aggregate by eye)
budget_curve.py	score@budget across runs at different max_tokens
stats_tests.py	Wilson CIs + Fisher exact for the headline tables (stdlib only)
nvml_sampler.py	1 s per-GPU telemetry CSV (works without nvidia-smi, ctypes NVML)
gpu_energy_counter.py	NVML cumulative energy counter (mJ, Volta+): exact per-GPU energy per stage
energy_analysis.py	Wh / J-per-solved / tokens-per-joule from the telemetry CSV
env_manifest.sh	hardware + driver + engine commit + model file sha256 manifest
make_figures.py	regenerates every figure in figures/ from the per-problem digest
bench_phase3.sh	the exact llama-bench matrix behind the speed table: solo / simultaneous / swapped x card x model, plus the mainline cross-check; one card per run via CUDA_VISIBLE_DEVICES
phase3b_drafter.sh	DSpark drafter harness: baseline / Q4_1 / bf16 server runs, per-prompt server-native timings, temp-0 byte-equivalence check
drafter_prompts.jsonl	the 5 fixed prompts (3 reasoning, 2 code) the drafter harness runs
phase3c_fit.sh	resident-VRAM ladder vs context size and KV dtype: loads the server, waits for /health, reads NVML

Appendix: the crow bridge

While both models were loaded we connected them in a blind 40-turn conversation (no system prompt, neutral human-sounding seed, private reasoning logged but never exchanged) and watched for a stable attractor. There is one, it arrives in ~15 turns, and by turn 33 the models exchange a literally identical five-word mantra. Neither ever suspects the other is a machine. Transcript, raw JSONL with both thinking channels, and the bridge script: appendix-crow/.

Provenance & disclosure

Run design, scripts, and analysis were AI-orchestrated (Claude Fable 5) and human-directed, reviewed, and operated; every number is reproducible from the scripts and raw JSONs in this repo. Models tested are the vendors' published GGUF artifacts (sha256 in the manifest).

License: MIT. Not affiliated with PrismML, Alibaba/Qwen, or any vendor.