Independent reproducible benchmark on local backends (Ollama + LM Studio): ~51% avg, per-content-type breakdown, and one edge case

[gamerhateyou]

First off — really nice engineering. The thing I appreciate most is how conservative the defaults are: it compresses hard where it safely can (repetitive logs) and leaves code and prose essentially untouched. That honesty in the defaults is rarer than it should be.

I wanted numbers on my own workload rather than from a homepage, so I built a small reproducible harness and ran Headroom 0.27.0 as a proxy in front of two local backends:

• Ollama qwen2.5:14b (WSL)
• LM Studio qwen3-14b (Windows)

Method: a fixed-seed dataset of 5 "needle in a haystack" scenarios — each a noisy tool output with one verifiable fact buried inside. For each scenario I measured (a) tokens saved and (b) whether the model still finds the needle on the compressed context vs the full one. Compression is identical across backends (it happens before the model; tokens counted with tiktoken), so the savings below are backend-independent.

Savings + needle survival

scenario	tokens	saved	needle
App log (200 lines, one 500)	6,527 → 346	−95%	lost
Long conversation (12 turns)	8,903 → 4,767	−46%	found
JSON (docker ps, 12 containers)	5,139 → 4,067	−21%	found
Code (~160 lines Python)	912 → 912	0% (protected)	found
RAG (8 chunks, 1 relevant)	840 → 840	0% (prose kept)	found
Total	22,321 → 10,932	−51%	4/5 vs 5/5 full

Latency (Ollama qwen2.5:14b, warm model, cache invalidated each run, median of 3):

• JSON (−21% tokens): 3.04s → 2.51s (1.2×)
• Long conversation (−46%): 6.01s → 3.02s (2×)
• Log (−95%): 5.38s → 1.28s (4.2×)

On local models the prefill saving is honestly the most tangible win — and compressed logs that used to overflow Ollama's default context window now fit comfortably.

The one edge case. The 200-line log compressed to ~5 kept lines. It correctly kept the ERROR line (right timestamp + endpoint) but dropped the indented stack-trace lines right under it — which is exactly where the cause was (psycopg2.OperationalError: connection pool exhausted). With full context the model named that cause; with the compressed one it honestly said the cause wasn't in the trace anymore. Identical on both backends, so it's information loss, not the model.

To be fair to the design: the compressed log ends with Retrieve more: hash=e42faf1f... — the CCR pointer. So it's not lost, it's recoverable: an MCP client (e.g. Claude Code) can call the retrieval tool and expand it. A plain client (raw Ollama / generic OpenAI app) can't. Might be worth making that asymmetry louder in the docs — aggressive log compression is great for triage, but for deep diagnosis the client needs to be able to re-expand.

Two notes for anyone benchmarking on local models:

1. Ollama's default context window is small and silently truncates long prompts — my first run was bogus because the blob got cut and the model said "you gave me no data". OLLAMA_CONTEXT_LENGTH=16384 fixed it.
2. With simple clients you need --no-ccr-inject-tool on the proxy, otherwise the injected retrieval tool trips them up.

Full write-up with the harness details and raw outputs (Italian, but the numbers/tables stand on their own): https://homelabz.cc/blog/headroom-guida-pratica-installazione-test-reali

(Separately filed the [all]/hnswlib install rollback as #1368.)

Happy to share the harness if it's useful to anyone.

[Amar07Singh]

Hey @gamerhateyou — how's your experience scaling Claude Code with headroom?

Do things start lagging and forcing you into annoying loop-and-reprompt cycles as the code grows, or is it keeping up?

[gamerhateyou]

Thanks Amar — happy to dig in, but I'd steer this back to the thread, since the benchmark and your question are two different paths.

What I measured (#1371) was Headroom as a compression proxy in front of two simple OpenAI-compatible clients — Ollama (qwen2.5:14b) and LM Studio (qwen3-14b) — not Claude Code over MCP. So I honestly can't speak to "scaling Claude Code with Headroom" from data: I didn't test that path.

What the numbers do say, across 5 reproducible needle-in-a-haystack scenarios (fixed seed):

• ~51% avg token reduction (22,321 → 10,932), but very content-dependent: logs −95%, long conversation −46%, JSON (docker ps) −21%, code and RAG prose 0% (protected by default).
• Accuracy: 5/5 full context, 4/5 compressed. The one miss was a log squeezed to −95% — Headroom kept the error line but dropped the indented stack-trace holding the real cause (connection pool exhausted). Same failure on both backends, so it's the missing info, not the model.
• Locally the win is mostly latency, not cost: that −95% log came back 4.2× faster.

The one bit relevant to your Claude-Code question: that dropped detail isn't gone — the Retrieve more: hash=… CCR tool can re-expand it. A dumb client like mine can't call it; an MCP agent like Claude Code can. So the single failure I found is plausibly the one a Claude Code setup would dodge — but I haven't measured it, so I won't claim it. If you have a reproducible Claude-Code-over-MCP harness, I'd genuinely run it and post the numbers.