v0.6.0 — Cache Lab

Find out why your prompt misses the model's cache, and what that costs you every turn.

Added

Cache Lab — a new page, and a new kind of measurement.

A local model doesn't re-read your whole prompt each turn: it reuses a KV cache of the tokens it has already processed. But that cache survives only on a shared prefix — it holds only for as long as the prompt still matches from the very first token. The moment one token differs, everything after it is thrown away and re-evaluated, however identical it was.

So a single timestamp at the top of a system prompt costs you the entire system prompt, every turn. Measured against a live daemon with the same 324-token prompt:

Where the timestamp sits	Prompt tokens	Reused from cache	Prefill
At the start	324	4	64.6 ms
At the end	324	290	18.6 ms

Same model, same words, same token count. 3.5× the prefill time, purely because of where one value sat.

Cache Lab does three things:

• Diagnoses — finds the values that change between turns (timestamps, dates, ids, long numbers) and renders your prompt in three colours: the prefix the cache can keep, the first value that breaks it, and everything forfeited downstream. That last region is usually most of the prompt, and usually has nothing wrong with it.
• Measures — sends two layouts, each twice, the second send carrying a changed value because that is what real traffic does. It is the second send that is reported; measuring the first would only show how well a prompt matches itself, which is ~100% for any layout and tells you nothing.
• Reports a verdict — tokens saved per turn, milliseconds saved per turn, speed-up. Every number came back from your own daemon. Nothing is estimated.

Sidebar item Cache Lab, route /cache-lab. Guide pages in English and French. Requires Ollama 0.33.3 or newer; on older daemons it shows "not reported" rather than a misleading 0%.

Fixed

README images were broken on npm. Seven screenshots per README used repo-relative paths, which npm could not resolve because package.json declared no repository. All image URLs are now absolute, and repository, homepage and bugs are declared.

Install

npx llmxray

docker pull djovaneli/llmxray:0.6.0