Repository navigation
0.11.2
For local models. On Ollama and most llama.cpp builds, which do not report cached tokens, Prefixwise no
longer calls every unchanged turn a routing miss: it says the count is missing and still names every
change to the prompt. Prefixwise inside vLLM with LMCache is validated live, and prefixwise on its
own shows how to start.
-
A server that reports no cached tokens is
unknown, not a routing miss. Ollama and many llama.cpp builds sendprompt_tokenswith noprompt_tokens_details, so their 0 cached means "not reported". Prefixwise read it as nothing reused and called every unchanged turn arouting_miss("reached a machine without the cache"). Now such a turn isunknown, with that evidence; the report says the server leaves the count out and that the hit rate is not known rather than zero; the session view shows?. Changes to the prompt (tool order, a timestamp, an edited message) are still named, from the prompt alone. A server that reports 0 is unchanged. Found by running a new user's first session against an Ollama-style server. -
prefixwisewith no command shows how to start: the five commands most people need, instead of an argument error. A report before any run says to run an agent through Prefixwise first. -
Prefixwise inside vLLM works with LMCache, live. One flag on
vllm serve, no proxy: after a GPU flood every next turn was a hit labelledhost_restored, 5 of 5, as through the proxy; thecanonical_toolsreplay on the same server matched the vLLM-alone run to the token; andprefixwise servedetected vLLM and its event stream from the URL alone, its first live record (record). docs/LMCACHE.md gains the one-flag setup.