llama.cpp server prompt processing

[kolbma]

I think there is some problem in communication with llama.cpp server or in session/thread handling.
Compared to other clients, lama.cpp needs to do quite often complete prompt (re)processing when agentty is used, like the session would start from the beginning, although I have only added little bit context.
This of course slows down work a lot.
But I have no idea what to look for in agentty to see what's going on.

[Medamineelkhattabi]

I don't think it's slot or session handling. agentty doesn't send cache_prompt or id_slot (llama.cpp defaults to cache_prompt: true and picks the slot itself), and its system prompt for a local server is stable. Reading v0.9.51, the more likely cause is that it rewrites older messages before each request, while most clients only append. llama.cpp reuses the KV cache only up to the first token that differs:

• A tool result over 2 KiB is cut to 2 KiB once it becomes the 9th newest (wire.hpp), so each round the result that just turned 9th changes, and everything after it (roughly the last 8 tool results) is reprocessed.
• Editing a file swaps every earlier read of it for a short pointer (wire_supersede.hpp). An early read means reprocessing from there.
• Auto-compaction fires at min(95% of the window, window minus 20k tokens) (session.hpp), about 12.8k on a 32k context, and replaces the history with a summary.

On SWA or hybrid models (gpt-oss, Gemma 3, Qwen3-Next) a change mid-prompt can mean a full reprocess unless a context checkpoint comes before it.

To confirm, run llama-server with -lv 4 and LLAMA_SERVER_SLOTS_DEBUG=1 LLAMA_SERVER_SLOTS_N_DIFF=40. Each request then prints the old and new tokens around the mismatch (code), and forcing full prompt re-processing when it starts over. In agentty, AGENTTY_LOG=perf=debug logs turn.cache ... hit= per request.

What helps: --cache-reuse 256 (reuses the unchanged chunks after a rewrite by shifting the KV cache; the server warns if the model can't), a bigger -c, --swa-full on SWA models, and keeping --cache-ram on so title, compaction and subagent requests don't cost you the main slot.