perf(claude): cache the fixed prefill, and report the cache counters - #3026
Conversation
A long line in the agent's work log lost its tail. A shell command showed up to its first flags and stopped; a failed tool call showed "permission denied writing to..." and cut off before the sentence that says what to do about it. Turns run 25-160s, so this log is most of what a user looks at while they wait, and the clipped half was routinely the actionable half. The lines wrap now, reusing the transcript's own wrap (components.WrapText) rather than a third wrapping path. The region's height ceiling is unchanged - still min(workLogMaxRows, viewport/2) - so a wrapped line costs older HISTORY, never height: on a 12-row terminal the log still gets two rows. Per-action row caps stop one 300-character command from spending the whole region on itself, and what those caps do cut ends in an ellipsis. Three things fell out of doing it properly: - WrapText only breaks on spaces, so a URL or a Windows path - the lines most likely to run long - had no break to wrap at and overran the pane. Rows wider than the measure are now hard-broken at a column boundary. - Text was being cut to one row's width at CAPTURE time, before the renderer ever saw it: at 74 columns for a narration, at 66 for a tool error's own message, at 60 on the legacy transport. Those bounds now sit where the renderer can still use them, so the wrap decides where a line ends. A tool error keeps its second line, which is where the remedy is. - The old height trim dropped the NEWEST action when it alone exceeded the budget, so a wrapped command on a short terminal rendered an empty region. It is clipped to fit and marked instead. The region's height is also held at each turn's peak. It grows as work happens, which reads as progress, but shrinking dropped the transcript above back down mid-sentence - and wrapping makes those swings bigger. Covered by table-driven tests at 60x12, 80x24, 100x40 and 200x50: tail survival on both event dialects and both failure paths, height within budget and under half the viewport, no row wider than the terminal, one long action unable to fill the region, and the height never shrinking mid-turn.
…de model `gaia run gaia --use-claude --claude-model claude-haiku-4-5` started LemonadeServer.exe anyway, and held the first answer behind a multi-minute `gaia init` before answering anything. The first-boot gate ran `gaia init` on every flagship launch, and `gaia init` auto-starts the local server unconditionally (_auto_start_server, src/gaia/installer/init_command.py). Claude mode only added --skip-chat-model, which skips the model download and nothing else — so the one flag whose purpose is to avoid the local backend was the flag that reliably brought it up. Claude sessions now skip the gate entirely and say on screen that it was skipped, what still needs Lemonade, and how to run it later. Alongside that: - The header names the model — `claude · haiku-4.5`, not a bare `claude` chip. Seeded from the launch flag, because the agent's model-state ping is not read until the first turn, so a session opened and read had no model info at all. - `--claude-model` and `/model <id>` validate against one list shared with the agent's own CLAUDE_MODELS, and a test parses stdio.py so the two cannot drift. An unknown id is refused with the accepted ids instead of reaching Anthropic to come back a 404 mid-turn. - A local `/model` switch with the server known-down is refused with both ways forward, phrased for where the session actually is, and fires once rather than forever — nothing refreshes that cached state between pings, so a sticky refusal would be a dead end no retry could clear. - Typing the space in `/model ` turns the slash palette into a model picker. Local ids stay behind bare `/model`; only the agent knows what is downloaded.
…n" reports Twice in one session a monochrome TUI was investigated as a rendering regression. Both times the cause was the launcher: a cmd.exe window spawned by Start-Process lands in legacy conhost, which reports no colour support, so detectStyle() resolves NoTTYStyle and glamour drops every chroma token. The process emits zero escape sequences — nothing in the renderer is wrong. The skill also said colour could not be judged from a capture. It can: format=ansi returns the raw frame, and counting escapes settles it in one call. - name Windows Terminal as the launch path, with the two wt.exe traps (a spaced --title breaks its parser; a new tab inherits the running Terminal's env, not the shell's, so env vars belong in the .bat) - replace the "cannot judge colour" bullet with the format=ansi check and what zero escapes actually means
…metrics The flagship re-sent ~17,000 tokens of prompt to a 4B model on every LLM call, 2-5 times per turn, so even a trivial question paid tens of seconds of prefill before the model produced a word. This cuts the fixed prefill 53% — 17,014 to ~8,007 tokens — without removing a capability. Three sources, all measured offline with tiktoken (no model contacted): - The tool list went out twice. Gemma takes native tool_calls, so all 66 JSON schemas ship in `tools=`; the system prompt then restated every name and summary in an `==== AVAILABLE TOOLS ====` block — 1,678 duplicate tokens per call. `_compose_system_prompt` now gates that block on the same condition that already gated `_response_format_template`. Non-native models still get it; for them it is the only place the tool names appear. - `gaia-voice` is always on and was 2,145 tokens of rationale prose — every rule followed by the incident that motivated it. Rewritten as instructions: 692 tokens, all 24 behavioural rules intact. - Dynamic tool loading was built, tested, and switched off for the flagship by two independent gates. `FULL_CORE_TOOLS`/`FULL_BUNDLES` now cover all 67 tools under the same union-equality drift guard, and the agent shows the model <=26 per turn instead of all of them. Prompt sections are also reordered static-first. llama.cpp reuses its KV cache only up to the first differing token, and the memory block sat at the very top, so a single `remember()` invalidated ~3,900 tokens that had not changed. `VOLATILE_PROMPT_FRAGMENTS` names the fragments that now compose last. `GAIA_TURN_LOG=<path>` turns on per-turn recording: fixed prefill, input tokens split cached vs new, output tokens, ttft, the wall-time split across model / tools / agent overhead, and absolute timestamps — one JSON object per turn, so two builds are diffable. Off by default. `--dev` draws the same breakdown under each answer in the TUI. Tool timing lives in `_execute_tool_timed`, which the agent loop calls, rather than inside `_execute_tool`: that method is copied onto test stand-ins by attribute assignment and read back with `inspect.getsource` by the coercion contract test, so wrapping it in place breaks both. Not done, and blocked rather than skipped: `gaia eval agent` against the committed baseline, and any post-change latency measurement. Both need Lemonade on Gemma-4-E4B, which is currently banned on this machine; a Claude run is not a substitute for either. docs/plans/gaia-agent-latency.md lists exactly what is owed and how to take it.
Review follow-ups on the wrap change, all in what the log actually shows: - A cut on the live row put its ellipsis after the elapsed clock, where it read as part of the timer. The row budget now reaches renderActivityItem, so the cut is made in the text before the clock is ever appended - and the marker no longer spends two columns of the viewport's edge reserve. - Only the live row carries the clock, but every wrapped row of that action was paying the eight columns it takes. Continuations use the full measure, which gives a long command back about two words a row. - A long argument could push the words that follow it out of its own phrase: "Looking for <query> in your documents" rendered without "in your documents". The argument is budgeted against the template now. - A tool error's second line survives on both error paths, not just the one that goes through a render card. A `summary` stays first-line-only: one that runs to several lines is a payload dump, not a sentence. - The held height pads above the log rather than below, so the blank rows land in the space that already separates it from the transcript instead of opening a gap between the log and the answer streaming under it. Each fix has a test that fails without it - verified by reverting the behaviour and watching the test go red.
…de model `gaia run gaia --use-claude --claude-model claude-haiku-4-5` started LemonadeServer.exe anyway, and held the first answer behind a multi-minute `gaia init` before answering anything. The first-boot gate ran `gaia init` on every flagship launch, and `gaia init` auto-starts the local server unconditionally (_auto_start_server, src/gaia/installer/init_command.py). Claude mode only added --skip-chat-model, which skips the model download and nothing else — so the one flag whose purpose is to avoid the local backend was the flag that reliably brought it up. Claude sessions now skip the gate entirely and say on screen that it was skipped, what still needs Lemonade, and how to run it later. Alongside that: - The header names the model — `claude · haiku-4.5`, not a bare `claude` chip. Seeded from the launch flag, because the agent's model-state ping is not read until the first turn, so a session opened and read had no model info at all. - `--claude-model` and `/model <id>` validate against one list shared with the agent's own CLAUDE_MODELS, and a test parses stdio.py so the two cannot drift. An unknown id is refused with the accepted ids instead of reaching Anthropic to come back a 404 mid-turn. - A local `/model` switch with the server known-down is refused with both ways forward, phrased for where the session actually is, and fires once rather than forever — nothing refreshes that cached state between pings, so a sticky refusal would be a dead end no retry could clear. - Typing the space in `/model ` turns the slash palette into a model picker. Local ids stay behind bare `/model`; only the agent knows what is downloaded.
One-line status prints, tool previews and work-log rows were cut off well before the window's right edge on a wide terminal. The layout now follows the real width, so a 240-column window shows a 238-column tool line instead of 78. The visible cap in logWidth was only half of it: toolNarration and toolResultDetail cut text to 74/66 columns when the event ARRIVED, before any width was known, so widening the layout alone would have shown nothing new. Capture bounds now exist only to stop an agent's runaway payload living in the model; layout decides what prints, from m.width, at render time. Prose deliberately does not follow the window. answerMeasure stays at 88 — for a paragraph, extra columns lose the eye on the carriage return, and a question wrapped to the pane above an answer capped at 88 reads as two unrelated blocks. For a single row, extra columns are simply more of that row, and for the rows whose tail carries the reason or the remedy, the tail is the point. Both constants now cross-reference each other so the split reads as chosen rather than overlooked. Measuring everything against the terminal exposed three rows that measured against nothing: the live line appended its clock after truncating, logWidth's floor of 16 overran any terminal under 20 columns, and the "still working" hint was a fixed 50 columns that wrapped to two rows on a narrow window and broke the height budget that had already been decremented for it. Also bounds setLiveStatus at capture, the one agent-supplied string with no bound at all.
Review follow-up on the terminal-width work. Four rows could still print past the last column, and the test that was supposed to catch them could not. wrapLog raised any measure under 8 columns back up to 8, overriding the caller's budget, so on a 20-column terminal a live row drew 21 and sheared the row below. A measure is what the caller can afford; it is floored at 1 now, never raised. The idle live line — "Getting started", the opening frame of every turn before any tool event exists — went through no measure at all. "Thinking about the next step" drew 39 columns and sheared anything narrower, which is most split panes. It now wraps like every other row. A resize reflows the log, which it could not before this branch: the measure was pinned at 74, so widening from 100 to 240 columns moved nothing. Now it moves 94 to 234, straight into the peak-height hold that exists to stop the log shrinking under a reader mid-sentence. A deliberate resize is not that, and holding the old peak stranded blank rows under the log for the rest of the turn. The hold is released on a width change only — the composer growing a row must not release it. widestLogMeasure now derives from the widest window the TUI accepts anywhere (control/server.go's 20-500 resize range) rather than a guessed 240, so no window this program will lay out can outrun what capture kept for it. The old cramped-terminal test asserted logWidth()+4 fits and never rendered a row, certifying a property the rendered rows did not have. It now draws every row type — idle, live, wrapped, outcome, --dev payload, repeat counter — at every width from the declared minimum of 20 up.
Asked to file an issue, the flagship could only draft one and tell the user to paste it themselves — the `shell:execute:gh` grant was strictly read-only, so `gh issue create` came back as an error. A triage that can never post is half a triage. The grant now sorts every invocation into three tiers instead of two: - ALLOW — reads, unchanged: they run with no prompt, because loading the skill is the consent and a triage is five to ten reads. - CONFIRM — `issue create|comment|edit`, `pr comment`, `label create|edit`. The user sees the exact command and answers yes / no / always-for-that-verb through the prompt the TUI already has. - REFUSE — never runs and never raises a prompt: `auth token`, `alias`, `extension`, `config`, every `gh api` write, `pr merge`, `issue close`, `label delete`, `repo delete`. Keeping the last two apart is the design, not an implementation detail. One prompt covering everything is a prompt users learn to click through, and then the click covers the credential print too. `validate_invocation` keeps its name but narrows to one question — "may this run with nobody asked?" — and answers no for CONFIRM as well as REFUSE, so a caller that has not been taught the third tier fails closed. Two holes found while reviewing the change, both closed here: The action must now be the first token after the subcommand. gh strips a value for any flag it cannot prove boolean at that level, including one it has never heard of, so `gh label -f list create -c FF0000 -R x/y` was classified as the read `gh label list` and ran unprompted while gh dispatched `create`. No allowlist of value-flags closes that — the next gh release adds a flag the table has never seen — so a leading flag is refused instead. A granted CLI is executed as argv rather than a shell string. Every check runs on shlex tokens while Windows handed cmd.exe the original string, so `--search x|echo …` was one argument to the gate and two commands to the shell, and `%VAR%` expanded into a value the approval prompt never displayed. A granted binary is a real executable and needs nothing shell=True provides. Pipelines and every ungranted command keep the old path. Also: the confirmation prompt no longer elides a comment body at 180 characters without saying so — "approve this exact command" has to be true.
… commands portable "Triage my inbox" had no procedure to follow — the skill only knew how to walk a named repo's backlog — so the agent answered from memory instead of calling gh, and invented an answer that looked plausible. - add an inbox procedure that ranks by the notification `reason` field, which is what actually answers "who is blocked on me?" - widen the frontmatter description so inbox phrasing selects this skill at all - one-line, double-quoted commands only: Windows hands the raw string to cmd.exe, where POSIX quoting dies as "The system cannot find the path specified" — an error the agent misread as "gh is not installed" - add a real gh preflight (`gh --version`, `gh auth status`) with per-OS install steps, and a rule that any other failure is a command bug, never a missing gh - trim the author-facing "Fork this" prose, which costs tokens every turn and tells the running agent nothing
# Conflicts: # hub/skills/github-triage/SKILL.md
Every Claude call re-sent the same ~13,700-token system prompt and tool schemas at full price, and the turn metrics could not have revealed it: the provider never set a `cache_control` breakpoint, so nothing was ever cached, and never read the cache fields back off `usage`, so the display would have printed "0 cached" either way. Two breakpoints, not one — end of the tools segment and end of the system block. Caching gives no partial credit, so with only the system marker any drift in the prompt (a memory confidence score, a skill body swapping in) would throw away the tool schemas as well. `usage.input_tokens` is the uncached *remainder*, not the prompt size, so `prompt_tokens` now sums it with the cache reads and writes — otherwise a working cache reads as a prompt that shrank by 98%. The streaming path takes the counters off `message_start`, the only event carrying them. The turn record and the --dev metrics block prefer the backend's own cache accounting wherever it reports any. Lemonade reports none, so the local prefix estimate still drives that display unchanged. Measured on claude-haiku-4-5 through the flagship TUI: a cold turn writes 13,696 tokens, and every turn after it reads them back — 96-98% of input served from cache, on every step of a multi-step turn.
github-actions
Bot
added
documentation
Request changesThis PR does three good things at once — it cuts the flagship agent's per-call prompt roughly in half, adds an opt-in per-turn latency recorder, and upgrades the Four things that change how the model behaves shipped without an agent eval. The system prompt lost ~2,800 tokens, its section order changed, the tool list is no longer restated in prose, and the flagship now shows the model at most 26 of its 67 tools per turn, picked by similarity to the question. None of that is visible to a unit test — a tool the model needed but wasn't shown looks like the model just answering badly. The PR's own plan document lists the eval as owed and blocked on the local backend being unavailable, which is honest, but the project treats this eval as a merge gate specifically because a prompt change that passed every unit test once broke document Q&A in production. Either run it against the committed baseline before merge, or land the observability, prompt-caching and A committed script only works on one person's machine. The new perf test runner has a developer's absolute Windows path and worktree name baked in, and its own safety guard makes it abort for anyone else. Take the path from the environment, or leave the script out of the commit. Real-world evidenceNo 🔍 Technical details🟡 Important1. LLM-affecting surfaces changed with no eval against the baseline Four gates in CLAUDE.md's "Run agent evals when changing LLM-affecting code paths" list are tripped by this diff:
gaia eval agent --category <cat>
gaia eval agent --compare \
tests/fixtures/eval_baselines/gemma-4-e4b-d71cd914/scorecard_<cat>.json \
<run>/scorecard.json2. W='C:\Users\14255\Work\gaia\.claudia-worktrees\claudia-task-25e62f25'
PY='C:\Users\14255\Work\gaia\.venv\Scripts\python.exe'
...
case "$resolved" in
*claudia-task-25e62f25*) ;;The guard is a good idea; the constant it checks makes the script abort for every other contributor and every CI runner. Derive 🟢 Minor3. The write-flag denylist also refuses
4. Stale slot count in the new drift guard ( Says 5. The new elision notice can itself be truncated ( The per-value cut now ends with 6. Sample output in the docs shows the pre-change numbers (
Strengths
|
Approve with suggestionsThis PR cuts the flagship agent's fixed prefill by 53% (17K → ~8K tokens) through four coordinated changes — removing the redundant tool-prose block for native tool-calling models, reordering the system prompt to put volatile fragments last, enabling dynamic tool loading for the full-profile, and trimming the gaia-voice skill body. On top of that it adds Claude prompt-caching breakpoints, a three-tier ALLOW/CONFIRM/REFUSE gate for the gh CLI grant, a shell-injection fix on Windows (granted CLIs now use argv not a shell string), and a dev-mode turn-metrics recorder. The implementation is well-structured and the test coverage is thorough. One 🟡 item must be resolved before this lands; one 🟢 nit is worth fixing. 🟡 The CLAUDE.md-required agent eval has not been run. Four LLM-affecting surfaces changed in this PR — the system prompt lost 2,832 tokens and was reordered, the AVAILABLE TOOLS prose block was removed for native models, and the flagship now sees at most 26 of 67 tools per turn instead of all 66. Per CLAUDE.md, 🟢 🔍 Technical details🟡 Eval — what to run # Terminal 1
python -m gaia.ui.server --port 4200 --host 127.0.0.1
# Terminal 2 — run serially, one category at a time
gaia eval agent --category rag_quality --agent-type doc
gaia eval agent --compare \
tests/fixtures/eval_baselines/gemma-4-e4b-d71cd914/scorecard_rag_quality.json \
<printed-output-path>/scorecard.jsonThe categories relevant to this diff are at minimum 🟢
W="${1:?Usage: runtests.sh <worktree-path> [pytest args...]}"
PY="${VIRTUAL_ENV:-}/Scripts/python.exe" # Windows .venv; adjust for the platformor simply remove it from tracked files and let each developer keep their own local copy. Implementation correctness notes (no action needed)
|
The quality gate failed on this branch. `query`, `agent_name` and `model_id` were passed to TurnRecorder twice — once to the constructor, which ignored all three, and again to finish(). Pylint reported the unused arguments, but the duplication mattered more: the model id was resolved a second time at the end of the turn and could disagree with the model the turn actually ran on. The recorder now holds them from construction and finish() takes only what is new by then. Also drops an `import time` that the module-level import made redundant, gives the per-turn attributes class-level defaults beside their neighbours, and marks the optional `print_turn_metrics` hook's unused argument as deliberate.
|
Verdict: Approve with suggestions — two 🟡 findings; the core implementation is correct, security-sound, and well-tested. 🟡 The eval baseline comparison required by CLAUDE.md is missing for every LLM-affecting change here. The system prompt lost 2,832 tokens, the tool block is now suppressed for native tool-calling models, the prompt order changed (static before volatile), and the flagship now sees at most 26 of 67 tools per ReAct step. CLAUDE.md is explicit: all four of those surfaces require a 🟡 Everything else looks good. The three-tier binary bridge (ALLOW/CONFIRM/REFUSE) is correctly implemented end-to-end: 🔍 Technical detailsEval requirement (🟡 — The plan doc's own "Numbers still owed" table at the end of Phase 2 lists the eval as item 1. The CLAUDE.md rule: "Changes 1, 3 and 4 all touch LLM-affecting surfaces (system prompt, tool schemas) and require Hardcoded paths (🟡 — W='C:\Users\14255\Work\gaia\.claudia-worktrees\claudia-task-25e62f25'
PY='C:\Users\14255\Work\gaia\.venv\Scripts\python.exe'A parameterized version: W="${GAIA_WORKTREE:-$(pwd)}"
PY="${GAIA_PYTHON:-python}"
if [ -z "$GAIA_WORKTREE" ]; then
echo "Set GAIA_WORKTREE to the absolute path of this worktree"; exit 1
fi |
The skill audit returned REVIEW on a high finding — the body reads as instructing the model to conceal an action from the user. The intent was the opposite: the sentence before already requires reporting the refusal openly and naming the command. Phrased as "do not tell the user that approving it is an option", it is genuinely ambiguous. Saying what it means instead — these are refused rather than gated, so no approval exists — keeps the behaviour and clears the finding.
|
Verdict: Approve with suggestions This PR ships three well-engineered improvements: a 53% cut to the flagship's fixed LLM prefill per call, opt-in Anthropic prompt caching, and an upgrade of the 🟡 Eval not run — required by CLAUDE.md before merge. Four LLM-affecting surfaces changed: the system prompt section order reversed (volatile last), the 🟢 🔍 Technical detailsEval gate — which categories matter most The surfaces that changed are exactly the ones
# Replace the two hardcoded lines with:
W="$(cd "$(dirname "$0")/.." && pwd)" # repo root, works on POSIX
PY="${VIRTUAL_ENV}/Scripts/python.exe" # or just `python` on the PATHOn Windows under Git Bash / MSYS the Security: Windows shell=False for skill-granted CLIs — confirmed correct
If two different
|
finish() no longer takes the turn's identity, so tests/unit/test_turn_metrics.py had to move with it. The one test that asserts a distinct query per record now constructs a recorder per turn, which is where the query belongs — it is fixed when the turn opens.
|
Verdict: Approve with suggestions The system-prompt optimization is well-engineered — 🟡 🟡 🔍 Technical detailseval gate — runtests.sh — # Edit these two lines for your machine before running:
W='<absolute path to this worktree>'
PY='<absolute path to your Python executable>'Alternatively, replace with test_optional_tools_are_present_on_a_full_install — |
The prompt-shrinking half of this branch already reached main with the amd#3022-amd#3026 stack, and amd#3026 rebuilt the turn recorder on top of it to add Anthropic cache accounting. Resolved to main everywhere that work landed -- keeping the branch's older copies would have reverted the cache counters, the confirmation-summary disclosure limits, ChatAgent's output_handler, and the execute_python_file confirmation gate. What survives is what never landed: approval time billed separately from tool time, the rendered-prefix ordering fix behind the cached/new split, and the absent-ttft rendering.
) Most of this branch already reached `main` with the amd#3022–amd#3026 stack. It has been merged down to what never landed: **13 files instead of 35, +478/−261 instead of +3738**. Three things the `--dev` turn breakdown reported wrongly. Every one needed a real turn with a real human at the keyboard to appear, which is why the unit suite was green: - A shell command that ran in 1.3s was recorded as **322.6s** — the approval prompt blocks inside the tool call, so the time someone spent deciding was billed to the tool. Approval is now timed separately and shown as its own `waiting on you` figure, leaving `tool_s` and `overhead_s` meaning what they say. - Cache reuse was understated **3.6×**. The proxy prompt appended the tool schemas *after* the conversation, but chat templates inject them alongside the system block at the front, so the shared prefix stopped at the first history message. A turn whose entire 12.2k system+tools header was reusable reported 27%. - A missing time-to-first-token drew as `0.0s`, which reads as an instant first token. It now reads `--`. It also clears the reviewer findings that rode onto `main` unfixed with that stack. The one contributors actually hit: `.perf/runtests.sh` was pinned to one developer's worktree — and leaked a local username — so the guard the plan doc tells people to use aborted for every one of them. **On the eval.** All five reviews blocked on `gaia eval agent`. That gate was about the prompt changes, which are now on `main`, merged without it. Nothing left here touches prompt composition, tool schemas, or model selection, so the gate does not apply to this PR — but **the eval is still owed for what merged**, and it is no longer hardware-blocked. <details> <summary>🔍 Technical details</summary> Two reviewer findings were checked and are **not** valid as written: - The `AVAILABLE TOOLS` assertion in `tests/mcp/test_mcp_cli_to_agent_workflow.py` is fine. The premise was that an agent with no `model_id` falls back to `DEFAULT_MODEL_NAME` and so takes the native path; in fact `Agent.__init__` stores the raw `None` (`agent.py:816`) and only the *client* gets the default (`:848`), so `_uses_native_tool_calls()` is False and the prose block is still emitted. Applying the suggested rewrite would have broken the test — it asserts on `_openai_tools`, which is `None` there. - `os.path.commonprefix` is not C-speed; it is a Python loop like the one it would replace. Measured on a 99K shared prefix: 5.61ms → 4.71ms. Binary search over slice equality gets 0.09ms, so that is what landed, fuzzed against the naive implementation over 200k random pairs. That investigation exposed a real drift risk, now fixed: `_openai_tools` re-derived the predicate that `_uses_native_tool_calls` documents itself as *"the single source of truth"* for, so the schema path and the prose gate could disagree. Both read one predicate now, pinned by a parametrised test. Still open, deliberately not fixed here — `_publish_turn_metrics` is called only from the printed-answer branch, so a turn that burns its step budget shows no `--dev` breakdown. The obvious fix (publish on the tail too) introduces a **cross-turn leak**: `SSEOutputHandler.print_turn_metrics` only stashes, and `print_final_answer` consumes-and-clears, so stashing on a path with no answer event would attach that record to a later turn. Worth its own change. </details> ## Test plan - [ ] `python -m pytest tests/unit/test_turn_metrics.py tests/unit/test_turn_metrics_wire.py tests/unit/test_system_prompt_composition.py tests/unit/test_dynamic_tool_filtering.py hub/agents/gaia/python/tests/test_full_tool_bundles.py` — 154 passed locally - [ ] `cd tui && go build ./... && go test ./internal/ui/chat/ ./internal/event/` - [ ] `bash .perf/runtests.sh tests/unit/test_turn_metrics.py -q --collect-only` prints `[guard] gaia -> …` and collects, instead of aborting; run from outside a checkout it still aborts with exit 2 - [ ] `git diff main...HEAD` touches 13 files and reverts nothing from amd#3026 — the cache counters, the `execute_python_file` confirmation gate, ChatAgent's `output_handler`, and the confirmation-summary disclosure limits are all still main's --------- Co-authored-by: Ovtcharov <kovtchar@amd.com> Co-authored-by: kovtcharov-amd <kalin.ovtcharov@amd.com>
3 participants
Every LLM call the Claude backend made re-sent the same ~13,700 tokens of system prompt and tool schemas at full price — Anthropic prompt caching is opt-in, and nothing in the provider ever asked for it. A ReAct turn is 2-5 calls, so the largest single cost in the product was being paid over and over for bytes that never changed. This turns it on, and fixes the usage parsing that would otherwise have reported the win as zero: the provider read only
input_tokensandoutput_tokens, never the cache counters, so the--devmetrics line printed0 cachedwhether caching worked or not.Measured on
claude-haiku-4-5through the flagship TUI: a cold turn writes 13,696 tokens to the cache, and every turn after it reads them back — 96-98% of input served from cache, on every step of a multi-step turn. Cached input bills at ~0.1x, so the repeated prefix now costs about a tenth of what it did. The cache also survives a TUI restart within its TTL, so relaunching costs nothing.This is complementary to the open token-reduction work (#3007 prefill, #3008 dynamic tools, the gaia-voice trim): those shrink the prefix, this stops paying for it on every repeat. If both land the effects multiply rather than add — a smaller prefix, charged at a tenth, once instead of per call.
🔍 Technical details
Two breakpoints, not one. Anthropic renders
tools→system→messages, so a marker on the system block covers the whole fixed prefill. A second marker at the end of the tools segment is there because caching gives no partial credit: with only the system marker, one byte of drift anywhere in the system prompt would discard the tool schemas too. Block-level markers rather than top-levelcache_control=onmessages.create()— top-level auto-placement marks the last cacheable block, which in an agent loop is the newest tool result, so every request would write a fresh entry and read almost nothing.input_tokensis the uncached remainder, not the prompt size.prompt_tokensnow sums it with the cache reads and writes; left alone, a working cache would have read as a prompt that shrank by 98%. The streaming path takes the counters offmessage_start, the only event that carries them.Metrics source. The turn record and the
--devblock prefer the backend's own cache accounting wherever it reports any (a cold turn that only wrote counts, so turn 1 and turn 2 are on the same scale). Lemonade reports none, so the local prefix estimate still drives that display unchanged — the two sources are recorded separately and never summed.Prefix-stability audit — what still limits the hit rate. Caching is a prefix match, so anything volatile ahead of a breakpoint caps the ceiling. Clean: no timestamp, uuid, session id, pid, or cwd anywhere in the system prompt or tool descriptions (the one
datetime.now()is already correctly prepended to the user message,memory.py:2114— do not "fix" it into the system prompt). Outstanding, in severity order, none addressed here:dynamic_tools=Truewith a 26-slot LRU cap over a ~66-tool registry (hub/agents/gaia/python/gaia_agent/agent.py:162,171) rewrites thetoolsarray per turn once at the cap. Tools render first, so an eviction invalidates everything. Worth considering pinning off under--use-claude: a cached full registry at 0.1x beats an uncached rotating subset at 1.0x.indexed_docs_sectionis volatile but sits ~600 tokens from the top of the system prompt, above ~2,500 tokens of static rules (gaia_agent_chat/profiles.py:141,agent.py:769-842). Indexing one document invalidates everything below it. Moving it toVOLATILE_PROMPT_FRAGMENTShelps llama.cpp too.(confidence: 0.87)per fact (memory.py:2044), and everyrecallbumps confidence by 0.02 and can reorder facts — a guaranteed per-turn byte change for no model benefit.systeminto a static head and volatile tail with a breakpoint between would salvage the static ~2,500 tokens from all of the above.The measured run above hit 96-98% because that session took the full-registry path, so the tools array was byte-stable.
Test plan
pytest tests/unit/test_claude_provider.py tests/unit/test_turn_metrics.py -q— 54 tests, including breakpoint placement on the outgoing request, no mutation of the caller's tool list, cache-field parsing on both the streaming and non-streaming paths, and an older SDK response with no cache fields.cd tui && go test ./internal/ui/chat/ ./internal/event/Full
pytest tests/unit -qwas run and diffed against the unmodified base commit: no new failures (605 vs 610 pre-existing environmental failures — this branch has 5 fewer, none related).Live, two identical consecutive turns:
Before / after
cache_read_input_tokens, straight from the turn log:Not run:
gaia eval agent. This touches an LLM-affecting surface, so CLAUDE.md requires it, and it has not been run — Lemonade is currently banned on this machine (the user's PC crashes), and a Claude-backed run is not a valid substitute for a local-model baseline. It remains outstanding and should be run before merge on a machine where Lemonade is available. Nothing here changes prompt text, tool schemas, or the tool-call envelope — only where the cache breakpoints sit and how usage is parsed — but that is an argument for expecting it to pass, not evidence that it did.