ai/anthropic: cache the prefix that does not change - #4901
Merged
Conversation
An agent's request is mostly the same request every time. The tools and the system prompt are byte-identical from one turn to the next, and identical again on every round of the tool loop — and none of it was marked cacheable, so all of it was re-sent and re-billed at full input rate on every call. A caller with a hundred tools is carrying tens of thousands of tokens of catalogue per turn that the API would have kept for them. One breakpoint, at the end of the system prompt. A request is ordered tools, then system, then messages, so a single mark there caches the tools as well — marking the tools too would spend a second of the four breakpoints a request gets and buy nothing. All three places that build a request: Generate, the follow-up inside the tool loop, and Stream. The loop is the one that matters most, since it re-sends the whole prefix up to ten times for one question. Below the smallest cacheable prefix the system prompt stays a plain string. Tools count towards that threshold because they sit inside the cached prefix: a one-line prompt with a hundred tools behind it is well worth caching and would never reach the limit on its own.
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
Contributor
There was a problem hiding this comment.
Pull request overview
This PR adds Anthropic prompt caching for the stable “prefix” of requests (tools + system prompt) by marking a single cache breakpoint at the end of the system prompt when the prefix is large enough, reducing repeated input tokens during normal turns and tool-loop follow-ups.
Changes:
- Introduces
cacheableSystemto conditionally emit a cacheable “system” block when prefix size crosses a minimum threshold. - Applies the cached system block to the Anthropic provider’s
Generate(initial + tool-loop follow-up) andStreamrequest construction. - Adds unit tests to pin the caching contract (single breakpoint on system, tools not marked, tools count toward threshold, small prompts remain plain strings).
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 3 comments.
| File | Description |
|---|---|
| ai/anthropic/anthropic.go | Adds cacheableSystem + threshold constant and wires cached “system” into Generate/follow-up/Stream request building. |
| ai/anthropic/cache_test.go | Adds tests validating caching behavior, breakpoint placement, and threshold interactions with tool size. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: aafaacb663
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Review feedback on the caching patch: - Add ai.WithoutCache() / Options.NoCache: caching stays on by default because the dominant caller — the agent tool loop — re-sends an identical prefix every round and repays the cache write within a single Generate, but a one-off caller whose prompt or tools change every request can now opt out, since cache writes bill at a premium. - Cache tool-only prefixes: with an empty system prompt the breakpoint moves to the last tool (on a copied slice and map — the caller's tools are never mutated), so a large stable catalogue without a system prompt is no longer re-billed every call. - Marshal the tools once as a slice for the size estimate instead of once per tool; rename minCacheChars to minCacheBytes and say bytes, since len() counts bytes; start the doc comment with the identifier; correct the below-minimum failure mode (the API quietly declines to cache — it does not refuse the block). Tests cover the tool-only breakpoint (last tool only, no mutation, not marked when a system prompt carries it) and the opt-out.
asim
added a commit
that referenced
this pull request
Aug 30, 2026
…y fixes (#4903) * docs(changelog): record the provider tool-loop, caching, and discovery fixes Add [Unreleased] entries for the three merged provider fixes: the cross-provider tool-execution loop (#4900), Anthropic prompt caching with the WithoutCache opt-out (#4901), and deterministic tool discovery (#4902) — so the next release rolls them into its section. * Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> --------- Co-authored-by: Claude <noreply@anthropic.com> Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Anthropic prompt caching for the request prefix that never changes. An agent's tools and system prompt are byte-identical from turn to turn — and identical again on every round of the tool loop — but nothing was marked cacheable, so the whole catalogue was re-sent and re-billed at full input rate on every call.
One breakpoint, at the end of the system prompt: a request is ordered tools → system → messages, so a single
cache_control: ephemeralmark there caches the tools as well. Marking the tools separately would spend a second of the four allowed breakpoints and buy nothing.Applied at all three request-building sites —
Generate, the tool-loop follow-up (the one that matters most: it re-sends the full prefix up to ten times per question), andStream.Below the smallest cacheable prefix (~1024 tokens ≈ 4096 chars, tools counted since they sit inside the prefix) the system prompt stays a plain string.
Tests
ai/anthropic/cache_test.gopins the contract: a big prefix is marked exactly once on the system block; tools are never marked separately; small/empty prompts stay plain strings; tools count toward the threshold.Verification
git am— authorship preserved (patch by @asim).go test ./ai/anthropic/(incl. the new cache tests),./agent/, and mock provider conformance (6/6) all green.Streamsite passesniltools — confirmed exact: the anthropic Stream path sends no tools.One review note
The
cacheableSystemdoc comment says the API refuses a cache block below the minimum cacheable prefix. My understanding of the API is that it accepts the block and silently declines to cache (a no-op, not an error). The conservative threshold is still worthwhile — no point marking prefixes that can't cache — but the stated failure mode may overstate; left as authored, flagging for the author's call.🤖 Generated with Claude Code
Generated by Claude Code