[Feature]: Prompt Cache Hit-Rate for MCP-Heavy Conversations — Stable Breakpoints, Tool Schema Stability, and Cache Observability

[SiinXu]

Background

Cherry implements Anthropic prompt caching via cache_control: ephemeral markers, but the feature is off by default — users must manually enable cacheControl.enabled: true and set a tokenThreshold in provider settings. When MCP tools are added to a conversation, three forces work against the cache: (1) MCP tool schemas are re-fetched from live servers on every request (syncMcpToolsToRegistry), so any schema drift busts the cache prefix; (2) the system prompt's deferred-tool namespace listing regenerates each turn — the code itself carries a // FIXME： maybe break cache comment acknowledging this; (3) cache hit/write tokens are tracked internally but folded into promptTokens for display, so users have no visibility — a code comment explicitly states "users can't see cache hit-rate or audit premium-rate cache writes." A real user (Refraction Hikari via support) reports a 30-40% hit rate with MCP enabled — and the current code behavior matches that number exactly. For heavy MCP users, this means paying full prompt cost every turn while believing cache is working.

Goal

Prompt cache works correctly out-of-the-box for users with MCP tools enabled; cache hit rate is visible in the UI; users can identify which MCPs are cache-hostile and act on the information.

Spec

• Anthropic prompt caching is on by default with sensible thresholds — system prompt always cached when above N tokens; last 2 user/assistant messages cached. User can adjust or disable, but should not need to enable it manually.
• MCP tool schemas are cached per-session rather than re-fetched per-request. A schema fetch happens once at session start and on explicit MCP config change events; not on every model call.
• The system prompt's deferred-tool namespace section is byte-stable for a given tool set — same tools enabled → identical bytes across turns. Changes only when tools are actually added or removed (not when message stream grows).
• Cache hit/write tokens are first-class fields in MessageStats (separate from promptTokens). Per-message indicator shows cache_read / cache_write / no_cache breakdown; per-conversation header shows aggregate hit rate; per-session stats panel shows cumulative input tokens saved via cache.
• MCP settings page surfaces a per-MCP "cache impact" indicator (e.g. "stable" / "occasionally invalidates" / "rebuilds every turn") so users can identify cache-hostile servers.
• The existing // FIXME： maybe break cache and // TODO: use context manager replace middleware markers in the codebase are resolved as part of this work.

Verification

• Fresh install, no manual cache config → start a Claude Sonnet conversation, run 5+ turns of normal chat → cache_read_tokens grows turn-over-turn; UI shows hit rate ≥ 70%.
• Add an MCP tool to the same conversation, run 5+ more turns → cache hit rate stays ≥ 60% (small overhead from tool definitions, but nothing like 30-40%).
• Compare a conversation with caching enabled vs disabled → stats panel shows a clear input-token savings number.
• Change an MCP server's configuration mid-conversation → next request invalidates only that MCP's cache; subsequent requests re-stabilize.
• Inspect Anthropic API responses in a debug view → both cache_read_input_tokens and cache_creation_input_tokens are surfaced as separate UI fields, not collapsed into promptTokens.
• In MCP settings, identify a known cache-hostile MCP (e.g. one with dynamically generated tool descriptions) → its row shows the "rebuilds every turn" indicator; switch to a stable MCP and the indicator changes to "stable".

Related

#15366 #14552