Cache breakpoint re-anchoring rewrites (instead of extends) the BYOK Anthropic prompt cache in long agent sessions

[willsf1862]

Type

Bug — cost regression (prompt-cache write churn from non-additive breakpoint movement) on the BYOK Anthropic path.

Environment

• VS Code: 1.126.0 (commit 7e7950d)
• GitHub Copilot Chat: <your Extensions → GitHub Copilot Chat version>
• OS: macOS (expected on all platforms)
• Model: BYOK Anthropic (Claude Opus 4.8) via ANTHROPIC_BASE_URL
• Occurs with all other extensions disabled: yes

Summary

Anthropic prompt caching is explicit and capped at 4 cache_control breakpoints per request.
As an agent conversation grows, the Copilot prompt renderer re-distributes those 4 breakpoints —
generally moving the last one toward the end so the bulk of accumulated history stays cached. That
intent is correct. The implementation is not byte-stable: when the breakpoint re-anchors, the
prefix bytes before the new anchor change, so Anthropic treats it as a cache write of the entire
prefix (cache_creation_input_tokens spikes, cache_read_input_tokens drops to 0) instead of an
incremental cache extension of the already-cached region.

The specific trigger observed is a synthetic whitespace anchor block appearing/disappearing. The
renderer injects {"type":"text","text":" ","cache_control":{"type":"ephemeral"}} when the last
content block isn't a valid cache anchor, and removes/relocates it on a later turn. That single
structural change to the cached region forces a full re-write.

Impact

On a real BYOK Anthropic workload (Claude Opus 4.8), over a clean-slate window of 155 consecutive
captured requests:

• ~8.4% of turns were full-prefix cache writes (cache_creation > cache_read), distinct from
  a separate workspace-tree bust (filed as the companion issue).
• These re-write turns appeared on long conversations (request bodies ~1.8 MB) and cost a full
  cache_creation of the entire prefix (observed up to ~560K tokens at 1.25× input rate) where an
  incremental extension would have cost almost nothing.
• Reuse factor (cache_read ÷ cache_creation) on the affected window sat at ~7.9× versus the 20–25× a
  well-tuned Anthropic prefix achieves.

Steps to reproduce

1. Use agent mode with a BYOK Anthropic model and a long-running conversation (many turns, large
   accumulated history).
2. Capture consecutive raw /v1/messages request bodies (e.g. via a transparent local forward proxy
   on ANTHROPIC_BASE_URL).
3. Find a turn where cache_creation_input_tokens spikes and cache_read_input_tokens is 0.
4. Diff that body against the previous turn. The first divergence is inside the message history
   (not the system prompt), at a cache_control marker — typically a
   {"type":"text","text":" ","cache_control":...} block present in one turn and absent/moved in the
   next.

Root cause (from the shipped extension, 1.126.0)

The Anthropic wire-format renderer places cache_control on the last cacheable block and walks
from the end backward to position the (capped) breakpoints:

// place the breakpoint on the last non-deferred block
for (let a = n.length - 1; a >= 0; a--) if (!n[a].defer_loading) { n[a].cache_control = r; break }
// …and a synthetic whitespace anchor when the tail isn't cacheable:
r && c1n(r) ? r.cache_control = {type:"ephemeral"}
: e.push({ type:"text", text:" ", cache_control:{type:"ephemeral"} })

Because the 4-breakpoint cap forces re-distribution as history grows, and the synthetic " " anchor
is added/removed depending on the tail block type, the bytes of the cached prefix change between
turns. Anthropic caches the longest identical prefix, so any change before the anchor
invalidates everything after it → a full re-write rather than an additive extension.

Why "just pin the breakpoint early" is not the fix

Anthropic only caches up to the last breakpoint; everything after it is billed at full input rate
every turn. Pinning the breakpoint early would keep the small static prefix cached but leave the
entire growing history uncached and re-billed at 1× every turn — worse than current behavior in
long sessions. The correct behavior is to move the breakpoint forward additively, preserving the
already-cached prefix bytes, so re-anchoring extends the cache instead of rewriting it.

Requested fix

1. Make breakpoint re-anchoring byte-stable / additive: when moving the trailing breakpoint
   forward, do not alter any bytes before the previously-cached anchor, so Anthropic extends the
   existing cache rather than re-creating it.
2. Specifically, stop inserting/removing the synthetic {"type":"text","text":" "} anchor in a way
   that mutates the cached prefix across turns; use a stable anchor position.
3. Consider pinning one of the 4 breakpoints permanently at the end of the static system/tools
   prefix (that region never changes) and using the remaining breakpoints for the growing tail — so
   the large static prefix is never re-written even when the tail re-anchors.

Workaround

Keep sessions short (start a fresh chat per task). A shorter conversation has a smaller tail and
fewer message blocks competing for the 4 breakpoints, so re-anchoring happens less often and
invalidates less when it does.

Related

• BYOK chat providers receive an internal cache_control LanguageModelDataPart sentinel that's not in the public API #313920 ("BYOK chat providers receive an internal cache_control LanguageModelDataPart sentinel
  that's not in the public API", Open, model-byok) documents the same cache_control /
  "ephemeral" sentinel from a different trigger (the sentinel leaking into serialized tool-result
  text). It states the same root principle this issue relies on:

  "A leaked {"$mid":24,...} substring on one turn changes the byte-exact prefix and misses every
  subsequent turn's cache — defeating the whole point of the sentinel."
  Same sentinel, same byte-stability failure class, two distinct triggers. Relevant first-party
  converter: extensions/copilot/src/extension/byok/common/anthropicMessageConverter.ts.

• Companion issue (workspace-structure tree injected ahead of the cached prefix) — the other
  independent cache-busting cause on the BYOK Anthropic path: <paste ticket 1 URL>.

[willsf1862]

Two follow-ups:

1. Placeholders: Copilot Chat version is 0.54.0; the companion issue (workspace tree
   injected ahead of the cached prefix) is Agent-mode workspace directory tree is injected ahead of the cached prefix and breaks BYOK Anthropic prompt caching #323641.

2. Relative size of the two causes. I classified the full-prefix cache-write turns in a
   capture window by where the first byte divergence fell. The majority diverged in the
   injected workspace tree near the head (Agent-mode workspace directory tree is injected ahead of the cached prefix and breaks BYOK Anthropic prompt caching #323641). A minority diverged deep in the message
   history at a cache_control marker — the re-anchoring described here — and those showed
   up on the longest conversations (request bodies ~1.8 MB), where the synthetic
   {"type":"text","text":" ",...} anchor count changed between turns (observed it go from
   1 to 0 across a pair, removing the block mid-history).

   So the two are independent and both reproduce; Agent-mode workspace directory tree is injected ahead of the cached prefix and breaks BYOK Anthropic prompt caching #323641 is more frequent, but this one
   isn't mitigable from the user side. Of the options above, pinning one breakpoint at the
   end of the static system/tools prefix seems like the lowest-risk first step, since that
   region never changes between turns.

[willsf1862]

One caveat for requested-fix #3 (pin a breakpoint to the static system/tools prefix), from
testing a local prefix-pin that does exactly that:

The tools array is not reliably static, so pinning at the end of tools is fragile.
VS Code contributes some relevance-gated language-model tools that enter/leave the
serialized tools[] between turns — e.g. the built-in vscode.mermaid-markdown-features
adds renderMermaidDiagram as the last tool on some turns and omits it on others. When a
breakpoint sits on the last tool, that toggle moves the cached boundary and cold-busts the
prefix: I measured cache_read collapsing to a constant ~33.5K tokens on exactly the turns
where the tool count flipped (68 ↔ 69), with otherwise byte-identical system+tools.

So if you implement #3, the safer anchor is at the end of system (before tools), or
the contributed tool set needs to be stable across turns within a session. A pin at the tools
boundary will be defeated by VS Code's own relevance-gated tool contributions.

(Pinning after system still leaves the ~88K-token tools block re-billed each turn, so the
ideal is both: stabilize the tool set and pin past it. Stabilizing tool membership for
caching is arguably the more general fix.)