Prefix-cache prompts: add an Anthropic cache breakpoint so title and summary share the transcript cache

[straub]

Summary

PREFIX_CACHE_OPTIMIZED_PROMPTS (#316) keeps the title and summary prompts byte-identical up to the end of the transcript, so a provider can reuse the transcript between the two calls. OpenAI models do this automatically. Anthropic models, called directly or through OpenRouter, only cache up to an explicit cache_control breakpoint. Speakr never sends one, so with an Anthropic text model the full transcript is billed twice for every recording: once for the title and once for the summary.

The new TEXT_MODEL_EXTRA_BODY in v0.10.16 can't fix this. A top-level cache_control caches the whole message. The title and summary user messages differ after the transcript, so neither call reuses the other's cache, and both pay the cache-write surcharge.

Measurements

These were measured through OpenRouter with anthropic/claude-sonnet-5.5, using a synthetic prompt of about 13k tokens.

Request	cache_write_tokens	cached_tokens	Cost
No cache flag	0	0	$0.0266
Top-level cache_control, first call	13,230	0	$0.0332
Same request again	0	13,230	$0.0015
Top-level flag, same prefix, task A	14,430	0	$0.0362
Top-level flag, same prefix, task B	14,428	0	$0.0363

The last two rows show the title/summary case: the same transcript prefix with a different instruction after it gets no cache hit.

On a real two-hour, six-speaker recording (about 71k prompt tokens), each recording costs:

	Title	Summary	Total
Now (no breakpoint)	~$0.145	~$0.177	~$0.32
Breakpoint after transcript (estimated)	~$0.18 (cache write, 1.25×)	~$0.05 (cache read, 0.1×, plus output)	~$0.23

The title and summary calls run seconds apart, well inside Anthropic's 5-minute cache lifetime. For gpt-5-mini, the existing token_usage rows already show the summary reading the title's cached prefix (cached_tokens 20,992 of 22,197).

Proposed change

When PREFIX_CACHE_OPTIMIZED_PROMPTS is on, send the user message as two text parts instead of one string. The first part is the shared prefix (_shared_user_prefix(transcript_text)) with cache_control, and the second is the task-specific suffix:

{"role": "user", "content": [
    {"type": "text", "text": _shared_user_prefix(transcript_text),
     "cache_control": {"type": "ephemeral"}},
    {"type": "text", "text": "\n".join(suffix_parts)},
]}

• Where: the summary and title paths in src/tasks/processing.py, which build _shared_user_prefix(...) + suffix today.
• Gating: OpenAI-compatible servers that don't expect cache_control may reject the unknown field, and some local servers don't accept array content. So the breakpoint could be gated on the model name (anthropic/ or claude) or on a setting such as TEXT_MODEL_CACHE_BREAKPOINTS=true.
• Byte-identical prefix still matters: OpenAI's automatic caching keeps working, because the concatenated text is unchanged.
• Chat: a breakpoint at the end of the transcript/context message would let follow-up messages reuse the cache. For chat alone, CHAT_MODEL_EXTRA_BODY with a top-level cache_control may already do this, because each follow-up repeats the earlier conversation exactly.

Environment

• Speakr v0.10.15-alpha (v0.10.16 code checked for TEXT_MODEL_EXTRA_BODY)
• TEXT_MODEL_BASE_URL=https://openrouter.ai/api/v1, TEXT_MODEL_NAME=anthropic/claude-sonnet-5.5
• PREFIX_CACHE_OPTIMIZED_PROMPTS=true, transcript_length_limit=-1

---

Drafted with an AI assistant; the measurements were run on my own instance.

[murtaza-nasir]

Thank you for the detailed proposal and the measurements. A new setting, TEXT_MODEL_CACHE_BREAKPOINTS, is added for this. With it and PREFIX_CACHE_OPTIMIZED_PROMPTS both set to true, the user message of the title and summary calls is sent as two text parts: the transcript block with cache_control, then the task instructions. The prompt text is unchanged. The setting is off by default, since it is not supported by many local and OpenAI-compatible servers. In a test through OpenRouter with Claude Haiku, the summary call read 8,887 of its 8,903 prompt tokens from the cache. This will be included in the next release.

[straub]

That's great, thank you!