cache_control for LiteLLM-routed providers

[Jash271]

Please read this first

• Have you read the docs? Yes — Agents SDK docs.
• Have you searched for related issues? Yes. Prior closed issues Support Anthropic Prompt Caching #905 ("Support Anthropic Prompt Caching", closed in July 2025 by the reporter with a custom-model workaround only — not a native SDK solution) and Anthropic caching not supported on LiteLLM #1257 ("Anthropic caching not supported on LiteLLM", closed with the wontfix label) both describe the same unmet need. PR Add prompt_cache_retention to ModelSettings #2095 added ModelSettings.prompt_cache_retention for the OpenAI Responses API path; this request is the Chat Completions / LiteLLM analog.

Describe the feature

Add a first-class ModelSettings.cache_system_prompt: bool | None flag. When set to True, the leading role=system message is emitted as a structured text content part tagged with cache_control: {"type": "ephemeral"} — the exact shape LiteLLM documents for Anthropic native, Bedrock, OpenRouter → Claude/Gemini, and Vertex AI. Those providers honor the marker as a prompt-caching breakpoint; subsequent calls with an identical prefix skip re-tokenization and bill at the cached rate.

Why this is needed

Long, stable system prompts are the common shape in Agents SDK apps, and for LiteLLM-routed providers the missing cache_control marker is the single biggest lever on input-token cost.

Today there is no SDK-side way to emit it:

1. Two sites in src/agents/models/chatcmpl_converter.py (_normalize_input_content_part_alias and extract_all_content) reconstruct text parts with only type and text, silently dropping any extra key. So even hand-written call_model_input_filter hooks that attach cache_control fail — the marker is stripped before reaching LiteLLM.
2. There is no ergonomic opt-in on ModelSettings. Per the closing comment on Support Anthropic Prompt Caching #905, the only existing workaround is to fork LitellmModel into a full custom-model implementation — heavyweight, brittle, and every consumer reinvents the same wrapper.

Empirical impact

Measured against a ~36k-char production system prompt via OpenRouter → Gemini 2.5 Pro with a patch applied: turn 2 reports 9,499 cached_tokens out of 9,544 prompt_tokens with the flag on, and 0 with it off.

Proposed API

Agent(
    instructions="<long system prompt>",
    model_settings=ModelSettings(cache_system_prompt=True),
)

The field sits alongside the existing prompt_cache_retention so the two caching paths have symmetric surface area. Default is None: when unset, all three Chat Completions-style paths (OpenAIChatCompletionsModel, LitellmModel, AnyLlmModel) produce byte-identical payloads to today — no behavior change for existing callers.

[seratch]

Hi, thanks for sharing this detailed write-up. The LiteLLM/provider prompt-caching use case is real, but I do not think ModelSettings.cache_system_prompt is the right abstraction for the SDK.

This is a Chat Completions/LiteLLM/provider-specific behavior, not a general model setting. Adding it to ModelSettings also makes the OpenAI Chat Completions and AnyLLM paths capable of emitting non-standard cache_control content parts when enabled, which can break strict OpenAI-compatible endpoints rather than being a guaranteed no-op.

For LiteLLM, the preferred direction is to keep this at the LiteLLM configuration layer. LiteLLM already supports cache_control_injection_points, and LitellmModel passes ModelSettings.extra_args through to litellm.acompletion(), so users can configure:

ModelSettings(
    extra_args={
        "cache_control_injection_points": [
            {"location": "message", "role": "system"}
        ]
    }
)

Thanks again for taking the time for sharing this feedback!