Preserve Claude prompt caching when changing effort mid-conversation

[aheritier]

Overview

Docker Agent already lets users change a model's reasoning effort at runtime with /effort and Shift+Tab. For Anthropic models, the selected value is currently applied as top-level output_config.effort on subsequent requests.

Changing that top-level value changes the rendered prompt and restarts Anthropic's prompt cache. In a long Claude session, lowering effort for a routine follow-up—or raising it for a difficult next step—can therefore cause the stable conversation prefix to be written again, increasing latency and cache-write cost.

Anthropic now provides a beta mechanism for changing effort mid-conversation while preserving the earlier cached prefix:

https://platform.claude.com/docs/en/build-with-claude/effort#change-effort-mid-conversation-beta

It uses an effort-only system message in the conversation history:

{
  "role": "system",
  "content": [],
  "output_config": {
    "effort": "low"
  }
}

The change applies from the next user turn and remains in effect until another effort-only message changes it. The beta currently requires mid-conversation-output-config-2026-07-01 and is limited to supported Claude models; unsupported models return HTTP 400.

Current behavior in Docker Agent

Runtime effort changes recreate the effective provider with a different thinking_budget (pkg/runtime/model_switcher.go). The Anthropic provider turns that into top-level output_config.effort (pkg/model/provider/anthropic/thinking.go).

The requested effort changes correctly, but Anthropic cannot reuse the prior cached prefix.

This affects both /effort and Shift+Tab. It is most visible in long, tool-heavy sessions where the stable system prompt, tool definitions, and conversation history are large enough for prompt caching to matter.

Desired outcomes

• On Claude models and API paths that support per-message effort, changing effort during a conversation preserves the reusable prompt-cache prefix instead of rewriting it from the start.
• The selected effort takes effect at the next user turn, matching Anthropic's semantics. A change made while the agent is processing tools must not be inserted between an assistant tool call and its tool results.
• Repeated effort selections before the next user turn resolve predictably, with the final selection taking effect.
• Effort state is isolated by session, agent, and active model. Concurrent background/sub-agent sessions and model or agent switches must not leak effort changes into one another.
• Conversation assembly preserves effort changes at the correct chronological position through history limits, hooks/transforms, retries, and compaction. An internal mismatch should fail visibly rather than silently applying the wrong effort.
• Effort-only control messages do not consume or displace the finite prompt-cache breakpoints used for real conversation content.
• Unsupported models, transports, fallback combinations, and thinking modes continue to behave safely. In particular, the beta must not be sent to models that reject it, and existing none or token-budget behavior must not regress.
• /effort, Shift+Tab, completion/picker state, and effort indicators remain consistent in both the full TUI and lean TUI, including clear feedback when a change is waiting for the next user turn.
• Documentation states the beta status, supported model/API combinations, cache guarantees and limitations, timing semantics, and behavior across model switches and session restoration.
• Validation demonstrates both that the requested effort changes and that later requests still report prompt-cache reuse for an unchanged prefix.

Analysis notes

A few existing behaviors make this more than a provider-field change:

• Docker Agent can make several model calls for one user turn. The effort transition must be tied to a user-message boundary, not an arbitrary loop iteration.
• Ordinary system messages are currently lifted out of Anthropic's chronological message list, while empty non-tool messages are removed during normalization. History limiting also treats system messages differently from conversation messages. An effort-only message needs to retain its special identity and position.
• Prompt-cache marking operates over a bounded tail of messages. Empty control messages must be transparent to that allocation.
• The current effort display and Shift+Tab cycle derive their state from the effective provider configuration. A cache-preserving implementation may need a session-aware source of truth even if it does not recreate the provider.
• Anthropic's standard and beta request paths are not interchangeable mid-prefix. Eligibility needs to account for the model, API path, configured fallbacks, gateways, and other transports rather than relying only on general adaptive-thinking support.
• Amazon Bedrock, Vertex AI, Docker Models Gateway, and remote Docker Agent sessions may need separate capability validation or follow-up support even though the user-facing outcome should remain coherent.

Open design questions

• Should the beta behavior be opt-in while experimental, enabled automatically when the full request path is known to support it, or both with an emergency opt-out?
• Should runtime effort changes remain process-local as they are today, or survive session close/reopen? If persisted, how should the active effort be reconstructed after compaction or model changes?
• Should unsupported Claude paths retain today's functional but cache-busting top-level change, or report that cache-preserving effort changes are unavailable?
• Which transports should be included initially: direct Anthropic only, or also Models Gateway, Vertex AI, and Bedrock after contract validation?
• Should remote/API clients gain the same session effort control as the local TUIs?

Related work

• Shift+Tab thinking cycle hides the max effort tier on Opus 4.7/4.8 (and Sonnet 4.6, Fable 5) — single-top-tier gating can't express xhigh + max #3100 — model-specific effort levels in the Shift+Tab cycle
• Clarify thinking_budget docs for Claude Opus 4.8 adaptive thinking #3093 — Claude adaptive-thinking documentation
• Adaptive thinking should only be set on supported models #3362 — adaptive thinking capability gating
• feat(tui): argument auto-completion for /effort #3731 — /effort argument completion
• script toolset renders tools in non-deterministic order, breaking prompt caching #3233 — prompt-cache stability