Skip to content

v1.1.29 — litellm auth/* prompt caching

Choose a tag to compare

@github-actions github-actions released this 02 Jul 04:24
· 63 commits to main since this release
244b631

Wires up Anthropic prompt caching on the Claude Code OAuth (auth/*) LiteLLM path — both the cache-token metering and the actual conversation-history caching. Long, tool-heavy runs now re-read the repeated context prefix at 0.1× instead of full price.

What changed

  • Meter cache tokens. The OAuth handler (config/claude_code_handler.py) now surfaces cache_creation_input_tokens / cache_read_input_tokens from the Anthropic response in the returned usage (non-streaming and streaming), and includes them in prompt_tokens so LiteLLM's cost path prices cache reads (0.1×) / writes (1.25×) instead of silently dropping them.
  • Cache the conversation, not just the system prompt. A cache_control breakpoint is now stamped on the last message block (incremental / auto-advancing), so the whole system + tool-result / history prefix is cached and re-read across turns. System breakpoints capped to 3 to stay within Anthropic's global limit of 4.

Impact

Previously only the system prompt carried cache_control and the response's cache counters were discarded, so caching was invisible/unpriced and the dominant tool-result/history bulk was never cached. This release closes both gaps.

Verification (live, real Anthropic via OAuth)

  • System caching: call 1 cache_creation=11427 → call 2 cache_read=11427.
  • Multi-turn history caching: Turn A cache_creation=9086 → Turn B cache_read=9086, cache_creation=35 (full prefix served at 0.1×, only the new delta written).

Downstream

Consumers that build FROM ghcr.io/purpleailab/decepticon-litellm:1.1.29 must also add cache pricing (cache_read_input_token_cost / cache_creation_input_token_cost) to any auth/* model_info that overrides the built-in cost map, or cache tokens price at $0.

Commits

  • feat(litellm): meter + cache the auth/* OAuth path (prompt caching) (#748)