feat(anthropic): auto-inject prompt-cache breakpoints when enabled on a model - #810
Conversation
|
Warning Review limit reached
Next review available in: 43 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (5)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
… a model Phase 1 (DP half) of the prompt-caching work. A direct Anthropic model with auto_prompt_caching enabled makes the gateway inject cache_control breakpoints into requests carrying none of their own — one on the last system block, one on the last content block of the final message — so callers get provider-side prompt-cache discounts with no client change. Callers that set their own markers win (full stand-down, which also keeps the request within Anthropic's 4-breakpoint cap). Injection lives only in the Anthropic bridge (chat + chat_stream), not in the shared build_request, so the Bedrock-invoke and Vertex rawPredict paths do not inject — gateway-authored markers there carry provider hazards (Bedrock per-model 5m-only tiers, Vertex per-project reject mode) deferred to Phase 2. Marker preservation (#804) already covers all three. Model gains auto_prompt_caching: {enabled, ttl(5m|1h)}. DP reads it here; the paired CP knob (cp-admin.yaml + dashboard) is tracked in the umbrella issue and MUST deploy after this, since Model is deny_unknown_fields.
Cold-audit HIGH: inject_cache_breakpoints stamped cache_control onto the
final message's last block (and the promoted system block) with no
empty-text guard. When that block is the empty-text degrade shape — an
image-only or caption-less final turn flattens to {type:text,text:""},
a blank system prompt promotes to the same — Anthropic 400s with
"cache_control cannot be set for empty text blocks" (distinct from the
below-minimum-length case, which is a documented silent no-op). Guard
both injection sites with is_empty_text_block.
Also closes audit coverage gaps: bridge-level streaming injection test
(chat_stream had zero coverage — handler-families lockstep), an
enabled:false gate test (present-but-disabled was untested), and a
multi-block final message test (only-last-block-marked was asserted only
against single-block messages).
22e3b3b to
a764fb8
Compare
|
@coderabbitai review |
✅ Action performedReview finished.
|
|
Merging: full CI green (build, e2e, rust unit, coverage ≥90%, lint, schema drift, generate-openapi). The independent 6-angle cold audit (see the triage above) is the substantive review gate here and is fully satisfied — it caught and fixed a HIGH (cache_control on an empty text block → Anthropic 400). CodeRabbit could not run (plan rate-limited; its check is a false-green, not a review), so it is not standing in as the review — the cold audit is. |
1 participant
What
When a direct Anthropic model has
auto_prompt_cachingenabled, the gateway now injects prompt-cache breakpoints into requests that carry none of their own — one on the last system block, one on the last content block of the final message. Callers get provider-side prompt-cache discounts (0.1× input on cache reads) and faster TTFT with zero client changes. Callers that set their owncache_controlmarkers are forwarded unchanged (stand-down).Phase 1 (DP half) of AISIX-Cloud#1110. Builds on #804 (Phase 0 — marker preservation), which this PR is stacked on.
Why two breakpoints, why these positions
Per the Anthropic prompt-cache prefix hierarchy (tools → system → messages, docs):
This is the convergent strategy across the mainstream gateways that auto-inject (a system+trailing pair, trailing marker advancing with the dialogue). We inject at most two markers and only into a request that had zero, so Anthropic's 4-breakpoint cap can never be breached.
Scope — direct Anthropic only
Injection is applied only in the Anthropic bridge (
chat+chat_stream), not in the sharedbuild_request— so the Bedrock-invoke and Vertex:rawPredictpaths (which also callbuild_request) do not inject. That's deliberate: gateway-authored markers on Bedrock/Vertex carry provider-specific hazards (per-model 5m-only TTL tiers on Bedrock; Vertex's per-project "explicit caching disabled → reject" mode) that pass-through fidelity doesn't. Bedrock injection is AISIX-Cloud#1110 Phase 2; Vertex is deferred there. Marker preservation (Phase 0, #804) already covers all three.Changes
DP config (
aisix-core)Model.auto_prompt_caching: Option<AutoPromptCaching>— nested{ enabled: bool, ttl: Option<CacheTtl> }, following theCooldownConfighouse pattern.CacheTtl=5m | 1h(default5m). Regeneratedschemas/resources/model.schema.json.DP injection (
aisix-provider-anthropic)inject_cache_breakpoints(req, ttl)inwire.rs: stand-down scan (anycache_controlon system/messages/tools → inject nothing), else mark the last system block (promoting a string-form system to a one-block array so the marker has somewhere to live) and the last content block of the final message.5memits the bare{"type":"ephemeral"}Anthropic treats as the 5-minute default;1hcarries the explicitttl.maybe_inject_cache_breakpoints(body, ctx)inbridge.rs: readsctx.model.auto_prompt_caching, gates onenabled, shared by the streaming and non-streaming paths so they can't drift.No min-token gate: below-minimum-length prompts are a documented silent no-op upstream (no error, normal billing, both cache counters 0), so speculative injection on short prompts is harmless. An empty text block is a different case — Anthropic 400s on
cache_controlattached to one — so injection skips empty/whitespace-only blocks (reachable via the image-only-turn degrade shape and a blank system prompt); seeis_empty_text_block.Billing / observability — already wired
No usage-plumbing change needed: the response path already parses
cache_creation_input_tokens/cache_read_input_tokens(wire.rs), and the control plane already prices them at distinctcache_read/cache_writerates — so budgets, spend reports, and the dashboard reflect injected-cache economics for free.Paired CP work (required before this is user-reachable) — AISIX-Cloud#1110
Per the repo's config-knob rule, this DP PR is one half. The
auto_prompt_cachingknob is not reachable until the control plane exposes it:cp-admin.yamlschema + Go model/validation/kine projection + dashboard form + en/zh i18n. That is a paired CP PR, tracked in AISIX-Cloud#1110.Rollout ordering (blocking constraint): DP
Modelis#[serde(deny_unknown_fields)], so this DP PR — which teaches the DP to readauto_prompt_caching— must deploy before the CP PR that writes it. If the CP shipped the field first, any model carrying it would be silently dropped from the DP snapshot and 404. This PR is the safe-to-land-first half by construction.Testing
wire.rsunit tests (9): system+trailing marking, 1h ttl, no-system case, stand-down on a client message marker, stand-down on a client tool marker, in-place marking of a client block-form system, empty-text final block skipped, blank system skipped, only-last-of-multi-block marked.bridge.rstests (3): non-streaming injects when enabled, streaming injects when enabled (captures the outbound body, closing the chat_stream coverage gap), does-not-inject whenenabled:false.anthropic-upstream-e2e.test.ts, 2 new): black-box wire-shape — anauto_prompt_caching-enabled model injects markers on system (block-array form) + final message at the configured1httl with earlier turns clean and a message-count pin; and a stand-down case where a caller-marked request comes out with exactly its own single marker.cargo test+ Phase-0 e2e green (injection e2e run 3×, stable);cargo fmt+ clippy clean.Audit round
Independent 6-angle cold audit (5 confirmed / 2 rejected), all addressed in the follow-up commit:
cache_controlonto an empty text block (image-only/caption-less final turn, or blank system) → Anthropic 400. Fixed withis_empty_text_blockguard on both injection sites + 2 unit tests.chat_streaminjection had zero test coverage. Added a bridge-level streaming test capturing the outbound body.enabled:falsegate untested. Added a bridge test.Deliberately out of scope
/v1/messagespassthrough injection: those callers are cache-aware and send their own markers (→ stand-down), so injection targets the OpenAI-shape callers who don't; tracked in AISIX-Cloud#1110 if a gap emerges.usage.cache_creation.{ephemeral_5m,ephemeral_1h}_input_tokensbreakdown (verifies 5m-vs-1h write billing) — AISIX-Cloud#1110 follow-up.