Conversation
docs: document multi-tenant tier (v4.0.0) in README
Add ChatMessage::system_cached, a system message with the existing `cache` flag set, for marking a large STABLE prefix (a task/system prompt re-sent every agent step) for prompt caching. Additive API; the plain `system` constructor is unchanged.
…okens The OpenAI-compatible client (used for OpenRouter) dropped prompt caching on the floor: a cache:true system message serialised as a plain string with no cache_control marker, and the response usage ignored cached tokens. So every agent step re-paid full prefill on the large static prompt. - Serialise a cache:true system message as the array form with an Anthropic-style ephemeral cache_control breakpoint, which OpenRouter passes through to Anthropic models. cache:false is unchanged. - Parse cached-token counts from the response usage (OpenAI prompt_tokens_details.cached_tokens and Anthropic-style top-level cache_read/creation fields) into Usage, so metering stays correct. Same model, same outputs -- only the prefix is read from cache. Covered by tests/cache_control.rs.
build_prompt rebuilds an identical system message (task + action catalog + rules) on every step of the agent loop, and the loop runs up to max_steps LLM calls per task. Mark it with system_cached so the static prefix is served from the prompt cache on steps 2..N instead of being re-prefilled each time -- a per-step latency and cost cut on the live execution path with no change to the model's outputs. The per-step DOM and history stay uncached (they change every step).
Minor bump: the prompt-cache support (cache_control pass-through, cache token parsing, system_cached) is additive and backward-compatible. Updates [workspace.package] version and the internal version pins in [workspace.dependencies] from 4.0.0 to 4.1.0.
An #[ignore]d network smoke test that sends two identical calls with a large cached system prefix and asserts the second reports cache_read_input_tokens > 0. Skipped by the default suite (the offline cache_control tests cover serialisation and parsing); run manually where a key exists to confirm end-to-end caching against OpenRouter.
Regenerated lockfile after the workspace version bump; only the ras-* crate versions change from 4.0.0 to 4.1.0.
The cache_control breakpoint was emitted for ANY cache:true system message, keyed only on the bool. This OpenAI-compatible client is shared by 7 providers (openai, openrouter, groq, cerebras, mistral, vercel, deepseek), so a system_cached prompt sent to direct OpenAI or an OpenRouter non-Anthropic model would carry an unknown cache_control field and risk an HTTP 400. Gate the emission: ChatOpenAICompatible derives model_supports_cache_control from the model id (anthropic/..., a bare claude... id, or an ...anthropic.claude... route) and threads it into to_dto_messages. When the target does not support it, the cache:true flag is inert -- the system content is sent as a plain string, exactly like published 4.0, so a non-Anthropic call never 400s. Adds regression tests: a non-Anthropic model with cache:true emits no breakpoint, and the model gate keys on the id.
feat: gated prompt caching (cache_control) + 4.1.0 bump
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Ships prompt caching for the OpenAI-compatible (OpenRouter) path, gated to Anthropic-routed models.
system_cachedmessage constructorcache_controlbreakpoint on cache:true system messages, gated bymodel_supports_cache_control(anthropic/…, bare claude…, …anthropic.claude…); parse cache_read/creation tokens into UsageNon-Anthropic models receive plain-string system content (inert, no HTTP 400). Full workspace build+test green; cache_control suite 7/7. Minor bump → CI publishes all crates to crates.io on the v4.1.0 tag.