Repository navigation
v0.3.1 — conversation affinity for prompt caching
Every turn of a conversation now reaches the place that already holds its prompt cache — the same pool account, and, on the fallback, the same OpenRouter provider. Non-breaking: the new setting defaults off and older configs load unchanged.
Conversation keys (#4)
Every request gets a conversation key, in order of preference:
- the
session-id(elsethread-id) header; prompt_cache_keyin the request body;- a stable FNV-1a hash of the request prefix —
instructions,toolsand the firstinputitem — which every turn of a conversation repeats verbatim. No similarity matching: prompt caching only hits on an exact prefix anyway.
- Pool: the key picks the account, so clients that send no session header (plain OpenAI-style clients on
/v1/chat/completions) now stay on one account too, not only Codex sessions. /v1/chat/completions: the client'sprompt_cache_keyis forwarded; otherwise the key is sent upstream asprompt_cache_key./v1/responsesbodies are still forwarded byte for byte.- Fallback:
sticky_session = trueon a[[fallback]]provider adds the key assession_idandprompt_cache_key(never overwriting the client's own) — the fields OpenRouter keeps provider stickiness on. OpenRouter bills cache writes at 1.25x input for GPT-5.6+, so without stickiness a turn that misses the cache is paid twice. Leave it off for Azure, which may rejectsession_id.
[[fallback]]
name = "openrouter"
sticky_session = trueCache observability
codexproxy_tokens_totalgainskind="cached"(a subset ofprompt) andkind="cache_write". A highcache_writewith a lowcachedmeans paying for a cache that is never read.- The access line gains
cached_tokens,cache_write_tokensandaffinity(header|body|derived). - Buffered and streamed chat completions now report
usage.prompt_tokens_details.cached_tokens.