Skip to content

v0.3.1 — conversation affinity for prompt caching

Choose a tag to compare

@thezillo thezillo released this 22 Sep 18:14
· 16 commits to main since this release

Every turn of a conversation now reaches the place that already holds its prompt cache — the same pool account, and, on the fallback, the same OpenRouter provider. Non-breaking: the new setting defaults off and older configs load unchanged.

Conversation keys (#4)

Every request gets a conversation key, in order of preference:

  1. the session-id (else thread-id) header;
  2. prompt_cache_key in the request body;
  3. a stable FNV-1a hash of the request prefix — instructions, tools and the first input item — which every turn of a conversation repeats verbatim. No similarity matching: prompt caching only hits on an exact prefix anyway.
  • Pool: the key picks the account, so clients that send no session header (plain OpenAI-style clients on /v1/chat/completions) now stay on one account too, not only Codex sessions.
  • /v1/chat/completions: the client's prompt_cache_key is forwarded; otherwise the key is sent upstream as prompt_cache_key. /v1/responses bodies are still forwarded byte for byte.
  • Fallback: sticky_session = true on a [[fallback]] provider adds the key as session_id and prompt_cache_key (never overwriting the client's own) — the fields OpenRouter keeps provider stickiness on. OpenRouter bills cache writes at 1.25x input for GPT-5.6+, so without stickiness a turn that misses the cache is paid twice. Leave it off for Azure, which may reject session_id.
[[fallback]]
name = "openrouter"
sticky_session = true

Cache observability

  • codexproxy_tokens_total gains kind="cached" (a subset of prompt) and kind="cache_write". A high cache_write with a low cached means paying for a cache that is never read.
  • The access line gains cached_tokens, cache_write_tokens and affinity (header | body | derived).
  • Buffered and streamed chat completions now report usage.prompt_tokens_details.cached_tokens.