OpenAI Compatible Cache Matrix

OpenAI Compatible Cache Matrix

ChatHub's built-in openaicompatible provider supports both /v1/chat/completions and /v1/responses, but compatible gateways do not all use the same cache hints. Use this page as a settings checklist.

The token-badge Most recent reported cache percent is not this matrix. It is cached tokens divided by all provider-reported input on the latest assistant reply in the current topic that included cache fields. See Memory & Context Compaction. The matrix below only controls which cache hints ChatHub sends on OpenAI-compatible requests.

The OpenAI-compatible API route radio applies to Graphile background chat as well as the browser send path and Settings Check. Connectivity Check streams like chat on both routes so the gateway can record First/TTFT, and still passes on streamed text, reasoning, or an explicit completed terminal (stop / completed) even if output_text is empty. Native OpenAI Astra auto-routing is unchanged.

Quick Preset Checklist

Backticked values are dropdown/select values. Built-in presets hide the detailed verified matrix; switch to Custom to expand every cache and Responses-parameter option for provider-probe testing.

Prompt key + store

Use this built-in mode when the provider-probe confirms Chat prompt_cache_key and Responses prompt_cache_key + store:true. This is the verified shared mode for pptoken.org and apikl.ai.

UI setting	Value
OpenAI-compatible API route	Responses API
OpenAI-compatible cache preset	Prompt key + store
Hidden preset matrix	Chat prompt_cache_key, no Session_id; Responses prompt_cache_key: Auto-generate, no Session_id, store: true; omit max_tokens, omit max_output_tokens, omit truncation, omit verbosity

Custom

Use this when the provider-probe confirms a different combination.

UI setting	How to choose
OpenAI-compatible API route	Select Responses API to call /v1/responses; select Chat Completions to call /v1/chat/completions.
OpenAI-compatible cache preset	Custom
Chat prompt_cache_key	Check only if the chat probe confirms body prompt_cache_key helps.
Chat Session_id	Check only if the chat probe confirms the Session_id header helps.
Responses prompt_cache_key	Select Auto-generate only if the Responses probe confirms body prompt_cache_key helps; otherwise select Do not send.
Responses Session_id	Check only if the Responses probe confirms the Session_id header helps.
Responses store	Select the value confirmed by the probe: Default, true, or false.
Responses max_tokens	Check only if the provider accepts max_tokens on /v1/responses.
Responses max_output_tokens	Check only if the provider accepts max_output_tokens on /v1/responses.
Responses truncation	Select Do not send, auto, or disabled based on the probe.
Responses verbosity	Select Do not send, text.verbosity, top-level verbosity, or both based on the probe.

Selecting a built-in preset writes the full hidden matrix and defaults the route to Responses API. Selecting Custom expands the matrix; any manual matrix or parameter edit keeps the preset as Custom.

What Each Setting Sends

UI setting	API path	What ChatHub sends
OpenAI-compatible API route = Responses API	/v1/responses	Routes requests to Responses API.
OpenAI-compatible API route = Chat Completions	/v1/chat/completions	Routes requests to Chat Completions API.
Chat prompt_cache_key checked	/v1/chat/completions	Body prompt_cache_key: <derived stable key>.
Chat Session_id checked	/v1/chat/completions	Header Session_id: <derived stable key>.
Responses prompt_cache_key = Auto-generate	/v1/responses	Body prompt_cache_key: <derived stable key>.
Responses prompt_cache_key = Do not send	/v1/responses	No body prompt_cache_key.
Responses Session_id checked	/v1/responses	Header Session_id: <derived stable key>.
Responses store = Default	/v1/responses	No store field is sent.
Responses store = true	/v1/responses	Body store: true.
Responses store = false	/v1/responses	Body store: false.
Responses max_tokens checked	/v1/responses	Body max_tokens from the model max-output setting.
Responses max_output_tokens checked	/v1/responses	Body max_output_tokens from the model max-output setting.
Responses truncation	/v1/responses	Sends no field, truncation: "auto", or truncation: "disabled".
Responses verbosity	/v1/responses	Sends no field, text: { verbosity }, top-level verbosity, or both.

Probe Workflow

Use the repo-local chathub-provider-probe skill and dedicated probe environment variables:

export PROBE_OPENAICOMPATIBLE_API_KEY='...'
export PROBE_OPENAICOMPATIBLE_PROXY_URL='https://provider.example.com/v1'
export PROBE_OPENAICOMPATIBLE_MODEL='gpt-5.5'

Interactive cache detection must be round-bounded:

node skills/chathub-provider-probe/scripts/probe-openai-compatible.mjs --phase cache-round --runs 6

Rules:

• Use six runs for the first cache round and later interactive rounds.
• Test one strategy per unconfirmed endpoint per round.
• Pause after every round and ask the user to verify the request IDs in provider dashboard/logs.
• Keep /v1/chat/completions and /v1/responses status separate.
• If one endpoint is confirmed, lock that strategy and continue only the unconfirmed endpoint.
• Use cache-matrix only for non-interactive exhaustion after explicit approval.

Runtime Notes

• Provider cache config is stored as openAICompatCache; Responses parameter compatibility is stored as openAICompatResponsesParams.
• Legacy responseStateMode: "provider" normalizes to the Prompt key + store behavior when no matrix exists.
• Legacy saved pptoken.org and apikl.ai preset values are accepted and normalized to Prompt key + store.
• Runtime-only fields are stripped before upstream request bodies are sent.
• Built-in presets apply their verified Responses parameter behavior. Custom applies the expanded parameter matrix exactly, so unknown providers can be tested freely.
• Responses Session_id-only mode must not force a body prompt_cache_key.
• Cache keys are derived from stable prompt parts such as model, system/developer messages, first user message, reasoning effort, and tools.

Debugging Cache Misses

Start with the redacted cache diagnostic flag on the ChatHub server:

DEBUG_OPENAICOMPATIBLE_CACHE=1

This uses the shared bounded cache schema while preserving the openai-compatible-cache-debug namespace. It logs trusted provider/runtime family, API/stream mode, server-derived policy booleans, input/tool counts, deployment-keyed request/response correlations, continuation metadata, and normalized cache counters. It does not log raw prompt text, model/response IDs, cache keys, tool names/schemas/arguments/results, API keys, URLs, headers, or response content.

Use full route debug only when exact payload/stream comparison is required:

DEBUG_OPENAICOMPATIBLE_CHAT_COMPLETION=1
DEBUG_OPENAICOMPATIBLE_RESPONSES=1

These full debug modes can expose prompt, tool, and file context. Prefer DEBUG_OPENAICOMPATIBLE_CACHE=1 when comparing ChatHub sessions with provider-probe or Codex behavior.

For Prompt key + store on Responses, the cache debug should show promptCacheKey.present:true, sessionId.present:false, store:true, and no optional Responses parameter fields. If promptCacheKey is absent, the saved config is only partially applying the preset.

For Prompt key + store on Chat Completions, the cache debug should show promptCacheKey.present:true and sessionId.present:false. Chat Session_id remains Custom-only because it can break real multi-turn sessions even when a repeated single-turn probe succeeds.

Final root-cause finding: earlier cache misses were not caused by a strict provider identity difference between pptoken.org and apikl.ai. The old split was wrong because apikl.ai can cache with store:true, and pptoken.org can cache Chat Completions when chat prompt_cache_key is enabled. Intermittent single-round misses can still come from provider-side shard/warmup behavior or request-shape changes; compare DEBUG_OPENAICOMPATIBLE_CACHE=1 fingerprints, turnShape, tools fingerprint, cache hint presence, and cached-token usage before changing the preset.

Shared Cache Status Contract

OpenAI-compatible diagnostics use the same event/status contract as native OpenAI, Azure, Anthropic, Moonshot/Kimi, DeepSeek, MiniMax, and Google/Vertex:

• request: effective API mode, stream mode, cache mechanism/support, policy booleans, counts, and keyed request correlation.
• usage: keyed request/response correlation and normalized read/write/miss/total token counters.
• usage_missing: the provider omitted usage or the runtime cannot observe it.
• terminal_error: bounded error class/code without raw error messages or response content.

cacheStatus=hit means cache-read tokens were reported above zero; mixed means both read and miss tokens were reported; write means cache-write tokens were reported; and miss requires explicit zero/read-miss telemetry. not_reported means cache telemetry was omitted, while unsupported means the runtime explicitly cannot report the contract. An absent counter is never converted into zero or counted as a miss.

DeepSeek maps prompt_cache_hit_tokens and prompt_cache_miss_tokens into the shared cache counters and is classified as supported. Anthropic request events inspect the final Messages payload and report cacheControlBreakpointCount plus effective cacheTTL (5m, 1h, or mixed); an ephemeral marker without ttl uses the documented 5m default.

Every request emits one terminal cache event across streaming and JSON modes: usage or usage_missing after a successful completion, and terminal_error after provider rejection, stream failure, or downstream cancellation. Cancellation still propagates to the upstream stream.

All cache-only switches require KEY_VAULTS_SECRET or NEXT_AUTH_SECRET. Tool continuations can be joined to cache events using the synthetic continuationId. The internal continuation envelope and effective cache policy are removed at the backend boundary and never sent to compatible gateways.