feat(chat): cache metrics in debug traces, drop user name from prompt#9
Merged
Conversation
…ame from system prompt Forward provider cache token counts (Anthropic cache read/write, OpenAI cached_tokens) through the gateway's OpenAI-compatible usage object, parse them client-side, and display them in the debug dialog — per physical-request trace (preview + detail) and in the session usage header. The AI SDK normalizes these into usage.inputTokenDetails regardless of provider. Also remove the user's name from the chat system prompt (keeping org, department, language and the compact hint) — irrelevant to assistant behaviour, a privacy concern to send to providers, and it makes the prompt prefix more homogeneous across users for prompt caching. Billing is unchanged (full input tokens). Anthropic reads stay 0 until cache_control breakpoints are added; OpenAI-family auto-caching shows real cache reads immediately. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Surface provider prompt-cache token counts in the chat debug dialog, and stop sending the user's name to model providers.
What changed:
cached_tokens) in its OpenAI-compatibleusageasprompt_tokens_details.{cached_tokens,cache_creation_tokens}. The AI SDK normalizes these intousage.inputTokenDetails, so it's provider-agnostic.100 in (80 cached) · …and detail) and in the session usage header.Why: part of the prompt-caching work — surface cache hit/write metrics so we can measure caching before tuning it; remove the user name because it has no bearing on assistant behaviour, is a privacy concern to send to providers, and makes the prompt prefix more homogeneous across users.
Regression risks:
cache_controlbreakpoints are added; OpenAI-family auto-caching will show real cache reads immediately.Verified:
check-typesclean,lint0 errors, gateway-response + session-recorder unit specs pass (including a new cache-extraction test).