Summary
Chat completions report prompt cache hits in usage.prompt_tokens_details.cached_tokens, and prompt_tokens already includes those tokens (prompt caching docs). The OTEL GenAI span set by mistralai.extra.observability.otel only carries gen_ai.usage.input_tokens and gen_ai.usage.output_tokens, so anything that prices a call from its span bills every cached token at the full input rate, ten times what the API charges for them. Nothing raises; the number just reads high.
#452 implemented this and was closed as stale in June with a note that it should still be done. #572 is the related typing gap (UsageInfo for chat does not declare prompt_tokens_details, so it lands in __pydantic_extra__).
Steps to reproduce
- Enable SDK tracing and make a chat completion whose prompt shares a 64+ token prefix with a previous one (
prompt_cache_key set).
- The API response
usage carries prompt_tokens_details.cached_tokens > 0.
- The exported span has
gen_ai.usage.input_tokens equal to the full prompt_tokens and no gen_ai.usage.cache_read.input_tokens.
Proposed fix
Emit gen_ai.usage.cache_read.input_tokens beside input_tokens whenever the usage payload carries a cache count, on both the non-streaming and streaming paths, reading the raw usage dict so it does not wait on #572. The semconv constant exists from opentelemetry-semantic-conventions 0.61b0 and the floor is 0.60b1, so the literal key is the fallback; no dependency change.
A branch with that change and the coverage folded into the two existing chat tests, per the review on #452, is here: arthi-arumugam-git/client-python@main...otel-cached-prompt-tokens (30 tests pass; with otel.py reverted the two updated tests fail on the missing attribute). Opening it as a pull request from the fork currently returns a permissions error on this repository, so I am filing it here first. Happy to open the PR if that is the preferred route.
Summary
Chat completions report prompt cache hits in
usage.prompt_tokens_details.cached_tokens, andprompt_tokensalready includes those tokens (prompt caching docs). The OTEL GenAI span set bymistralai.extra.observability.otelonly carriesgen_ai.usage.input_tokensandgen_ai.usage.output_tokens, so anything that prices a call from its span bills every cached token at the full input rate, ten times what the API charges for them. Nothing raises; the number just reads high.#452 implemented this and was closed as stale in June with a note that it should still be done. #572 is the related typing gap (
UsageInfofor chat does not declareprompt_tokens_details, so it lands in__pydantic_extra__).Steps to reproduce
prompt_cache_keyset).usagecarriesprompt_tokens_details.cached_tokens > 0.gen_ai.usage.input_tokensequal to the fullprompt_tokensand nogen_ai.usage.cache_read.input_tokens.Proposed fix
Emit
gen_ai.usage.cache_read.input_tokensbesideinput_tokenswhenever the usage payload carries a cache count, on both the non-streaming and streaming paths, reading the raw usage dict so it does not wait on #572. The semconv constant exists fromopentelemetry-semantic-conventions0.61b0 and the floor is 0.60b1, so the literal key is the fallback; no dependency change.A branch with that change and the coverage folded into the two existing chat tests, per the review on #452, is here: arthi-arumugam-git/client-python@main...otel-cached-prompt-tokens (30 tests pass; with
otel.pyreverted the two updated tests fail on the missing attribute). Opening it as a pull request from the fork currently returns a permissions error on this repository, so I am filing it here first. Happy to open the PR if that is the preferred route.