Token count discrepancy Between pydantic-evals and logfire metrics

[manthan-dl]

Initial Checks

• I confirm that I'm using the latest version of Pydantic AI
• I confirm that I searched for my issue in https://github.com/pydantic/pydantic-ai/issues before opening this issue

Description

When viewing evaluation results in Logfire, there's a discrepancy between the token counts displayed in the right panel (pydantic-evals metrics) and the aggregated metrics shown in the left panel (logfire metrics).

Screenshot

Image source: https://ai.pydantic.dev/evals/how-to/logfire-integration/#what-gets-sent-to-logfire

Even in the official documentation screenshot, the numbers don't match between the two panels.

Question

Either I am not able to understand the logic behind the difference or there might be some bug here. Can someone please help me understand this?

Example Code

Python, Pydantic AI & LLM client version

python="3.10.0"
pydantic-ai="1.20.0"

[DouweM]

@manthan-dl The difference is that the data on the right shows the token usage of the evaluated function (e.g. agent), excluding token usage by evaluators (e.g. LLMJudge), whereas the left hand side shows the total token usage for the evaluate transform_recipe span, including both the evaluated function and evaluators themselves.

So in the example from the docs, the difference stems from the use of LLMJudge: https://ai.pydantic.dev/evals/evaluators/llm-judge/#recipe-generation-with-case-specific-rubrics

Could that also explain what you're seeing?