Inaccurate usage counts when using Anthropic?

[AviVahl]

Description

I've created a coding agentic flow using @anthropic-ai/sdk@0.60.0, and later converted the same flow to use the ai package and @ai-sdk/anthropic.

Using the official anthropic sdk directly, I've had to track usage myself for each step of a conversation/tool usage. I've summed up the numbers and everything seemed correct number-wise:

Input tokens: 10572
Output tokens: 499
Cache creation input tokens: 5485
Cache read input tokens: 16455

In the first step, I wrote 5485 tokens to the cache (on the system prompts). There were then 3 conversation steps where the cache was read (5485 * 3 = 16455). Everything makes sense.

I then converted my flow to the ai package.

I've used streamText, and after result.toUIMessageStream() has all been consumed, I've printed:

console.log("Usage:");
console.log(await result.usage);

which gave me:

Usage:
{
  inputTokens: 4532,
  outputTokens: 222,
  totalTokens: 4754,
  cachedInputTokens: 5439
}

totalTokens = inputTokens + outputTokens, so nothing interesting there.

cachedInputTokens... wasn't sure what this is. Based on the number it seemed like cache write, but where's the cache read? I later found out that this is not the write.

Upon further reading of the documentation, I found result.providerMetadata, so I printed:

  console.log("Usage:");
  console.log(await result.usage);
  console.log("Anthropic Usage:");
  console.log((await result.providerMetadata)?.anthropic?.usage);

Which gave me:

Usage:
{
  inputTokens: 3056,
  outputTokens: 240,
  totalTokens: 3296,
  cachedInputTokens: 5491
}
Anthropic Usage:
{
  input_tokens: 3056,
  output_tokens: 3,
  cache_creation_input_tokens: 0,
  cache_read_input_tokens: 5491,
  cache_creation: { ephemeral_5m_input_tokens: 0, ephemeral_1h_input_tokens: 0 },
  service_tier: 'standard'
}

Observations:

• output_tokens === 3 seems wrong, and does not match outputTokens. The number in outputTokens does make sense.
• cache_creation_input_tokens === 0 makes no sense. I did not cache those tokens prior to the conversation, and they were first written to the cache in that chat.
• cache_read_input_tokens === 5491 matches the cachedInputTokens, but also makes no sense. The chat had several steps with several tool calls. It should have read the cache several times. Number seems like it points only to the last step.

I believe I've ruled out a bug in my code, so I wonder whether I'm missing something obvious.

Could it be the reported usage numbers are wrong when using ai (with anthropic; multi-step conversation with tools)?

I find that very unlikely in such widely used package, but this is the only conclusion I could have deduced.

AI SDK Version

ai: 5.0.24
@ai-sdk/anthropic: 3.0.15

Code of Conduct

• I agree to follow this project's Code of Conduct

[lgrammel]

Can you provide a full code snippet that can be run to reproduce this issue?

I checked the usage information processing in https://github.com/vercel/ai/blob/main/packages/anthropic/src/anthropic-messages-language-model.ts and did not see any obvious bug

[AviVahl]

I'll try to create a contrived example and post back. Probably tomorrow.

Until then, related question: are both result.usage and result.providerMetadata.anthropic.usage the sums of all the steps in a multi-step tool usage?

[AviVahl]

Alright, this took a while, but demonstrates exactly what I've reported:
https://github.com/AviVahl/ai-usage-bug

When I run the above example:

Usage:
{
  inputTokens: 4892,
  outputTokens: 482,
  totalTokens: 5374,
  cachedInputTokens: 5163
}
Anthropic Usage:
{
  input_tokens: 4892,
  output_tokens: 1,
  cache_creation_input_tokens: 0,
  cache_read_input_tokens: 5163,
  cache_creation: { ephemeral_5m_input_tokens: 0, ephemeral_1h_input_tokens: 0 },
  service_tier: 'standard'
}

[vgire]

I see the same problem :

Usage {
  usage: {
    inputTokens: 1,
    outputTokens: 344,
    totalTokens: 345,
    cachedInputTokens: 43866
  }
}
Anthropic usage {
  usage: {
    input_tokens: 1,
    output_tokens: 2,
    cache_creation_input_tokens: 321,
    cache_read_input_tokens: 43866,
    cache_creation: { ephemeral_5m_input_tokens: 321, ephemeral_1h_input_tokens: 0 },
    service_tier: 'standard'
  }
}

Note that the await result.usage is the same than the usage field in the finish-step event when you loop around result.fullStream

Beside the reported problems on output_tokens and cache_creation fields it would be nice to have a cacheWriteInputTokens field in the await result.usage and in the finish-step event.

[AviVahl]

Alright, so using the totalUsage field gives me correct numbers for inputTokens, outputTokens, and cachedInputTokens (cache read):

Usage:
{
  inputTokens: 25050,
  outputTokens: 3916,
  totalTokens: 28966,
  reasoningTokens: undefined,
  cachedInputTokens: 36036
}

Updated the example.

The (await result.providerMetadata)?.anthropic?.usage field is still off. Seems to only be last step, with incomplete numbers:

Anthropic Usage:
{
  input_tokens: 5285,
  output_tokens: 1,
  cache_creation_input_tokens: 0,
  cache_read_input_tokens: 5148,
  cache_creation: { ephemeral_5m_input_tokens: 0, ephemeral_1h_input_tokens: 0 },
  service_tier: 'standard'
}

[AviVahl]

So what I ended up doing to get numbers that make sense:

• use result.totalUsage (instead of result.usage) to get most numbers (input, output, cache read).
• hook into onStepFinish and extractStepCacheCreationInputTokens(step) to get the cache creation tokens for each step and sum them up.

Where the function definition is:

function extractStepCacheCreationInputTokens({ providerMetadata }: ai.StepResult<any>): number {
  const anthropicUsage = providerMetadata?.anthropic?.usage as Record<string, number> | undefined;
  const anthropicCacheCreationTokens = anthropicUsage?.cache_creation_input_tokens;
  return typeof anthropicCacheCreationTokens === "number" ? anthropicCacheCreationTokens : 0;
}

The above gave me numbers that make sense.

Should be noted I did not extract 5m vs 1h cache tokens, as we're using only 5m on our end. You could extract it from the provider metadata.

Would be nice if the ai package would:

• Avoid the confusing usage vs totalUsage in StreamTextResult. it makes sense for a single step, not for the result of all steps.
• Provide me with the cache creation token counts for 5m vs 1h.
• Sum Anthropic's providerMetadata on StreamTextResult of multi-step flow. The provider should have done it. Having it equal to the result of the last step is weird.

[lgrammel]

Anthropic raw usage was incorrect, addressed via #8945

[gsabran]

@lgrammel I think there's still an issue.

Per Anthropic's doc:

• cache_creation_input_tokens: Number of tokens written to the cache when creating a new entry.
• cache_read_input_tokens: Number of tokens retrieved from the cache for this request.
• input_tokens: Number of input tokens which were not read from or used to create a cache.

The input tokens from anthropic should be cache_creation_input_tokens + input_tokens. I'm not sure how we should track the fact that the cached input tokens are more expensive that non cached one, but all those are new input tokens.

[gsabran]

I do not have permission, but I would suggest re-opening the issue.

[gsabran]

Reading the Vercel's doc, it was not clear to me whether inputTokens ("The number of input (prompt) tokens used.") should also include prompt tokens sent in the request and already cached, or only uncached new tokens.