Tracking Tokens Across Multiple Auto-Invoked Function Calls

[jaredlhamilton]

Description

As a developer, I need to track token usage. This includes token usage across multiple auto-invoked function calls (streaming and non-streaming).

Current Behavior

When using ToolCallBehavior.AutoInvokeKernelFunctions, only the final response's token usage is accessible via InnerContent:

var result = await chatService.GetChatMessageContentAsync(chatHistory, executionSettings, kernel);
if (result.InnerContent is OpenAI.Chat.ChatCompletion innerContent && innerContent.Usage != null)
{
    // Only captures tokens from the final response
    var tokens = innerContent.Usage.InputTokenCount + innerContent.Usage.OutputTokenCount;
}

Problem

Auto-invoked functions make multiple API calls internally, each consuming tokens. Currently, there's no way to access the intermediate token usage.

Expected Behavior

Token usage should be aggregated across all auto-invoked function calls and made available in the final response. This should work for both:

• Non-streaming: Aggregate token counts in the response
• Streaming: Provide token usage updates as they occur or in a final summary

Impact

Without accurate token tracking, developers cannot:

• Monitor actual costs
• Implement usage limits
• Optimize prompts effectively

Related discussion: #10322

[moonbox3]

Adding @markwallace-microsoft for help.