Python: Emit usage chunks via callback as they have empty content #12447
+47
−14
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
markwallace-microsoft
added
the
python
Python Test Coverage Report •
Python Unit Test Overview
|
||||||||||||||||||||||||||||||
TaoChenOSU
reviewed
Jun 11, 2025
| @@ -450,9 +450,22 @@ async def invoke_stream( | |||
| response.name = self.name | |||
| response_builder.append(response.content) | |||
|
|
|||
| # Divert usage-based responses to the intermediate message handler, if present | |||
There was a problem hiding this comment.
Question: is this what dotnet is doing? I wonder if this is needed as we are draining the messages below.
There was a problem hiding this comment.
We drain function calling related methods below because they exist in the chat history. Usage related chunks aren't contained in the chat history, so they need specific handling here.
There was a problem hiding this comment.
Should the usage chunks be part of the chat history at the chat completion service level?
There was a problem hiding this comment.
Why not just emit via callback? Then if callers want to handle the usage, they are free too at their discretion. This will remove emitting empty content as we're now doing.
eavanvalkenburg
approved these changes
Jun 19, 2025
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Motivation and Context
We recently made an update to emit streaming chunk usage for the ChatCompletionAgent, but that can now include further complications as responses with usage will have empty content. We should be emitting this information via the optional
on_intermediate_messagecallback, so that the caller has content provided in the normal flow (anything un-related to the final message should be handled via the callback).Description
Fix the usage chunk flow and update the sample to show this behavior.
Contribution Checklist