Python: Azure AI Agent fails to emit intermediate messages for integrated tools (Code Interpreter/Bing)

[rutzsco]

For Azure AI Agents with integrated tools enabled (specifically Code Interpreter and Bing Grounding), the intermediate message callback is not being invoked to indicate when a tool call starts and completes. Without this capability, a Chat UI cannot show users the real-time steps taken to fulfill their requests.

To reproduce

1. Create an agent with Bing Search and Code Interpreter enabled.
2. Ask for the current weather in City, State.
3. Observe that the intermediate message callback is not invoked during the search step.

Expected behavior
As described in the documentation, the callback function should fire for each integrated tool call in the Azure AI Agent Service:
https://learn.microsoft.com/en-us/semantic-kernel/frameworks/agent/agent-streaming?pivots=programming-language-python#handling-intermediate-messages-with-a-streaming-response

Platform

• Language: Python
• Package: pip version 1.29.0
• AI model: OpenAI GPT-4o
• IDE: VS Code
• OS: Windows

Additional Information

While debugging I could see events coming back from the AI Agent Service but not handled in agent_thread_actions.py:

semantic-kernel/python/semantic_kernel/agents/azure_ai/agent_thread_actions.py

Line 425 in 97c2524

async def _process_stream_events(

[moonbox3]

Hi @rutzsco, thanks for filing. #11850 has the fixes for these issues. Please do read through the PR description and let me know if you have any questions.

One note as I called out in the PR description: the code_interpreter tool call is not handled in the traditional SK plugin tool call sense -- meaning, we're not returning FunctionCallContent with the arguments - the code interpreter inputs (Python code) is returned as StreamingTextContent for the invoke_stream path with the metadata populated with {"code": True}. This is similar to how it's done for the AzureAssistantAgent/OpenAIAssistantAgent.

For the Bing tool, this was a complete gap. I added support for it, with the caveat that the streaming case isn't currently returning the requesturl as part of the received tool call. I raised the issue with the SDK team.