Skip to content

Commit d266ac5

Browse files
moonbox3moonbox3
authored
Python: Properly surface streaming code responses and handle Bing Grounding for AzureAIAgent (#11850)
### Motivation and Context The AzureAIAgent streaming event path for handling streaming code output as well as both streaming and non-streaming Bing Grounding tool calls were missing in the current implementation. This PR adds handling for those two scenarios. <!-- Thank you for your contribution to the semantic-kernel repo! Please help reviewers and future users, providing the following information: 1. Why is this change required? 2. What problem does it solve? 3. What scenario does it contribute to? 4. If it fixes an open issue, please link to the issue here. --> ### Description Updates to allow for proper handling of streaming code using the AzureAIAgent - code_interpreter: note that the code interpreter tool call is handled as returning `StreamingTextContent` with the `metadata` dict configured with `{"code": True}`. - bing_grounding: support for the Bing Grounding Tool is added. There is a current issue with streaming events and the Bing Grounding tool - the `requesturl` as part of the tool call is coming back empty. This issue has been raised with the appropriate team. Both streaming and non-streaming do properly return the `StreamingAnnotationContent` or `AnnotationContent` respectively. - Samples were added to show interaction with the AzureAIAgent and these tools, some also showing how to use the `on_intermediate_message` callback. - Fixes #11847 <!-- Describe your changes, the overall approach, the underlying design. These notes will help understanding how your code works. Thanks! --> ### Contribution Checklist <!-- Before submitting this PR, please make sure: --> - [X] The code builds clean without any errors or warnings - [X] The PR follows the [SK Contribution Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md) and the [pre-submission formatting script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts) raises no violations - [X] All unit tests pass, and I have added new tests where possible - [X] I didn't break anyone 😄
1 parent 966fa8c commit d266ac5

6 files changed

+442
-8
lines changed

‎python/samples/concepts/README.md

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -10,6 +10,9 @@
1010
- [Azure AI Agent with Auto Function Invocation Filter Streaming](./agents/azure_ai_agent/azure_ai_agent_auto_func_invocation_filter_streaming.py)
1111
- [Azure AI Agent with Auto Function Invocation Filter](./agents/azure_ai_agent/azure_ai_agent_auto_func_invocation_filter.py)
1212
- [Azure AI Agent with Azure AI Search](./agents/azure_ai_agent/azure_ai_agent_azure_ai_search.py)
13+
- [Azure AI Agent with Bing Grounding Streaming with Message Callback](./agents/azure_ai_agent/azure_ai_agent_bing_grounding_streaming_with_message_callback.py)
14+
- [Azure AI Agent with Bing Grounding](./agents/azure_ai_agent/azure_ai_agent_bing_grounding.py)
15+
- [Azure AI Agent with Code Interpreter Streaming with Message Callback](./agents/azure_ai_agent/azure_ai_agent_code_interpreter_streaming_with_message_callback.py)
1316
- [Azure AI Agent File Manipulation](./agents/azure_ai_agent/azure_ai_agent_file_manipulation.py)
1417
- [Azure AI Agent Prompt Templating](./agents/azure_ai_agent/azure_ai_agent_prompt_templating.py)
1518
- [Azure AI Agent Message Callback Streaming](./agents/azure_ai_agent/azure_ai_agent_message_callback_streaming.py)

‎python/samples/concepts/agents/azure_ai_agent/azure_ai_agent_bing_grounding.py

Lines changed: 86 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,86 @@
1+
# Copyright (c) Microsoft. All rights reserved.
2+
3+
import asyncio
4+
5+
from azure.ai.projects.models import BingGroundingTool
6+
from azure.identity.aio import DefaultAzureCredential
7+
8+
from semantic_kernel.agents import AzureAIAgent, AzureAIAgentSettings, AzureAIAgentThread
9+
from semantic_kernel.contents import AnnotationContent
10+
11+
"""
12+
The following sample demonstrates how to create an Azure AI agent that
13+
uses the Bing grounding tool to answer a user's question.
14+
15+
Note: Please visit the following link to learn more about the Bing grounding tool:
16+
17+
https://learn.microsoft.com/en-us/azure/ai-services/agents/how-to/tools/bing-grounding?tabs=python&pivots=overview
18+
"""
19+
20+
TASK = "Which team won the 2025 NCAA basketball championship?"
21+
22+
23+
async def main() -> None:
24+
async with (
25+
DefaultAzureCredential() as creds,
26+
AzureAIAgent.create_client(credential=creds) as client,
27+
):
28+
# 1. Enter your Bing Grounding Connection Name
29+
bing_connection = await client.connections.get(connection_name="<your-bing-grounding-connection-name>")
30+
conn_id = bing_connection.id
31+
32+
# 2. Initialize agent bing tool and add the connection id
33+
bing_grounding = BingGroundingTool(connection_id=conn_id)
34+
35+
# 3. Create an agent with Bing grounding on the Azure AI agent service
36+
agent_definition = await client.agents.create_agent(
37+
name="BingGroundingAgent",
38+
instructions="Use the Bing grounding tool to answer the user's question.",
39+
model=AzureAIAgentSettings().model_deployment_name,
40+
tools=bing_grounding.definitions,
41+
)
42+
43+
# 4. Create a Semantic Kernel agent for the Azure AI agent
44+
agent = AzureAIAgent(
45+
client=client,
46+
definition=agent_definition,
47+
)
48+
49+
# 5. Create a thread for the agent
50+
# If no thread is provided, a new thread will be
51+
# created and returned with the initial response
52+
thread: AzureAIAgentThread | None = None
53+
54+
try:
55+
print(f"# User: '{TASK}'")
56+
# 6. Invoke the agent for the specified thread for response
57+
async for response in agent.invoke(messages=TASK, thread=thread):
58+
print(f"# {response.name}: {response}")
59+
thread = response.thread
60+
61+
# 7. Show annotations
62+
if any(isinstance(item, AnnotationContent) for item in response.items):
63+
for annotation in response.items:
64+
if isinstance(annotation, AnnotationContent):
65+
print(
66+
f"Annotation :> {annotation.url}, source={annotation.quote}, with "
67+
f"start_index={annotation.start_index} and end_index={annotation.end_index}"
68+
)
69+
finally:
70+
# 8. Cleanup: Delete the thread and agent
71+
await thread.delete() if thread else None
72+
await client.agents.delete_agent(agent.id)
73+
74+
"""
75+
Sample Output:
76+
77+
# User: 'Which team won the 2025 NCAA basketball championship?'
78+
# BingGroundingAgent: The Florida Gators won the 2025 NCAA basketball championship, defeating the Houston Cougars 65-63 in the final to secure their third national title【3:5†source】【3:6†source】【3:9†source】.
79+
Annotation :> https://www.usatoday.com/story/sports/ncaab/2025/04/07/houston-florida-live-updates-national-championship-score/82982004007/, source=【3:5†source】, with start_index=147 and end_index=159
80+
Annotation :> https://bleacherreport.com/articles/25182096-winners-and-losers-2025-mens-ncaa-tournament, source=【3:6†source】, with start_index=159 and end_index=171
81+
Annotation :> https://wtop.com/ncaa-basketball/2025/04/ncaa-basketball-champions/, source=【3:9†source】, with start_index=171 and end_index=183
82+
""" # noqa: E501
83+
84+
85+
if __name__ == "__main__":
86+
asyncio.run(main())

‎python/samples/concepts/agents/azure_ai_agent/azure_ai_agent_bing_grounding_streaming_with_message_callback.py

Lines changed: 136 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,136 @@
1+
# Copyright (c) Microsoft. All rights reserved.
2+
3+
import asyncio
4+
from functools import reduce
5+
6+
from azure.ai.projects.models import BingGroundingTool
7+
from azure.identity.aio import DefaultAzureCredential
8+
9+
from semantic_kernel.agents import AzureAIAgent, AzureAIAgentSettings, AzureAIAgentThread
10+
from semantic_kernel.contents import (
11+
ChatMessageContent,
12+
FunctionCallContent,
13+
StreamingAnnotationContent,
14+
StreamingChatMessageContent,
15+
)
16+
17+
"""
18+
The following sample demonstrates how to create an Azure AI agent that
19+
uses the Bing grounding tool to answer a user's question.
20+
21+
Additionally, the `on_intermediate_message` callback is used to handle intermediate messages
22+
from the agent.
23+
24+
Note: Please visit the following link to learn more about the Bing grounding tool:
25+
26+
https://learn.microsoft.com/en-us/azure/ai-services/agents/how-to/tools/bing-grounding?tabs=python&pivots=overview
27+
"""
28+
29+
TASK = "Which team won the 2025 NCAA basketball championship?"
30+
31+
intermediate_steps: list[ChatMessageContent] = []
32+
33+
34+
async def handle_streaming_intermediate_steps(message: ChatMessageContent) -> None:
35+
intermediate_steps.append(message)
36+
37+
38+
async def main() -> None:
39+
async with (
40+
DefaultAzureCredential() as creds,
41+
AzureAIAgent.create_client(credential=creds) as client,
42+
):
43+
# 1. Enter your Bing Grounding Connection Name
44+
# <your-bing-grounding-connection-name>
45+
bing_connection = await client.connections.get(connection_name="skbinggrounding")
46+
conn_id = bing_connection.id
47+
48+
# 2. Initialize agent bing tool and add the connection id
49+
bing_grounding = BingGroundingTool(connection_id=conn_id)
50+
51+
# 3. Create an agent with Bing grounding on the Azure AI agent service
52+
agent_definition = await client.agents.create_agent(
53+
name="BingGroundingAgent",
54+
instructions="Use the Bing grounding tool to answer the user's question.",
55+
model=AzureAIAgentSettings().model_deployment_name,
56+
tools=bing_grounding.definitions,
57+
)
58+
59+
# 4. Create a Semantic Kernel agent for the Azure AI agent
60+
agent = AzureAIAgent(
61+
client=client,
62+
definition=agent_definition,
63+
)
64+
65+
# 5. Create a thread for the agent
66+
# If no thread is provided, a new thread will be
67+
# created and returned with the initial response
68+
thread: AzureAIAgentThread | None = None
69+
70+
try:
71+
print(f"# User: '{TASK}'")
72+
# 6. Invoke the agent for the specified thread for response
73+
first_chunk = True
74+
async for response in agent.invoke_stream(
75+
messages=TASK, thread=thread, on_intermediate_message=handle_streaming_intermediate_steps
76+
):
77+
if first_chunk:
78+
print(f"# {response.name}: ", end="", flush=True)
79+
first_chunk = False
80+
print(f"{response}", end="", flush=True)
81+
thread = response.thread
82+
83+
# 7. Show annotations
84+
if any(isinstance(item, StreamingAnnotationContent) for item in response.items):
85+
print()
86+
for annotation in response.items:
87+
if isinstance(annotation, StreamingAnnotationContent):
88+
print(
89+
f"Annotation :> {annotation.url}, source={annotation.quote}, with "
90+
f"start_index={annotation.start_index} and end_index={annotation.end_index}"
91+
)
92+
finally:
93+
# 8. Cleanup: Delete the thread and agent
94+
await thread.delete() if thread else None
95+
await client.agents.delete_agent(agent.id)
96+
97+
print("====================================================")
98+
print("\nResponse complete:\n")
99+
# Combine the intermediate `StreamingChatMessageContent` chunks into a single message
100+
filtered_steps = [step for step in intermediate_steps if isinstance(step, StreamingChatMessageContent)]
101+
streaming_full_completion: StreamingChatMessageContent = reduce(lambda x, y: x + y, filtered_steps)
102+
# Grab the other messages that are not `StreamingChatMessageContent`
103+
other_steps = [s for s in intermediate_steps if not isinstance(s, StreamingChatMessageContent)]
104+
final_msgs = [streaming_full_completion] + other_steps
105+
for msg in final_msgs:
106+
if any(isinstance(item, FunctionCallContent) for item in msg.items):
107+
for item in msg.items:
108+
if isinstance(item, FunctionCallContent):
109+
# Note: the AI Projects SDK is not returning a `requesturl` for streaming events
110+
# The issue was raised with the AI Projects team
111+
print(f"Function call: {item.function_name} with arguments: {item.arguments}")
112+
113+
print(f"{msg.content}")
114+
115+
"""
116+
Sample Output:
117+
118+
# User: 'Which team won the 2025 NCAA basketball championship?'
119+
# BingGroundingAgent: The Florida Gators won the 2025 NCAA men's basketball championship, defeating the Houston Cougars 65-63. It marked Florida's third national title and their first since back-to-back wins in 2006-2007【5:0†source】
120+
Annotation :> https://www.usatoday.com/story/sports/ncaab/2025/04/07/houston-florida-live-updates-national-championship-score/82982004007/, source=Florida vs Houston final score: Gators win 2025 NCAA championship, with start_index=198 and end_index=210
121+
【5:5†source】
122+
Annotation :> https://www.nbcsports.com/mens-college-basketball/live/florida-vs-houston-live-score-updates-game-news-stats-highlights-for-2025-ncaa-march-madness-mens-national-championship, source=Houston vs. Florida RECAP: Highlights, stats, box score, results as ..., with start_index=210 and end_index=222
123+
.
124+
====================================================
125+
126+
Response complete:
127+
128+
Function call: bing_grounding with arguments: None
129+
Function call: bing_grounding with arguments: None
130+
131+
The Florida Gators won the 2025 NCAA men's basketball championship, defeating the Houston Cougars 65-63. It marked Florida's third national title and their first since back-to-back wins in 2006-2007【5:0†source】【5:5†source】.
132+
""" # noqa: E501
133+
134+
135+
if __name__ == "__main__":
136+
asyncio.run(main())

‎python/samples/concepts/agents/azure_ai_agent/azure_ai_agent_code_interpreter_streaming_with_message_callback.py

Lines changed: 139 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,139 @@
1+
# Copyright (c) Microsoft. All rights reserved.
2+
3+
import asyncio
4+
from functools import reduce
5+
6+
from azure.ai.projects.models import CodeInterpreterTool
7+
from azure.identity.aio import DefaultAzureCredential
8+
9+
from semantic_kernel.agents import AzureAIAgent, AzureAIAgentSettings, AzureAIAgentThread
10+
from semantic_kernel.contents import ChatMessageContent, StreamingChatMessageContent
11+
12+
"""
13+
The following sample demonstrates how to create an Azure AI agent that
14+
uses the code interpreter tool and returns streaming responses to answer a coding question.
15+
Additionally, the `on_intermediate_message` callback is used to handle intermediate messages
16+
from the agent.
17+
"""
18+
19+
TASK = "Use code to determine the values in the Fibonacci sequence that that are less then the value of 101."
20+
21+
intermediate_steps: list[ChatMessageContent] = []
22+
23+
24+
async def handle_streaming_intermediate_steps(message: ChatMessageContent) -> None:
25+
intermediate_steps.append(message)
26+
27+
28+
async def main() -> None:
29+
async with (
30+
DefaultAzureCredential() as creds,
31+
AzureAIAgent.create_client(credential=creds) as client,
32+
):
33+
# 1. Create an agent with a code interpreter on the Azure AI agent service
34+
code_interpreter = CodeInterpreterTool()
35+
agent_definition = await client.agents.create_agent(
36+
model=AzureAIAgentSettings().model_deployment_name,
37+
tools=code_interpreter.definitions,
38+
tool_resources=code_interpreter.resources,
39+
)
40+
41+
# 2. Create a Semantic Kernel agent for the Azure AI agent
42+
agent = AzureAIAgent(
43+
client=client,
44+
definition=agent_definition,
45+
)
46+
47+
# 3. Create a thread for the agent
48+
# If no thread is provided, a new thread will be
49+
# created and returned with the initial response
50+
thread: AzureAIAgentThread | None = None
51+
52+
try:
53+
print(f"# User: '{TASK}'")
54+
# 4. Invoke the agent for the specified thread for response
55+
is_code = False
56+
last_role = None
57+
async for response in agent.invoke_stream(
58+
messages=TASK, thread=thread, on_intermediate_message=handle_streaming_intermediate_steps
59+
):
60+
current_is_code = response.metadata.get("code", False)
61+
62+
if current_is_code:
63+
if not is_code:
64+
print("\n\n```python")
65+
is_code = True
66+
print(response.content, end="", flush=True)
67+
else:
68+
if is_code:
69+
print("\n```")
70+
is_code = False
71+
last_role = None
72+
if hasattr(response, "role") and response.role is not None and last_role != response.role:
73+
print(f"\n# {response.role}: ", end="", flush=True)
74+
last_role = response.role
75+
print(response.content, end="", flush=True)
76+
thread = response.thread
77+
if is_code:
78+
print("```\n")
79+
print()
80+
finally:
81+
# 6. Cleanup: Delete the thread and agent
82+
await thread.delete() if thread else None
83+
await client.agents.delete_agent(agent.id)
84+
85+
print("====================================================")
86+
print("\nResponse complete:\n")
87+
# Combine the intermediate `StreamingChatMessageContent` chunks into a single message
88+
filtered_steps = [step for step in intermediate_steps if isinstance(step, StreamingChatMessageContent)]
89+
streaming_full_completion: StreamingChatMessageContent = reduce(lambda x, y: x + y, filtered_steps)
90+
# Grab the other messages that are not `StreamingChatMessageContent`
91+
other_steps = [s for s in intermediate_steps if not isinstance(s, StreamingChatMessageContent)]
92+
final_msgs = [streaming_full_completion] + other_steps
93+
for msg in final_msgs:
94+
print(f"{msg.content}")
95+
96+
r"""
97+
Sample Output:
98+
# User: 'Use code to determine the values in the Fibonacci sequence that that are less then the value of 101.'
99+
100+
```python
101+
def fibonacci_sequence(limit):
102+
fib_sequence = []
103+
a, b = 0, 1
104+
while a < limit:
105+
fib_sequence.append(a)
106+
a, b = b, a + b
107+
return fib_sequence
108+
109+
# Get Fibonacci sequence values less than 101
110+
fibonacci_values = fibonacci_sequence(101)
111+
fibonacci_values
112+
```
113+
114+
# AuthorRole.ASSISTANT: The values in the Fibonacci sequence that are less than 101 are:
115+
116+
\[0, 1, 1, 2, 3, 5, 8, 13, 21, 34, 55, 89\]
117+
====================================================
118+
119+
Response complete:
120+
121+
def fibonacci_sequence(limit):
122+
fib_sequence = []
123+
a, b = 0, 1
124+
while a < limit:
125+
fib_sequence.append(a)
126+
a, b = b, a + b
127+
return fib_sequence
128+
129+
# Get Fibonacci sequence values less than 101
130+
fibonacci_values = fibonacci_sequence(101)
131+
fibonacci_values
132+
The values in the Fibonacci sequence that are less than 101 are:
133+
134+
\[0, 1, 1, 2, 3, 5, 8, 13, 21, 34, 55, 89\]
135+
"""
136+
137+
138+
if __name__ == "__main__":
139+
asyncio.run(main())

0 commit comments

Comments
 (0)