.Net: Bug: 500 response when using OpenAIResponseAgent

[itayfisz]

Describe the bug
When using OpenAIResponseAgent and enumerating the responses in InvokeAsync, there are a few successful responses, but eventually there is an exception:

HTTP 500 (: InternalServerError)

Backend returned unexpected response. Please contact Microsoft for help.

Stack-trace:

   at OpenAI.ClientPipelineExtensions.ProcessMessageAsync(ClientPipeline pipeline, PipelineMessage message, RequestOptions options)
   at OpenAI.Responses.OpenAIResponseClient.CreateResponseAsync(BinaryContent content, RequestOptions options)
   at OpenAI.Responses.OpenAIResponseClient.CreateResponseAsync(IEnumerable`1 inputItems, ResponseCreationOptions options, CancellationToken cancellationToken)
   at Microsoft.SemanticKernel.Agents.OpenAI.Internal.ResponseThreadActions.InvokeAsync(OpenAIResponseAgent agent, ChatHistory history, AgentThread agentThread, AgentInvokeOptions options, CancellationToken cancellationToken)+MoveNext()
   at Microsoft.SemanticKernel.Agents.OpenAI.Internal.ResponseThreadActions.InvokeAsync(OpenAIResponseAgent agent, ChatHistory history, AgentThread agentThread, AgentInvokeOptions options, CancellationToken cancellationToken)+System.Threading.Tasks.Sources.IValueTaskSource<System.Boolean>.GetResult()
   at Microsoft.SemanticKernel.Agents.OpenAI.OpenAIResponseAgent.InvokeAsync(ICollection`1 messages, AgentThread thread, AgentInvokeOptions options, CancellationToken cancellationToken)+MoveNext()
   at Microsoft.SemanticKernel.Agents.OpenAI.OpenAIResponseAgent.InvokeAsync(ICollection`1 messages, AgentThread thread, AgentInvokeOptions options, CancellationToken cancellationToken)+MoveNext()
   at Microsoft.SemanticKernel.Agents.OpenAI.OpenAIResponseAgent.InvokeAsync(ICollection`1 messages, AgentThread thread, AgentInvokeOptions options, CancellationToken cancellationToken)+System.Threading.Tasks.Sources.IValueTaskSource<System.Boolean>.GetResult()

Reproduced with both o3 and o3-pro models.

While it seems to be an Azure OpenAI bug (and they definitely shouldn't be returning 500 in any case), it seems like SemanticKernel is constructing the call in an unexpected way. I've tested the raw OpenAI Response Client with these models, and it was working alright (though I didn't test them with function calling).

To Reproduce
Ran the following code. The references to functions from my codebase are used only for retrieving prompts.
It manages to invoke a function, and after that, the error described above it returned.

        var client = _kernel.GetRequiredService<IOpenAIProvider>().Client.GetOpenAIResponseClient("o3-pro");

        var agent = new OpenAIResponseAgent(client)
        {
            Name = "Investigator",
            Description = "Test",
            Instructions = await _agentsPromptProvider.GetInvestigatorPromptAsync(),
            Kernel = _kernel,
            StoreEnabled = false,
            Arguments = new KernelArguments(new OpenAIPromptExecutionSettings
            {
                ServiceId = ModelTypes.Reasoning,
            })
        };

        var thread = new ChatHistoryAgentThread();

        await foreach (var message in agent.InvokeAsync(
            thread: thread,
            message: await _agentsPromptProvider.GetInitPromptAsync(investigationId, extractedIcmInfo), cancellationToken: ct))
        {
            _investigationsManager.AddLog(investigationId, new InvestigationLog(LogSource.ChatMessage, LogLevel.Information, "test", message.ToString()));
        }

Expected behavior
Return all responses successfully.

Platform

• Language: C#
• Source:

<PackageReference Include="Microsoft.SemanticKernel" Version="1.59.0" />
<PackageReference Include="Microsoft.SemanticKernel.Agents.Core" Version="1.59.0" />
<PackageReference Include="Microsoft.SemanticKernel.Agents.OpenAI" Version="1.59.0-preview" />
<PackageReference Include="Microsoft.SemanticKernel.Agents.Orchestration" Version="1.59.0-preview" />
<PackageReference Include="Microsoft.SemanticKernel.Agents.Runtime.InProcess" Version="1.59.0-preview" />
<PackageReference Include="Microsoft.SemanticKernel.Connectors.AzureOpenAI" Version="1.59.0" />
<PackageReference Include="Microsoft.SemanticKernel.Connectors.OpenAI" Version="1.59.0" />
<PackageReference Include="Azure.AI.OpenAI" Version="2.2.0-beta.4" />

• AI model: Tried with o3 and o3-pro
• IDE: Visual Studio
• OS: Windows

Update

I'm adding here the payload that makes the /responses API return 500 (generated by SemanticKernel, using the code snippet shared above).

Note: I've obfuscated the literal values.

{
	"instructions": "<obfuscated>",
	"model": "o3-pro",
	"input": [
		{
			"type": "message",
			"role": "user",
			"content": [
				{
					"type": "input_text",
					"text": "<obfuscated>"
				}
			]
		},
		{
			"type": "reasoning",
			"id": "rs_686fb17e4db481a2b989875b0bb7ff4508b0738159a31990",
			"summary": []
		},
		{
			"type": "function_call",
			"id": "fc_686fb17e4ec081a28abbf8712e1ddad708b0738159a31990",
			"status": "completed",
			"call_id": "call_2Wns39eGzLV3PRN5slLCAoKC",
			"name": "my-tool-name",
			"arguments": "{\"some_arg\":\"some_val\"}"
		},
		{
			"type": "function_call_output",
			"call_id": "call_2Wns39eGzLV3PRN5slLCAoKC",
			"output": "<obfuscated>"
		}
	],
	"user": "my-agent-name",
	"parallel_tool_calls": true,
	"store": false,
	"tool_choice": "auto",
	"tools": [
		{
			"type": "function",
			"name": "my-tool-name",
			"description": "Some description",
			"parameters": {
				"type": "object",
				"required": [
					"some_arg"
				],
				"properties": {
					"some_arg": {
						"type": "string"
					}
				}
			},
			"strict": false
		}
	]
}

[ItsYaBoiNeil]

Ran into the same situation using python, gpt-4o and AzureResponsesAgent as well, it will intermittently work as intended, but more than half the time, it will error out after encountering error code 500 3 times

[itayfisz]

I just figured that the error constantly happens after there is a FunctionResultContent item.
There is probably about the function call + result that isn't serialize properly in the payload sent to the responses API.

[itayfisz]

I'm adding here the payload that makes the /responses API return 500 (generated by SemanticKernel, using the code snippet shared above).

Note: I've obfuscated the literal values.

{
	"instructions": "<obfuscated>",
	"model": "o3-pro",
	"input": [
		{
			"type": "message",
			"role": "user",
			"content": [
				{
					"type": "input_text",
					"text": "<obfuscated>"
				}
			]
		},
		{
			"type": "reasoning",
			"id": "rs_686fb17e4db481a2b989875b0bb7ff4508b0738159a31990",
			"summary": []
		},
		{
			"type": "function_call",
			"id": "fc_686fb17e4ec081a28abbf8712e1ddad708b0738159a31990",
			"status": "completed",
			"call_id": "call_2Wns39eGzLV3PRN5slLCAoKC",
			"name": "my-tool-name",
			"arguments": "{\"some_arg\":\"some_val\"}"
		},
		{
			"type": "function_call_output",
			"call_id": "call_2Wns39eGzLV3PRN5slLCAoKC",
			"output": "<obfuscated>"
		}
	],
	"user": "my-agent-name",
	"parallel_tool_calls": true,
	"store": false,
	"tool_choice": "auto",
	"tools": [
		{
			"type": "function",
			"name": "my-tool-name",
			"description": "Some description",
			"parameters": {
				"type": "object",
				"required": [
					"some_arg"
				],
				"properties": {
					"some_arg": {
						"type": "string"
					}
				}
			},
			"strict": false
		}
	]
}