[Microsoft.Extensions.AI.OpenAI] Add logging support to HandleToolCallsAsync

[paulbatum]

When using OpenAIRealtimeExtensions.HandleToolCallsAsync there is no way to capture the error information from failed tool calls. The errors are reported to the model but then swallowed without any additional logging:

extensions/src/Libraries/Microsoft.Extensions.AI.OpenAI/OpenAIRealtimeExtensions.cs

Lines 107 to 145 in 5161cb9

	private static async Task<ConversationItem?> GetFunctionCallOutputAsync(
	this ConversationItemStreamingFinishedUpdate update,
	IReadOnlyList<AIFunction> tools,
	bool? detailedErrors = false,
	JsonSerializerOptions? jsonSerializerOptions = null,
	CancellationToken cancellationToken = default)
	{
	if (!string.IsNullOrEmpty(update.FunctionName)
	&& tools.FirstOrDefault(t => t.Metadata.Name == update.FunctionName) is AIFunction aiFunction)
	{
	var jsonOptions = jsonSerializerOptions ?? AIJsonUtilities.DefaultOptions;
	var functionCallContent = FunctionCallContent.CreateFromParsedArguments(
	update.FunctionCallArguments, update.FunctionCallId, update.FunctionName,
	argumentParser: json => JsonSerializer.Deserialize(json,
	(JsonTypeInfo<IDictionary<string, object>>)jsonOptions.GetTypeInfo(typeof(IDictionary<string, object>)))!);
	try
	{
	var result = await aiFunction.InvokeAsync(functionCallContent.Arguments, cancellationToken).ConfigureAwait(false);
	var resultJson = JsonSerializer.Serialize(result, jsonOptions.GetTypeInfo(typeof(object)));
	return ConversationItem.CreateFunctionCallOutput(update.FunctionCallId, resultJson);
	}
	catch (JsonException)
	{
	return ConversationItem.CreateFunctionCallOutput(update.FunctionCallId, "Invalid JSON");
	}
	catch (Exception e) when (!cancellationToken.IsCancellationRequested)
	{
	var message = "Error calling tool";
	if (detailedErrors == true)
	{
	message += $": {e.Message}";
	}
	return ConversationItem.CreateFunctionCallOutput(update.FunctionCallId, message);
	}
	}

The equivalent logging code for the non realtime case is in place:
extensions/src/Libraries/Microsoft.Extensions.AI/ChatCompletion/FunctionInvokingChatClient.cs at 5161cb90e1db3c3b6192ce40a3406dabc53db35a · dotnet/extensions.

[stephentoub]

cc: @SteveSandersonMS

[SteveSandersonMS]

@paulbatum and I discussed this earlier. I agree with the proposal. The realtime function invoker should be given similar logging semantics to FunctionInvokingChatClient where possible.

[stephentoub]

The realtime function invoker should be given similar logging semantics to FunctionInvokingChatClient where possible.

Rather than doubling-down on these bespoke helpers, I wonder if at this point we should start working on the abstraction for the bidirectional streaming APIs. We know we're going to want something that encompasses OpenAI's realtime APIs, as well as the text-to-audio and audio-to-text scenarios, and wanting to layer in automatic function invocation highlights the need for a similar pipeline model.