.NET: How to make a tool return both text and image?

[itayfisz]

How do I make an AITool return both image and text?

static ? GetImageAndText()
{
    string text = "...";
    string base64Image = "....";


    return ? // How do I return both the text and the image?
}

In SK, I have returned a ChatMessageContent, which contains ChatMessageContentItemCollection, that can contain TextContent and ImageContent.

[stephentoub]

AIFunction.InvokeAsync is typed to return object, so you can return anything you want. By default the AIFunctions returned by AIFunctionFactory.Create JSON serialize the result object, so if you want you can return an AIContent or a collection of AIContent and that will be serialized. If you don't want it to be serialized, you can set the AIFunctionFactoryOptions.MarshalJson delegate to just echo its input, in which case no serialization will happen. We're considering possibly special-casing sick returns in AIFunctionFactory as well so that such a MarshalResult use isn't necessary. (Note that the OpenAI library doesn't yet support expressing non-text function results.)

cc: @eiriktsarpalis

[itayfisz]

@stephentoub Thanks!

In the OpenAI API, multimodal messages look like this:

{
  "role": "assistant",
  "content": [
    { "type": "text", "text": "Here’s your image" },
    { "type": "image_url", "image_url": { "url": "data:image/png;base64,..." } }
  ]
}

From what you described, the AIFunction.InvokeAsync just serializes whatever the tool returns into JSON text (unless MarshalJson is customized), so the output still ends up as plain text — not as structured content items like OpenAI expects for multimodal messages.

If I got this right, that means there’s currently no way for an AITool to return something that would translate into a real image_url entry in the OpenAI request/response format.
So basically any image data coming from a tool would still be treated as text, not an actual image the model can render.

If that’s the case, it’s kind of a blocker for migrating some of our multimodal flows from SK, since those rely on returning both TextContent and ImageContent in the same message.

[stephentoub]

I don't understand why you're blocked. Why can't you just set MarshalResult? Or worst case, deserialize the result before using it however you want to use it?

[itayfisz]

With or without the MarshalResult, eventually MAF sends the content property to the response API as a string.

{
  "role": "tool",
  "tool_call_id": "call_xyz",
  "content": " < this is a string > "
}

In order to be able to represent images, let alone images and text, we need the content to be an array like this:

{
  "role": "tool",
  "tool_call_id": "call_abc123",
  "content": [
    { "type": "output_text", "text": "Done!" },
    { "type": "image_url", "image_url": { "url": "data:image/png;base64,..." } }
  ]
}

And MarshalResult only let's me control how this string is serialized.

If I'm wrong about this, I'd appreciate to see an example where it works.

[stephentoub]

With or without the MarshalResult, eventually MAF sends the content property to the response API as a string.

That's because of the other thing I said:
"Note that the OpenAI library doesn't yet support expressing non-text function results."
openai/openai-dotnet#759

And MarshalResult only let's me control how this string is serialized.

No, MarshalResult is object-in / object-out. If you just return the input object, the AIFunction.InvokeAsync method will return the exact object reference the delegate returned, no serialization involved. Any serialization you see would be later in the pipeline, in this case most likely in the OpenAI chat client when taking the FunctionResultContent and serializing its object Result to go onto the wire. Once the OpenAI library supports non-text function results, the IChatClient implementation can be updated to use it.

[markwallace-microsoft]

@itayfisz Does the feedback provided resolve your issue?

[stephentoub]

The latest release of Microsoft.Extensions.AI.OpenAI on nuget addresses this. An AIFunction can return AIContent (TextContent, DataContent, UriContent, HostedFileContent) or enumerables of such content, and it'll use new APIs in OpenAI 2.6.0 to send it appropriately.

[zh6335901]

The latest release of Microsoft.Extensions.AI.OpenAI on nuget addresses this. An AIFunction can return AIContent (TextContent, DataContent, UriContent, HostedFileContent) or enumerables of such content, and it'll use new APIs in OpenAI 2.6.0 to send it appropriately.

Hi, @stephentoub , I use latest release of Microsoft.Extensions.AI.OpenAI on nuget, It still serailized AIContent as string, look like this:

{
      "role": "tool",
      "content": "[\r\n  {\r\n    \"$type\": \"text\",\r\n    \"text\": \"{\\\"browserState\\\":{\\\"pages\\\":[{\\\"id\\\":1,\\\"url\\\":\\\"https://www.baidu.com/\\\",\\\"title\\\":\\\"\\\\u767E\\\\u5EA6\\\\u4E00\\\\u4E0B\\\\uFF0C\\\\u4F60\\\\u5C31\\\\u77E5\\\\u9053\\\"}],\\\"currentPageId\\\":1},\\\"result\\\":\\\"Page opened successfully with url: https://www.baidu.com/, Actual url: https://www.baidu.com/\\\"}\"\r\n  },\r\n  {\r\n    \"$type\": \"uri\",\r\n    \"uri\": \"https://files.octrpa.com\",\r\n    \"mediaType\": \"image/png\"\r\n  }\r\n]",
      "tool_call_id": "call_5d4b46b988874e488340a56222af"
}

Finally, i have to use ChatClient middleware to solve it (Set message's RawRepresentation):

private async Task<ChatResponse> OnChatting(
    IEnumerable<ChatMessage> messages,
    ChatOptions? options,
    IChatClient innerChatClient,
    CancellationToken cancellationToken)
{
    foreach (var message in messages)
    {
        if (message.Role == ChatRole.Tool)
        {
            var functionResultContent = message.Contents.OfType<FunctionResultContent>().FirstOrDefault();
            if (functionResultContent is null)
                continue;

            var result = functionResultContent.Result;
            if (result is IEnumerable<AIContent> contents && message.RawRepresentation is null)
            {
                var contentParts = contents.Select(c => c switch
                {
                    TextContent textContent => OpenAI.Chat.ChatMessageContentPart.CreateTextPart(textContent.Text),
                    DataContent dataContent => OpenAI.Chat.ChatMessageContentPart.CreateImagePart(new Uri(dataContent.Uri)),
                    _ => throw new NotSupportedException($"Unsupported content type: {c.GetType().FullName}")
                });

                message.RawRepresentation = OpenAI.Chat.ChatMessage.CreateToolMessage(
                    functionResultContent.CallId,
                    contentParts
                );
            }
        }
    }

    var response = await innerChatClient.GetResponseAsync(messages, options, cancellationToken);

    Output?.Invoke(response.Text);

    return response;
}

[stephentoub]

, I use latest release of Microsoft.Extensions.AI.OpenAI on nuget, It still serailized AIContent as string, look like this

Are you using chat completion or responses?

It looks like chat completion. Try responses instead.

[zh6335901]

, I use latest release of Microsoft.Extensions.AI.OpenAI on nuget, It still serailized AIContent as string, look like this

Are you using chat completion or responses?

It looks like chat completion. Try responses instead.

Could you clarify what you mean by ‘chat completion’ versus ‘responses’? My current agent-building code looks like this:

  var openAIClient = new OpenAIClient(
    new ApiKeyCredential("xxxx"),
    new OpenAIClientOptions { Endpoint = new Uri("xxxx") }
  );

  var chatClient = openAIClient
      .GetChatClient("gpt-5-mini")
      .AsIChatClient()
      .AsBuilder()
      .Use(getResponseFunc: OnChatting, getStreamingResponseFunc: null)
      .Build();

  _agent = chatClient
      .CreateAIAgent(new ChatClientAgentOptions
      {
          Instructions = SystemPrompt_EN,
          ChatOptions = new ChatOptions { Tools = ListTools() }
      })
      .AsBuilder()
      .Use(OnFunctionCalling)
      .Build();

I have tried openAIClient.GetOpenAIResponseClient("gpt-5-mini"), But it throw MissingMethodException error when invoke GetResponseAsync method:

Method not found: 'OpenAI.Responses.WebSearchTool OpenAI.Responses.ResponseTool.CreateWebSearchTool(OpenAI.Responses.WebSearchToolLocation, System.Nullable`1<OpenAI.Responses.WebSearchToolContextSize>)'.

[stephentoub]

I have tried openAIClient.GetOpenAIResponseClient("gpt-5-mini")

That is what I meant, using GetOpenAIResponseClient instead of GetChatClient. That controls which protocol you're using to speak to OpenAI. To my knowledge, OpenAI doesn't support non-text tool results with chat completion, only with responses.

But it throw MissingMethodException error when invoke GetResponseAsync method

That's unexpected. What does your csproj look like? What version of the OpenAI package are you referencing?

[zh6335901]

I have tried openAIClient.GetOpenAIResponseClient("gpt-5-mini")

That is what I meant, using GetOpenAIResponseClient instead of GetChatClient. That controls which protocol you're using to speak to OpenAI. To my knowledge, OpenAI doesn't support non-text tool results with chat completion, only with responses.

But it throw MissingMethodException error when invoke GetResponseAsync method

That's unexpected. What does your csproj look like? What version of the OpenAI package are you referencing?

OpenAI package version is 2.7.0. And i rollback to 2.6.0, it throw another error:
Unknown ReasoningStatus value. (Parameter 'value').

My csproj package reference is look like now:

  <ItemGroup>
    <PackageReference Include="Microsoft.Agents.AI.OpenAI" Version="1.0.0-preview.251114.1" />
    <PackageReference Include="Microsoft.Extensions.AI.OpenAI" Version="10.0.0-preview.1.25560.10" />
  </ItemGroup>