feat: support ImageContent in tool return value

[Ao-Last]

Feature Type

I cannot use LiveKit without it

Feature Description

Currently, @function_tool functions only support primitive return types (str, int, float, bool, None). Returning ImageContent fails validation silently fnc_call_out is set to None and the LLM receives no tool response.

This makes it impossible to implement tools that return visual content to the LLM (e.g. a read_whiteboard() or take_screenshot() tool).

Several providers already support multimodal content in tool results at the API level (Anthropic tool_result, Gemini FunctionResponse), so this could be wired up for supported backends. OpenAI's agents-python SDK also supported this feature openai/openai-agents-python#1898

If this is an existing feature, please let me know. If this is on the road map or the community would like to support this feaure, I could take it or help with it.

Workarounds / Alternatives

Manually inject ImageContent into chat_ctx inside the tool body and return a plain string description:

@function_tool
async def read_screenshot(self, context: RunContext) -> str:
    b64 = base64.b64encode(get_screenshot()).decode()
    context.session.chat_ctx.add_message(
        role="user",
        content=["Current screenshot:", ImageContent(image=f"data:image/png;base64,{b64}")],
    )
    return "screenshot added to context."

This workaround makes the content visible to the LLM but it should originally being passed as tool result/function response instead of a user message.

Additional Context

No response

[tinalenguyen]

Hi, thanks for the report! The workaround is the only way to return an ImageContent within a function tool at the moment. Ideally the return types are general and accepted by all LLM providers, though I see your use case. Maybe ImageContent can be silently dropped if not supported, I will look into a solution

[Ao-Last]

@tinalenguyen thanks! I would like to share my thoughts and possible solution also. We could cooperate on the possible solution :)

[Ao-Last]

Here is a summary of how I view the current state and a proposal:

1. The Current State

LiveKit's Current State: @function_tool expects primitive return types (like str, int). Later a str() will be applied. Multimodal data is unsupported. Workaround hacking it into the chat_ctx as a standard user message may break the LLM's state machine and context alignment because LLM provides use special role or type like role: "tool" to distinguish a function output from a user message.

Modern LLM APIs: The major providers (e.g. OpenAI, Anthropic, Gemini) have moved away from treating tool/function responses as simple strings. At the protocol level, they now accept a list of structured content blocks for tool results. For instance, OpenAI's tool response accepts an array of text | image | file; Anthropic's tool_result accepts an array of text and image blocks; Gemini supports multi-part functionResponse with inlineData.

2. The Scope of the Feature

If we strictly look at my initial request, simply adding ImageContent to the allowed return types is relatively straightforward. We could do type-checking at the provider adapter level: if it's a string, treat it as usual; if it's ImageContent, map it to the provider's specific image payload (and perhaps silently drop or replace it with a text warning for unsupported providers). However, it is still much behind the LLM's ability which now supports multiple output from a function call/tool use.

3. My Proposal: Refactoring the Tool Output Structure

To future-proof the framework and support the multimodal evolution of models, I suggest we consider refactoring the function_tool output structure, rather than just adding ImageContent as a standalone primitive. Introduce a List Return Type: Allow tools to return a list of heterogeneous content objects (e.g., list[str | ImageContent], or introduce a standardized ToolOutput list). The implementation from OpanAI's agents python sdk I mentioned above could be a good reference.

Adapter-Level Mapping: The provider adapters (OpenAI, Anthropic, Gemini) would iterate through this list and natively map each item to their respective multi-part payload schemas. For backward compatibility, we could provide a str() method for filtering non-str contents and concat all string contents to a single string. For LLM providers that do not support images in tool calls , the adapter can easily filter out the ImageContent objects from the list and replace them with a standardized text placeholder like [Image omitted: unsupported by provider]. This prevents crashes but still gives the LLM context that an image was generated but could not be displayed. I believe this approach not only solves the immediate issue but natively aligns LiveKit Agents with the industry standard for Agentic workflows.

---

Ref:

1. Anthropic API for tool_result
2. OpenAI API for tool output