Feature Request: Tools returning image content digestible by the agent

[igormicadei]

Is there any plan or possibility to allow tools to return image content (i.e. MessageImageContent) as part of their output, so that the agent can directly see and reason over an image produced by a tool call?

A concrete use case: a tool that renders a specific page of a PDF, crops a region, or processes an image in any way, and hands the resulting image back to the agent as a first-class output, not just a text signal, so the LLM can visually inspect it in the very same reasoning step.

Currently ToolOutputonly serializes to text. Is a multimodal tool output type (e.g. ImageToolOutput, or a ToolOutput that can carry MessageImageContent chunks) something that could be supported?

[Tomas2D]

Thanks for the well-scoped request — yes, this is feasible. I dug into the code; here is where it stands and how it could be done.

Current state

• ToolOutput (python/beeai_framework/tools/types.py) is text-only by contract: every agent flattens results via get_text_content() when building the ToolMessage (e.g. agents/requirement/_runner.py, agents/tool_calling/agent.py).
• However, MessageToolResultContent.result is typed Any, and the LiteLLM adapter passes it through verbatim as the tool message content (adapters/litellm/chat.py, _transform_input). So the message layer can already carry structured content parts — the flattening happens in the tool/agent layer.

The real complication: provider support

• Anthropic supports image blocks inside tool_result natively — direct pass-through can work.
• OpenAI Chat Completions only accepts text in tool-role messages.
• Gemini functionResponse is JSON-only.

So a universal implementation needs the fallback other frameworks use: put the text part in the tool result as usual, and attach images to a synthetic UserMessage inserted right after the ToolMessage (e.g. "Image produced by tool call X"). The model sees the image in the same reasoning iteration, and it works with every provider. UserMessage already supports MessageImageContent, so no message-layer changes are needed for this path.

Implementation sketch

1. Add ToolOutput.to_message_content() -> list[MessageTextContent | MessageImageContent] with a default implementation of [MessageTextContent(text=self.get_text_content())] — fully backward compatible.
2. Add an ImageToolOutput (or a general multimodal output) carrying MessageImageContent parts plus a text fallback.
3. Update the agent runners to build tool results from those parts: text parts go into MessageToolResultContent; image parts go either inline (when the provider supports it) or into the trailing UserMessage fallback.
4. Capability detection per provider (LiteLLMs vision/provider metadata) to choose inline vs. fallback.
5. Mirror the same design in TypeScript afterwards.

Ill open a tracked enhancement issue for this. Would you be interested in contributing the implementation? Happy to guide you through it.