Support multimodal image content from defineTool() results

[Minoo7]

Summary

Allow a defineTool() tool to return model-facing image content to the parent agent, alongside text and structured JSON output.

Flue already accepts images on user messages and on session.prompt() / session.task() / session.skill(). Pi also supports image blocks in tool results. The missing path is an image produced or loaded by a tool during an agent turn: current custom-tool output is JSON-serialized into one text block, so the parent model never receives the pixels.

Background & Motivation

I'm building an agent that works with PDFs, plans, photographs, screenshots, and other files in its sandbox. I do not want to attach every image up front. The useful pattern is on-demand inspection:

1. The agent lists or searches available files.
2. It decides which page, photograph, or screenshot matters.
3. It calls a tool to load that image.
4. The image comes back in the tool result, and the same agent reasons over it with the conversation and task context it already has.

Today step 4 is not expressible through Flue's public tool API. ToolRunEnvelope.output is JSON-serializable, and createCustomTools() turns it into a Pi text block with JSON.stringify(output). Returning an object shaped like { type: "image", data, mimeType } therefore gives the model JSON text (and a very large base64 string), not an image.

MCP tools show the same underlying gap from another direction: formatMcpResult() currently renders image items as placeholders such as [Image: image/png, … base64 chars]. This was raised for MCP in #291, but I think the useful primitive is broader than an MCP-specific fix: any Flue tool should be able to return image content to its caller.

Why the existing options are not equivalent

A tool can make a nested vision call:

const analysis = await harness.prompt(question, {
  images: [{ type: "image", data: base64Png, mimeType: "image/png" }],
});

return { output: analysis.text };

That is useful when the tool itself owns a fixed analysis task, but it is not equivalent to returning the image:

• The parent agent receives another model's interpretation, not the source image.
• The tool author has to choose the visual question before the parent agent sees the image.
• Details can be lost in the textual summary, especially for plans, forms, diagrams, and small labels.
• Follow-up visual questions require additional nested calls.
• It adds another model invocation, with extra latency and cost.
• The nested call does not have exactly the same reasoning state and context as the parent agent that selected the file.

Putting base64 in output is also not a workaround: it is serialized as text, wastes context and persistence budget, and still does not become a vision input.

Use cases this would unblock

• PDF and document inspection: let an agent select a page from a large document and inspect the rendered page only when needed.
• Browser and UI automation: return a screenshot after an interaction so the same agent can do visual QA, read canvas content, or diagnose layout problems.
• Plans, maps, and photographs: inspect evidence while retaining the parent agent's domain context and previous findings.
• Generated charts and diagrams: let the agent reason about axes, labels, colors, legends, and visual anomalies in a rendered artifact.
• MCP and connector results: preserve image content returned by an upstream tool instead of reducing it to a placeholder.

This matters because images are often discovered during a task. User-message attachments solve initial input, but not agent-driven retrieval after the conversation has started.

Goals

• Let a defineTool() result include model-facing text and image blocks that are delivered to the parent model as one tool result.
• Keep structured, schema-validated output available separately for application/UI data.
• Reuse Flue's existing durable attachment storage and canonical attachment references rather than embedding raw base64 in retained events.
• Apply the same image count/byte limits, redaction behavior, and context materialization/eviction rules discussed in Message admission has no aggregate attachment budget; tool-result images have no size check at all #529 and Model-context rebuild materializes every visible image; compaction is token-denominated and cannot prevent image-byte OOM #531. Tool results should not create a new unbounded image path.
• Preserve image content from MCP tools through the same mechanism where possible.
• Define clear behavior when the selected provider/model does not support image input.
• Keep existing JSON-only tools source-compatible.

Example

The exact API is a design choice, but a separate model-facing content field would preserve the current meaning of output:

const inspectImage = defineTool({
  name: "inspect_image",
  description: "Load an image from the sandbox for visual inspection.",
  input: v.object({ path: v.string() }),
  async run({ harness, data }) {
    const bytes = await harness.sandbox.readFileBuffer(data.path);

    return {
      content: [
        { type: "text", text: `Loaded ${data.path}` },
        {
          type: "image",
          data: Buffer.from(bytes).toString("base64"),
          mimeType: "image/png",
        },
      ],
      output: { path: data.path },
    };
  },
});

Conceptually, content would map to Pi's (TextContent | ImageContent)[] and Flue's canonical model-facing tool-result content, while output would retain its existing structured-data role.

Non-goals

• Changing the current user-message attachment API.
• Making every tool output multimodal or replacing ordinary JSON output.
• Requiring a nested harness.prompt() call merely to expose an image to the parent agent.
• Bypassing attachment admission, redaction, persistence, compaction, or provider capability checks.
• Expanding this proposal to arbitrary audio/video/binary tool results before the image path is designed.

Related work

• How do i get images from MCP? #291 — MCP image results are reduced to placeholders.
• Message admission has no aggregate attachment budget; tool-result images have no size check at all #529 — attachment byte admission and budgeting.
• Model-context rebuild materializes every visible image; compaction is token-denominated and cannot prevent image-byte OOM #531 — image materialization and model-context budgets.
• feat(sdk): images on prompt() / skill() / task() #99 — image support for prompt() / skill() / task() inputs.
• fix(runtime): redact image bytes from events #220 / Raw image bytes are retained in observable and persisted run events #221 — retaining image presence while redacting raw bytes from events.

Flue already appears to have most of the internal representation needed: Pi tool results can contain image blocks, and Flue persists tool-result images as attachment references. The request is to expose that capability through the public tool authoring path without weakening the durability and budget constraints around it.