Question: Can plugins access image bytes or local temp file paths for multimodal analysis?

[cyberprophet]

Question

When building a plugin that delegates image analysis to a sub-agent (e.g., a "multimodal-looker"), we need the agent to access the uploaded image data. However, it's unclear whether the plugin protocol supports passing image bytes or local temporary file paths to an agent's context.

Specifically:

1. Image bytes in tool results: If a tool returns binary image data (e.g., from a user upload stored server-side), can the agent/model actually consume those bytes for multimodal analysis? Or are tool results limited to text/JSON?

2. Local temp file paths: If uploaded images are written to a temporary file on the host machine, can the agent read from that local path? The agent runs in the same process, but it's not clear whether the security model permits arbitrary local file reads from within a plugin context.

3. Intended pattern: What is the recommended way for a plugin to give an agent access to user-uploaded images for visual analysis? Should we:

   • Embed base64-encoded image data in the tool result?
   • Pass a file URI and rely on the agent's built-in file-read capability?
   • Use a different mechanism entirely?

Context

We are building a plugin that uses multimodal-looker to analyze uploaded images. The images are written to temp files and the path is passed to the agent, but the agent does not appear to read them. We have also tried forwarding raw bytes through the plugin protocol without success.

We would like to understand whether this is a current limitation, a misconfiguration on our side, or an intentionally unsupported path.

[cyberprophet]

Root Cause Found: isSubtask strips all file parts from subagent prompts

After extensive debugging, we've confirmed the exact code path that prevents multimodal-looker (and any subagent) from receiving image data.

The Problem

File: packages/opencode/src/session/prompt.ts, lines 1928-1944

const isSubtask = (agent.mode === "subagent" && command.subtask !== false) || command.subtask === true
const parts = isSubtask
  ? [
      {
        type: "subtask" as const,
        // ...
        // TODO: how can we make task tool accept a more complex input?
        prompt: templateParts.find((y) => y.type === "text")?.text ?? "",
      },
    ]
  : [...templateParts, ...(input.parts ?? [])]

When agent.mode === "subagent" (which multimodal-looker is), isSubtask is always true. The code then extracts ONLY the text portion of the prompt, discarding all file parts — including images passed as content blocks.

What We Tried (All Failed)

Approach	Why It Fails
Pass image as file path in prompt text	Agent receives path string but no image data; read tool disabled in look_at invocation
Pass image as base64 data URI in parts[]	isSubtask strips the file part entirely at line 1941
Pass image via look_at tool (proper channel)	look_at correctly prepares filePart and passes it in parts[], but the isSubtask gate discards it before the LLM ever sees it
Use read tool from within multimodal-looker	look_at sets read: false; even if enabled, the agent has no way to know which file to read since the path context is also text-only
Register agent with mode: "primary" from plugin	Not viable — only one primary agent allowed (the controller), and mode is enforced at platform level
Set subtask: false from plugin config	Plugin config hook has no access to the command.subtask parameter

Evidence

The multimodal-looker session starts and receives the prompt, but the LLM's response is always:

{"rawDescription":"Could not access the image file at the provided path, so no visible attributes could be analyzed."}

The agent literally cannot see the image — it receives only the text instruction with no visual data attached.

Impact

multimodal-looker is effectively non-functional for any plugin that needs to pass images to it. The agent exists and is properly configured with a vision-capable model, but the platform's subtask handling prevents it from ever receiving multimodal content.

The existing TODO comment on line 1940 (// TODO: how can we make task tool accept a more complex input?) suggests this limitation is already known internally.

Suggested Fix

Preserve input.parts file entries when isSubtask === true, for example:

const parts = isSubtask
  ? [
      {
        type: "subtask" as const,
        prompt: templateParts.find((y) => y.type === "text")?.text ?? "",
        // ... existing fields
      },
      // Preserve file parts from input for multimodal subagents
      ...(input.parts?.filter((p) => p.type === "file") ?? []),
    ]
  : [...templateParts, ...(input.parts ?? [])]

Or alternatively, allow command.subtask = false to be set via the agent config so that multimodal agents can opt out of the text-only subtask path.

Environment

• OpenCode: latest dev
• oh-my-openagent: latest
• Tested with look_at tool and direct task(subagent_type="multimodal-looker")

[hmannekomimi-ctrl2]

👋 This issue remains relevant. The proposed fix in PR #20021 was auto-closed due to low reaction count, but the underlying bug (subtask prompts stripping file parts from input.parts) still affects anyone trying to use vision-capable subagents via look_at tool.

The fix is a one-liner in packages/opencode/src/session/prompt.ts:

...(input.parts?.filter((p) => p.type === "file") ?? []),

Could a maintainer please take a look and consider applying this or an equivalent fix?