Skip to content

feat: support images in mcp tool responses#4100

Merged
rekram1-node merged 2 commits into
anomalyco:devfrom
bcheung:support-images-in-tool-response
Nov 14, 2025
Merged

feat: support images in mcp tool responses#4100
rekram1-node merged 2 commits into
anomalyco:devfrom
bcheung:support-images-in-tool-response

Conversation

@bcheung

@bcheung bcheung commented Nov 9, 2025

Copy link
Copy Markdown
Contributor

Vision-capable models can now consume images from mcp tool responses.

https://ai-sdk.dev/docs/ai-sdk-core/tools-and-tool-calling#multi-modal-tool-results

This resolves the merge conflict in the original PR: #1589

Closes: #1563

CleanShot 2025-11-09 at 22 12 39

Comment thread packages/opencode/src/session/prompt.ts Outdated
@bcheung
bcheung force-pushed the support-images-in-tool-response branch from d8cedfb to c450a57 Compare November 10, 2025 04:12
@bcheung bcheung changed the title feat: support images in tool responses feat: support images in mcp tool responses Nov 10, 2025
@bcheung

bcheung commented Nov 10, 2025

Copy link
Copy Markdown
Contributor Author

One thing to point out: this only supports images in mcp tools, not tools from the registry.
To support tool responses from the registry, need to parse the attachments from the result here in toModelOutput: https://github.com/sst/opencode/blob/c450a5707ce8446e3ce9f2f3fa4099bf426f8b38/packages/opencode/src/session/prompt.ts#L531-L595

However, I realize that extracting base64 data from the url might get a bit messy and don't want to block this PR bc of that so leaving it out.

@bcheung
bcheung force-pushed the support-images-in-tool-response branch from c450a57 to 27c57b2 Compare November 13, 2025 03:14
@rekram1-node

Copy link
Copy Markdown
Collaborator

gonna merge today sorry for delays

@rekram1-node

Copy link
Copy Markdown
Collaborator

/oc revert the toModelOutput changes but keep the rest as is. If you don't know what im saying read the previous commits.

DO NOT commit your changes

@opencode-agent

opencode-agent Bot commented Nov 14, 2025

Copy link
Copy Markdown
Contributor

Perfect! I've reverted the toModelOutput changes back to the original simple implementation that just returns the text output, while keeping all the other changes from the commit (including the execute function modifications that support image attachments).

New%20session%20-%202025-11-14T20%3A58%3A44.356Z
opencode session  |  github run

Co-authored-by: rekram1-node <rekram1-node@users.noreply.github.com>
@rekram1-node

Copy link
Copy Markdown
Collaborator

Im gonna merge it like this, I mentioned on discord I dont think the toModelOutput change was needed? And tbh I think we can delete that block altogether but maybe im missing something

@rekram1-node
rekram1-node merged commit 37cf365 into anomalyco:dev Nov 14, 2025
3 checks passed
output,
output: textParts.join("\n\n"),
attachments,
content: result.content, // directly return content to preserve ordering when outputting to model

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

remember to remove this when removing toModelOutput

@bcheung

bcheung commented Nov 14, 2025

Copy link
Copy Markdown
Contributor Author

Im gonna merge it like this, I mentioned on discord I dont think the toModelOutput change was needed? And tbh I think we can delete that block altogether but maybe im missing something

took a quick skim, looks like toModelOutput isn't used as it was replaced with MessageV2.toModelMessage()
I'm not entirely sure though, but you know more about the internals of opencode

just left a note to not forget to remove this as it was directly used for toModelOutput when I thought this was the function that sent messages to the model
#4100 (comment)

xywsxp pushed a commit to xywsxp/opencode that referenced this pull request Apr 24, 2026
Co-authored-by: opencode-agent[bot] <opencode-agent[bot]@users.noreply.github.com>
Co-authored-by: rekram1-node <rekram1-node@users.noreply.github.com>
Rwanbt pushed a commit to Rwanbt/opencode that referenced this pull request May 5, 2026
Co-authored-by: opencode-agent[bot] <opencode-agent[bot]@users.noreply.github.com>
Co-authored-by: rekram1-node <rekram1-node@users.noreply.github.com>
AIALRA-0 pushed a commit to AIALRA-0/opencode-turn-engine that referenced this pull request Jun 10, 2026
Co-authored-by: opencode-agent[bot] <opencode-agent[bot]@users.noreply.github.com>
Co-authored-by: rekram1-node <rekram1-node@users.noreply.github.com>
AIALRA-0 pushed a commit to AIALRA-0/opencode-turn-engine that referenced this pull request Jun 10, 2026
Co-authored-by: opencode-agent[bot] <opencode-agent[bot]@users.noreply.github.com>
Co-authored-by: rekram1-node <rekram1-node@users.noreply.github.com>
avion23 pushed a commit to avion23/opencode that referenced this pull request Jun 10, 2026
Co-authored-by: opencode-agent[bot] <opencode-agent[bot]@users.noreply.github.com>
Co-authored-by: rekram1-node <rekram1-node@users.noreply.github.com>
Rwanbt pushed a commit to Rwanbt/opencode that referenced this pull request Jun 14, 2026
Co-authored-by: opencode-agent[bot] <opencode-agent[bot]@users.noreply.github.com>
Co-authored-by: rekram1-node <rekram1-node@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

MCP tool returning an image is not working properly

2 participants