Support Image Content in MCP tool results

[hallmark]

What would you like to be added?

Support the Image Content MCP tool result data type. When an MCP server responds with base64 encoded image data as part of its tool result, Gemini is not able to "see" the image.

Let's improve Gemini CLI to allow it to "see" images within MCP tool responses.

I can currently think of two possible approaches:

1. Generative AI SDK
   • Have image content be natively supported in the functionResponse Part type of the Gemini Generative AI SDKs. There is currently no indication (function calling example) that multimodal content that the model will understand as multimodal is supported in function responses. Only text or structured/JSON text.
   • Or introduce a new response content type specific to MCP tool use, that is distinct and able to closely match the MCP spec. Example: https://docs.anthropic.com/en/docs/agents-and-tools/mcp-connector#response-content-types
2. Gemini CLI
   • Modify the Gemini CLI, as an MCP app, to intercept image content in MCP tool responses.
   • Potentially substitute a text placeholder: "The tool responded with an image here. The image can be found after this function response."
   • Then add inlineData with the base64 image data, pushed as a new Part after the functionResponse.
   • Possible code location to understand this suggestion:

     gemini-cli/packages/core/src/tools/mcp-tool.ts

     Lines 84 to 89 in bf873a1

     	const responseParts: Part[] = await this.mcpTool.callTool(functionCalls);
     	return {
     	llmContent: responseParts,
     	returnDisplay: getStringifiedResultForDisplay(responseParts),
     	};

   • Bonus: I'd yolo support for MCP Audio Content here as well and see if it works! I don't see why it wouldn't. 🤷

Why is this needed?

This autonomous agent loop is getting popular:

Start with a UI mock -> write UI code -> take_screenshot -> edit code -> take_screenshot -> ..

MCP tools such as Puppeteer that are able to return screenshot image responses will be very handy to make this work within Gemini CLI.

Additional context

This is a similar or the same issue as with Cline: cline/cline#1865

[SEOTrafficAI]

This is definitely needed! Tried to have Gemini CLI "see" screenshots from the browsermcp screenshot tool. Gemini CLI sees the text, but cannot answer "what is the color of the button?". So any kind of iteration work on design becomes impossible here.

[sarahsturgeon]

It's not great, but if you upload the images to a webserver and then have your MCP tool instruct Gemini to webfetch the images, it seems to work.

[sarahsturgeon]

Update: JK, gemini is just really good at lying and guessing 😒

My next thought is, it seems like when you @image.png it does seem to actually process the image. I'll verify.

If true, conceivably the MCP server could save an image to some temporary dir and then somehow instruct gemini to pick it up (with read-file?)

I could probably read the tool code to figure out which of these might work but I'd way prefer to remain confused and frustrated 👍

FYI: When I tried to return "type": "image" in my MCP server's content response, I saw a zod error essentially saying it wasn't supported, but I did see that "type": "audio" was? Might be worth just trying it?

[sarahsturgeon]

Update:

• Yes, adding images to context using @image.png definitely works; gemini will parse and understand them 👍
• Asking gemini using natural language to "take a look at image.png and image2.png and tell me what you see" also works
• Asking gemini to read files from within the ./.gemini/extensions/my_extension/ dir also seems to work

So having your MCP server save the files to disk and then returning an instruction to gemini to read them seems very plausible. Will try and report back

[drstrangelooker]

When returning an image from an MCP server, an option to save it locally would be useful too.

[sarahsturgeon]

Another Update:

It definitely works. I return a link to an image, tell Gemini to download + read it - seems to work fine.

Not looking forward to rewriting this when MCP image results are supported, but, hey, I'll take it :)

[luccasparoni]

I'd love to be able to read images from MCP as well.
I'm working on a PR to fix this. Hopefully will be able to do this soon, I'm just not sure yet on implementation details.

(the description matches the image, no hallucination)

In the mean time, to download images from web you can ask the CLI to run curl -o new-filename.jpg URL_OF_THE_IMAGE and then read the downloaded image. Add this instruction to the memory and you can just send the links to it