Return an image from a service response as an image, like ha_get_camera_image does

[Greminn]

The case
A Home Assistant custom integration has an action that returns a photo with return_response. The response looks like this:

{"id": "8ffadd57ad19", "mime": "image/jpeg", "size": 73141, "encoding": "base64", "data": "/9j/4AAQ…"}

When an agent calls it through ha_call_service, it gets that JSON as text. A 73 KB photo is about 97,000 characters. The client refused to put it in context ("exceeds maximum allowed tokens") and saved it to a file, so the model never saw the picture. A photo near 1 MB would be over a million characters.

The idea
ha_get_camera_image already returns real image content, so ha-mcp can do this. Would you consider doing the same for a service response? For example, when a response has a mime starting with image/ and base64 data, return it as MCP image content. I don't know how ha-mcp builds its results, so a different rule might suit it better. I'd like to know whether this is something you'd want before anyone writes code.

Why it might matter
Any integration that returns a picture from an action has this same problem. Agents then have to decode the text into a file and open it with a separate tool.

Details
Home Assistant 2026.10.

[sergeykad]

Thanks for the clear write-up. Yes, this fits ha-mcp. A picture sent as text is useless to the model, and ha-mcp already returns real images from two tools, so the building blocks exist.

Some points that would shape it:

1. Check the bytes, not the key names. mime, encoding and data are your integration's own format; Home Assistant has no standard for images in action responses. A safer rule is to base64-decode the value and accept it only when the bytes really are a JPEG, PNG, GIF or WebP image. mime would then be a hint only. The rule would not depend on one integration's key names, and an ordinary string would not be mistaken for an image.
2. The base64 has to leave the JSON. If the image is added as an image block but the text stays in the response, nothing is saved. The data value would be replaced with a short note (for example <image/jpeg, 73141 bytes, see image 1>), and id, mime and size stay. The model can still refer to the image, and a client that does not show images still learns that one was there.
3. Size stays a limit. ha-mcp cannot resize images, so a 1 MB photo still travels as about 1.3 MB. This fixes the model's context, not the transfer size. Very large images may still hit client limits.

There are two ways to solve this:

1. Detect inline images in action responses, as described above. This works with your integration as it is now.
2. Add a tool that fetches an image by reference. Home Assistant core does not put image bytes in action responses. ai_task.generate_image, for example, stores the image and returns a media_source_id and a signed url. A tool that turns a media_source_id, an HA URL or an image.* entity into a picture would cover those cases. It would also cover your integration if the action returned a reference instead of the bytes.

Which one fits depends on your integration: can you change its action to return a reference instead of the base64 data?

[Greminn]

Thanks, this is helpful. The integration is Home Desk. Its get_attachment action returns exactly the shape in my first post: id, mime, size, encoding: "base64" and data. It refuses photos over 1 MB by default and never resizes them.

To answer your question: a reference is possible, but not as it stands. The photo is already served at /api/home_desk/attachment/<ticket>/<id>, and that URL needs authentication. It is not a signed URL and not a media_source_id. Returning a reference would be new work on the integration side, and the agent would still need something that can fetch it. So a fetch-by-reference tool in ha-mcp would be useful to us.

Detecting inline images (your first option) works with the action as it is today. Your three points all make sense to us: decoding the bytes to check the type, replacing data with a short note, and the size caveat.