Feature request: agent-emitted images inline in assistant messages (assistant-side rendering) #2995
usavv1547-cyber
started this conversation in
Ideas
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
Summary
Allow the agent (assistant) to include images in its replies so they render inline in the conversation — e.g., charts/plots produced by the agent (matplotlib PNGs), diagrams, or screenshots of results. Today the protocol already reserves this, but nothing wires it end-to-end.
Why it matters
Agents routinely produce visual artifacts (backtest charts, data plots, architecture diagrams). Right now the only way to show them in the Web UI is a clickable file link, or a fragile workaround: serving the file from a local HTTP server and embedding a markdown image (
http://127.0.0.1:PORT/x.png) — which only works because the markdown image renderer happens to allow http(s) URLs.What already exists (verified in the current code)
Protocol —
ImageBlock(type: 'image',attachment: ImageAttachmentRef) is already part ofContentBlockMapinpackages/llm. The type comment explicitly says:Storage —
dsh-attachmentdefines the durable attachment vocabulary and limits (maxImageBytes,maxImagesPerMessage,maxImagePixels, media types);dsh-attachment-localimplements it.UI components —
dsh-client-ui-attachmentalready shipsMessageImage/ImageGallery/ImageLightbox, keyed byattachment.attachmentId(currently used for user uploads).What's missing
ImageBlockin assistant content (adapters currently declare text-only output).ImageBlockinside assistant message content in the conversation UI — the markdown renderer only handles![]()images, and its URL sanitizer allows onlyhttp:/https:/mailto:(sodata:URLs are rejected).Suggested minimal path
SaveImageAttachment) and receive anImageAttachmentRefto embed.ImageBlock→ the existingMessageImagecomponent, mirroring the user-side path.Happy to iterate on the design if this aligns with the roadmap. Thanks for the project!
All reactions