feat: support images and PDFs in tool results

[gadenbuie]

Closes #106 (rewritten)
Closes #161

Summary

When ellmer/chatlas tools return ContentImage or ContentPDF objects, shinychat now renders them visually inside tool result cards: images as <img> elements, PDFs as a filename badge with a PDF icon.

Tools can also return mixed content lists — e.g. list(ContentText("summary"), content_image_file("plot.png"), ContentText("stats")) — and each item renders in order with appropriate formatting. Text items in mixed lists default to markdown rendering.

How it works

The server sends a new valueType = "content_extra" with a JSON array of typed items:

[
  {"type": "text", "value": "Generated 50 samples", "value_type": "markdown"},
  {"type": "image", "src": "data:image/png;base64,..."},
  {"type": "text", "value": "Mean: 0.123, SD: 0.987", "value_type": "markdown"}
]

Changes by layer

• JS: New content_extra value type in ToolResult with renderers for image, pdf, and text item types. ContentExtraText dispatches on value_type (markdown, html, code, plain text). New PdfBadge component.
• R: tool_default_display() detects Content objects in tool result values and serializes them via as_content_extra_item_or_text(). ContentText extracts @text; images/PDFs serialize to structured data.
• Python: Handles both streaming path (raw ContentToolResult with image/PDF values) and turn-replay path (standalone content items after chatlas expansion). Suppresses <tool-content> XML wrapper tags from chatlas expansion.

Design notes

• Text items in mixed content default to value_type: "markdown". There's currently no per-item API to override this — a future enhancement could add that via the display options.
• Mixed content lists expect all items to be Content subclasses (e.g. ContentText() for text, not bare strings).
• Custom display options (html/markdown/text) still take full precedence over automatic content-extra rendering.

R example app

library(shiny)
library(bslib)
library(ellmer)
library(shinychat)

tool_result_open <- function(x, ...) {
  ContentToolResult(
    value = x,
    extra = list(
      display = list(open = TRUE, show_request = FALSE, ...)
    )
  )
}

get_sample_image <- tool(
  function(title = "Random Normal Distribution") {
    path <- tempfile(fileext = ".png")
    png(path, width = 600, height = 400)
    hist(rnorm(100), main = title, col = "steelblue", border = "white")
    dev.off()
    tool_result_open(content_image_file(path))
  },
  name = "get_sample_image",
  description = "Generate a histogram plot of random normal data and return it as an image.",
  title = type_string("Title for the histogram plot.")
)

get_image_from_url <- tool(
  function(label = "placeholder") {
    w <- sample(2:6, 1) * 100
    h <- sample(2:6, 1) * 100
    tool_result_open(
      content_image_url(sprintf("https://loremflickr.com/%d/%d?lock=42", w, h))
    )
  },
  name = "get_image_from_url",
  description = "Return a remote image from a URL to demonstrate URL-based image rendering.",
  label = type_string(
    "A label for the image (not displayed, just for identification)."
  )
)

get_sample_pdf <- tool(
  function(title = "Sample PDF Report") {
    path <- tempfile(fileext = ".pdf")
    pdf(path, width = 7, height = 5)
    plot(1:10, main = title, type = "b", col = "darkred")
    dev.off()
    tool_result_open(content_pdf_file(path))
  },
  name = "get_sample_pdf",
  description = "Generate a simple PDF plot and return it to demonstrate PDF rendering in tool results.",
  title = type_string("Title for the PDF plot.")
)

get_mixed_content <- tool(
  function(n = 50L) {
    path <- tempfile(fileext = ".png")
    data <- rnorm(n)
    png(path, width = 600, height = 400)
    hist(
      data,
      main = sprintf("Distribution (n=%d)", n),
      col = "steelblue",
      border = "white"
    )
    dev.off()
    tool_result_open(
      list(
        ContentText(sprintf("Generated %d samples", n)),
        content_image_file(path),
        ContentText(
          sprintf("```\nMean: %.3f, SD: %.3f\n```", mean(data), sd(data))
        )
      )
    )
  },
  name = "get_mixed_content",
  description = "Generate random data and return a summary with an embedded histogram. Returns text, image, and text interleaved.",
  n = type_integer("Number of random samples to generate.")
)

ui <- page_fillable(
  chat_mod_ui("chat")
)

server <- function(input, output, session) {
  chat <- chat_anthropic(
    model = "claude-haiku-4-5",
    system_prompt = paste(
      "You are a helpful assistant demonstrating shinychat's tool result rendering.",
      "When asked, use the available tools to generate and return images or PDFs.",
      "You have three tools:",
      "- get_sample_image: generates a histogram plot as an inline image",
      "- get_image_from_url: returns a remote image URL as an image",
      "- get_sample_pdf: generates a PDF and returns it as a PDF badge",
      "- get_mixed_content: generates data with text summary + histogram + stats interleaved",
      "Use them when the user asks to see images or PDFs."
    )
  )
  chat$register_tool(get_sample_image)
  chat$register_tool(get_image_from_url)
  chat$register_tool(get_sample_pdf)
  chat$register_tool(get_mixed_content)
  chat_mod_server("chat", chat)
}

shinyApp(ui, server)

Python example app

import random
import tempfile

import matplotlib.pyplot as plt
import numpy as np
from chatlas import (
    ChatAnthropic,
    ContentToolResult,
    content_image_file,
    content_image_url,
    content_pdf_file,
)
from chatlas.types import ContentText
from shiny import App, Inputs, Outputs, Session, ui
from shinychat import Chat, chat_ui
from shinychat.types import ToolResultDisplay


def tool_result_open(x):
    return ContentToolResult(
        value=x,
        extra={"display": ToolResultDisplay(open=True, show_request=False)},
    )


def get_sample_image(title: str = "Random Normal Distribution"):
    """
    Generate a histogram plot of random normal data and return it as an image.

    Parameters
    ----------
    title
        Title for the histogram plot.
    """
    path = tempfile.mktemp(suffix=".png")
    fig, ax = plt.subplots(figsize=(6, 4))
    ax.hist(np.random.normal(size=100), color="steelblue", edgecolor="white")
    ax.set_title(title)
    fig.savefig(path)
    plt.close(fig)
    return tool_result_open(content_image_file(path))


def get_image_from_url(label: str = "placeholder"):
    """
    Return a remote image from a URL to demonstrate URL-based image rendering.

    Parameters
    ----------
    label
        A label for the image (not displayed, just for identification).
    """
    w = random.choice(range(2, 7)) * 100
    h = random.choice(range(2, 7)) * 100
    return tool_result_open(
        content_image_url(f"https://loremflickr.com/{w}/{h}?lock=42")
    )


def get_sample_pdf(title: str = "Sample PDF Report"):
    """
    Generate a simple PDF plot and return it to demonstrate PDF rendering in tool results.

    Parameters
    ----------
    title
        Title for the PDF plot.
    """
    path = tempfile.mktemp(suffix=".pdf")
    fig, ax = plt.subplots(figsize=(7, 5))
    ax.plot(range(1, 11), marker="o", color="darkred")
    ax.set_title(title)
    fig.savefig(path, format="pdf")
    plt.close(fig)
    return tool_result_open(content_pdf_file(path))


def get_mixed_content(n: int = 50):
    """
    Generate random data and return a summary with an embedded histogram.
    Returns text, image, and text interleaved.

    Parameters
    ----------
    n
        Number of random samples to generate.
    """
    data = np.random.normal(size=n)
    path = tempfile.mktemp(suffix=".png")
    fig, ax = plt.subplots(figsize=(6, 4))
    ax.hist(data, color="steelblue", edgecolor="white")
    ax.set_title(f"Distribution (n={n})")
    fig.savefig(path)
    plt.close(fig)
    return tool_result_open([
        ContentText(text=f"Generated {n} samples"),
        content_image_file(path),
        ContentText(text=f"Mean: {data.mean():.3f}, SD: {data.std():.3f}"),
    ])


chat_client = ChatAnthropic(
    model="claude-haiku-4-5",
    system_prompt=(
        "You are a helpful assistant demonstrating shinychat's tool result rendering. "
        "When asked, use the available tools to generate and return images or PDFs. "
        "You have three tools: "
        "- get_sample_image: generates a histogram plot as an inline image "
        "- get_image_from_url: returns a remote image URL as an image "
        "- get_sample_pdf: generates a PDF and returns it as a PDF badge "
        "- get_mixed_content: generates data with text summary + histogram + stats interleaved "
        "Use them when the user asks to see images or PDFs."
    ),
)
chat_client.register_tool(get_sample_image)
chat_client.register_tool(get_image_from_url)
chat_client.register_tool(get_sample_pdf)
chat_client.register_tool(get_mixed_content)

app_ui = ui.page_fillable(
    chat_ui("chat"),
    fillable_mobile=True,
)


def server(input: Inputs, output: Outputs, session: Session):
    chat = Chat(id="chat")

    @chat.on_user_submit
    async def _():
        user = chat.user_input()
        response = await chat_client.stream_async(user, content="all")
        await chat.append_message_stream(response)


app = App(app_ui, server)

Test plan

• R: Tool returning content_image_file() renders inline image in tool result card
• R: Tool returning content_image_url() renders remote image
• R: Tool returning content_pdf_file() renders PDF badge
• R: Mixed content list (ContentText + image + ContentText) renders interleaved
• Python: Same four tool types render correctly during streaming
• Python: Images/PDFs render correctly when replaying from history
• Python: No <tool-content> XML wrapper text visible in UI
• Error tool results still display error text in code block
• Custom display (html/markdown/text) still takes precedence over content-extra

[cpsievert]

Thanks!