Skip to content

Add chat-with-multimodal example#116

Merged
chris-colinsky merged 2 commits into
mainfrom
feature/example-chat-with-multimodal
Jun 1, 2026
Merged

Add chat-with-multimodal example#116
chris-colinsky merged 2 commits into
mainfrom
feature/example-chat-with-multimodal

Conversation

@chris-colinsky
Copy link
Copy Markdown
Member

@chris-colinsky chris-colinsky commented Jun 1, 2026

Summary

First of three new examples picked from the audit (this one, plus production-observability-with-timing-middleware and a crash-and-resume drama on example 08 to follow).

New examples/11-chat-with-multimodal/ demonstrates the headline of proposal 0046 (ChatPrompt + PlaceholderSegment, shipped in v0.11.0) end-to-end:

  • ChatPrompt with ContentSegment (system + user) and PlaceholderSegment for chat-history injection. Multi-turn conversation memory threads through state via the append reducer; each turn's render() injects the full prior history at the placeholder slot.
  • Multimodal turn. One of four scripted turns attaches a NASA public-domain Apollo 16 LM "Orion" photograph via ImageURLBlockTemplate. Same chat template; only the trailing user ContentSegment's content shape changes (string vs content-blocks list). System + placeholder segments are identical across both shapes.
  • Error handling at the invoke() boundary. main() catches NodeException, inspects exc.__cause__ for LlmProviderError to surface the canonical category. Comments name the other two legitimate handler locations (RetryMiddleware, node-internal try/except).
  • Streaming transcript. Each turn prints from inside the respond node body so the conversation arrives as the graph executes. A _TURN_DELAY_S = 0.5s pacing constant lets the reader follow along.
  • --traces argparse flag (default off) opts in to the OTel observer with a console exporter. Without it the chat runs without any observer attached; with it, JSON spans stream to stderr alongside the conversation on stdout.

Complementary to example 09 (tool calling); chat history threading and tool calling are separate primitives. Example 03 owns the observer-hooks story end-to-end, so this example points readers there for the observability details rather than re-teaching them.

Bonus pickup: docs/examples/index.md was missing example 10 from its catalog before this PR. Caught alongside the example 11 entry and added.

Things worth noting

  • Image URL hosting matters. OpenAI's vision pipeline downloads the image from the URL during the chat completion call. Some CDNs (notably upload.wikimedia.org) block OpenAI's image fetcher and return a ProviderInvalidRequest. The default URL points at images-assets.nasa.gov which is known to work. Comment in the code calls this out so users overriding IMAGE_URL know what's safe.
  • Inline _NoFetchBackend stub. The example constructs ChatPrompt objects inline and calls manager.render() directly without going through fetch(). The backend stub exists purely to satisfy PromptManager.__init__'s "at least one backend" check. Production deployments supply a real backend and call manager.fetch(name, label) before render.

Test plan

  • Verified end-to-end against OpenAI gpt-4o-mini: 4 turns, multimodal turn 2 returns a real LM description, conversation memory carries forward (turn 3 references "the LM you described"), final state has 8 messages.
  • Pyright clean (0 errors on the example file).
  • ruff + format clean.
  • No em dashes.
  • mkdocs strict build clean.
  • AGENTS.md drift test passes after regen (the example's docstring first-paragraph extraction lands in the bundled examples index).
  • Full suite (1080 pass, no regressions).
  • Manual: confirm the new entry appears under Examples in the docs nav with the catalog entry on the index page.

Out of scope

New examples/11-chat-with-multimodal/ demonstrates the headline of
proposal 0046 (ChatPrompt + PlaceholderSegment, shipped in v0.11.0)
end-to-end:

- ChatPrompt with ContentSegment (system + user) and
  PlaceholderSegment for chat-history injection.
- Multi-turn conversation memory threaded through state via the
  ``append`` reducer; each turn's render() sees the full prior
  history through the placeholder slot.
- Multimodal turn: one of four scripted turns attaches a NASA
  public-domain Apollo 16 LM "Orion" photograph via
  ImageURLBlockTemplate. Same chat template, only the trailing
  user ContentSegment's content shape changes (string vs
  content-blocks list); system + placeholder segments are
  identical across both shapes.
- Error handling at the invoke() boundary: try/except NodeException
  in main(), inspect exc.__cause__ for LlmProviderError to surface
  the canonical category string. Comments name the other two
  legitimate handler locations (RetryMiddleware, node-internal
  try/except).
- Streaming transcript: each turn prints from inside the respond
  node body so the conversation arrives as the graph executes
  rather than waiting for invoke() to return. A 0.5s
  ``_TURN_DELAY_S`` pacing constant lets the reader follow along.
- ``--traces`` argparse flag (default off) opts in to the OTel
  observer with a console exporter. Without it the chat runs
  without any observer attached; with it, JSON spans stream to
  stderr alongside the conversation on stdout.

Complementary to example 09 (tool calling); chat history threading
and tool calling are separate primitives. Example 03 owns the
observer-hooks story, so this example points readers there for the
observability details rather than re-teaching them.

Bonus pickup: docs/examples/index.md was missing example 10 from
its catalog before this PR. Caught alongside the example 11 entry
and added.
Copilot AI review requested due to automatic review settings June 1, 2026 23:13
Copy link
Copy Markdown

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds a new runnable + documented example demonstrating multi-turn chat history injection with ChatPrompt/PlaceholderSegment, including one multimodal (image URL) turn, and wires it into the docs navigation and example catalogs.

Changes:

  • Add examples/11-chat-with-multimodal/ demo script showing conversation memory via append + placeholder injection and optional OTel tracing.
  • Add docs page + navigation/catalog entries for example 11 (and fill the missing catalog entry for example 10).
  • Update generated examples inventory (AGENTS.md) and changelog entry.

Reviewed changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated 3 comments.

Show a summary per file
File Description
src/openarmature/AGENTS.md Registers example 11 in the generated examples inventory.
mkdocs.yml Adds example 11 to the docs nav under Examples.
examples/11-chat-with-multimodal/main.py New end-to-end demo script for chat history + multimodal turn + optional tracing.
docs/examples/index.md Adds catalog entries for examples 10 and 11.
docs/examples/11-chat-with-multimodal.md New documentation page for example 11.
CHANGELOG.md Adds Unreleased “Added” entries for example 11 and the catalog fix.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread examples/11-chat-with-multimodal/main.py Outdated
Comment thread docs/examples/11-chat-with-multimodal.md Outdated
Comment thread docs/examples/11-chat-with-multimodal.md Outdated
Three CoPilot findings on PR #116:

1. main.py docstring referenced PlaceholderSegment(name="history")
   but the actual field is named ``placeholder``. The code's
   chat-template construction uses ``placeholder=`` correctly; only
   the docstring narrative was wrong. A reader copy-pasting from the
   docstring would have hit a Pydantic ValidationError.
2. Walk-through doc's intro paragraph still said "Apollo 11 Lunar
   Module" after the image URL swap to the Apollo 16 "Orion" shot.
   The code's docstring + inline comment got updated when the URL
   changed but the walk-through intro was missed.
3. Sample "Reading the output" block showed an ``upload.wikimedia.org``
   image URL the example explicitly warns against. Updated to the
   actual default ``images-assets.nasa.gov`` URL so the sample
   matches a real run.
@chris-colinsky chris-colinsky merged commit 8519ba1 into main Jun 1, 2026
6 checks passed
@chris-colinsky chris-colinsky deleted the feature/example-chat-with-multimodal branch June 1, 2026 23:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants