Add chat-with-multimodal example#116
Merged
Merged
Conversation
New examples/11-chat-with-multimodal/ demonstrates the headline of proposal 0046 (ChatPrompt + PlaceholderSegment, shipped in v0.11.0) end-to-end: - ChatPrompt with ContentSegment (system + user) and PlaceholderSegment for chat-history injection. - Multi-turn conversation memory threaded through state via the ``append`` reducer; each turn's render() sees the full prior history through the placeholder slot. - Multimodal turn: one of four scripted turns attaches a NASA public-domain Apollo 16 LM "Orion" photograph via ImageURLBlockTemplate. Same chat template, only the trailing user ContentSegment's content shape changes (string vs content-blocks list); system + placeholder segments are identical across both shapes. - Error handling at the invoke() boundary: try/except NodeException in main(), inspect exc.__cause__ for LlmProviderError to surface the canonical category string. Comments name the other two legitimate handler locations (RetryMiddleware, node-internal try/except). - Streaming transcript: each turn prints from inside the respond node body so the conversation arrives as the graph executes rather than waiting for invoke() to return. A 0.5s ``_TURN_DELAY_S`` pacing constant lets the reader follow along. - ``--traces`` argparse flag (default off) opts in to the OTel observer with a console exporter. Without it the chat runs without any observer attached; with it, JSON spans stream to stderr alongside the conversation on stdout. Complementary to example 09 (tool calling); chat history threading and tool calling are separate primitives. Example 03 owns the observer-hooks story, so this example points readers there for the observability details rather than re-teaching them. Bonus pickup: docs/examples/index.md was missing example 10 from its catalog before this PR. Caught alongside the example 11 entry and added.
There was a problem hiding this comment.
Pull request overview
Adds a new runnable + documented example demonstrating multi-turn chat history injection with ChatPrompt/PlaceholderSegment, including one multimodal (image URL) turn, and wires it into the docs navigation and example catalogs.
Changes:
- Add
examples/11-chat-with-multimodal/demo script showing conversation memory viaappend+ placeholder injection and optional OTel tracing. - Add docs page + navigation/catalog entries for example 11 (and fill the missing catalog entry for example 10).
- Update generated examples inventory (
AGENTS.md) and changelog entry.
Reviewed changes
Copilot reviewed 6 out of 6 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| src/openarmature/AGENTS.md | Registers example 11 in the generated examples inventory. |
| mkdocs.yml | Adds example 11 to the docs nav under Examples. |
| examples/11-chat-with-multimodal/main.py | New end-to-end demo script for chat history + multimodal turn + optional tracing. |
| docs/examples/index.md | Adds catalog entries for examples 10 and 11. |
| docs/examples/11-chat-with-multimodal.md | New documentation page for example 11. |
| CHANGELOG.md | Adds Unreleased “Added” entries for example 11 and the catalog fix. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Three CoPilot findings on PR #116: 1. main.py docstring referenced PlaceholderSegment(name="history") but the actual field is named ``placeholder``. The code's chat-template construction uses ``placeholder=`` correctly; only the docstring narrative was wrong. A reader copy-pasting from the docstring would have hit a Pydantic ValidationError. 2. Walk-through doc's intro paragraph still said "Apollo 11 Lunar Module" after the image URL swap to the Apollo 16 "Orion" shot. The code's docstring + inline comment got updated when the URL changed but the walk-through intro was missed. 3. Sample "Reading the output" block showed an ``upload.wikimedia.org`` image URL the example explicitly warns against. Updated to the actual default ``images-assets.nasa.gov`` URL so the sample matches a real run.
This was referenced Jun 2, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
First of three new examples picked from the audit (this one, plus production-observability-with-timing-middleware and a crash-and-resume drama on example 08 to follow).
New
examples/11-chat-with-multimodal/demonstrates the headline of proposal 0046 (ChatPrompt+PlaceholderSegment, shipped in v0.11.0) end-to-end:ContentSegment(system + user) andPlaceholderSegmentfor chat-history injection. Multi-turn conversation memory threads through state via theappendreducer; each turn'srender()injects the full prior history at the placeholder slot.ImageURLBlockTemplate. Same chat template; only the trailing userContentSegment'scontentshape changes (string vs content-blocks list). System + placeholder segments are identical across both shapes.main()catchesNodeException, inspectsexc.__cause__forLlmProviderErrorto surface the canonical category. Comments name the other two legitimate handler locations (RetryMiddleware, node-internal try/except).respondnode body so the conversation arrives as the graph executes. A_TURN_DELAY_S = 0.5spacing constant lets the reader follow along.--tracesargparse flag (default off) opts in to the OTel observer with a console exporter. Without it the chat runs without any observer attached; with it, JSON spans stream to stderr alongside the conversation on stdout.Complementary to example 09 (tool calling); chat history threading and tool calling are separate primitives. Example 03 owns the observer-hooks story end-to-end, so this example points readers there for the observability details rather than re-teaching them.
Bonus pickup:
docs/examples/index.mdwas missing example 10 from its catalog before this PR. Caught alongside the example 11 entry and added.Things worth noting
upload.wikimedia.org) block OpenAI's image fetcher and return aProviderInvalidRequest. The default URL points atimages-assets.nasa.govwhich is known to work. Comment in the code calls this out so users overridingIMAGE_URLknow what's safe._NoFetchBackendstub. The example constructsChatPromptobjects inline and callsmanager.render()directly without going throughfetch(). The backend stub exists purely to satisfyPromptManager.__init__'s "at least one backend" check. Production deployments supply a real backend and callmanager.fetch(name, label)before render.Test plan
gpt-4o-mini: 4 turns, multimodal turn 2 returns a real LM description, conversation memory carries forward (turn 3 references "the LM you described"), final state has 8 messages.Out of scope
FilesystemPromptBackendto support chat-template sidecars so this kind of demo can use the canonical backend rather than an inline stub. Worth a follow-on.