Mermaid in chat: implementation proposal for an expanded viewer and error recovery

[jamesainslie]

I have a working Mermaid viewer in my fork and would like to share the implementation for possible future upstream use. This builds on Ideas #6812, with particular attention to inspecting large diagrams, recovering from invalid source, and sharing behavior with mobile.

I read the contribution guidance and the maintainer decision on #6191. I understand Mermaid is currently outside the product direction because its runtime, layout, security, and maintenance costs are not justified for the team. I am posting this as an implementation proposal in Ideas. Any upstream contribution would depend on maintainer interest and an agreed scope.

Problem and user experience

Agents frequently return architecture and flow diagrams as Mermaid fences. Reading the source makes relationships harder to follow, while copying it into another tool interrupts the conversation. Rendering alone leaves a second problem: a wide diagram can fit the chat column with labels too small to read.

The fork provides:

• A light/dark themed inline diagram with source toggle, source copy, and Expand.
• An expanded modal with Fit, Center, 100%, and zoom controls. Users can drag to pan, zoom around the pointer with the wheel, and pinch on touch devices. These interactions are confined to the expanded canvas, leaving normal timeline scrolling available inline.
• Keyboard controls while the canvas has focus: arrows to pan, Shift for larger steps, + / - to zoom, F to fit, C to center, 0 for actual size, and Escape to close. The web modal returns focus to its opener.
• A waiting state for an unfinished fence during streaming. A completed fence can render while the rest of the response continues.
• A readable source fallback when rendering fails, with error details and Retry. In an editable thread, Ask agent to fix appends the original source and renderer error to a composer draft. The user reviews and sends it; the action does not start a turn automatically or rewrite conversation history.

Proposed implementation

The implementation is in fork PR #7, merged as d4070441d. Its three commits separate shared rendering and dependencies, web integration, and mobile integration.

Shared helpers in packages/client-runtime handle fence completeness, repair prompts, rendering, and viewport geometry. Mobile uses the existing Markdown parser to split diagram blocks from surrounding text; web recognizes Mermaid through the existing ChatMarkdown code-fence renderer. Other code languages keep their current rendering path.

The renderer dynamically imports Mermaid, currently pinned to 11.17.2. Rendering is serialized because Mermaid configuration is global, and SVG promises are cached by source and light/dark theme with a 32-entry LRU limit. Failed entries are evicted so Retry can recover from transient loading failures. The fork sets limits of 50,000 source characters and 500 edges, uses strict security mode with HTML labels disabled, and protects trust and theme configuration from diagram-authored overrides. Temporary rendering nodes are cleaned up after failures.

Generated SVG is displayed through an SVG data-URL image. This isolates its styles and identifiers from the chat DOM. Mermaid still performs layout and temporary DOM work during generation. Viewport transforms operate on the resulting image, so panning and zooming do not regenerate the diagram. Event handlers and resize observers are disposed when the viewer closes.

Web and desktop share the React viewer and existing dialog primitives. The mobile prototype uses a native full-screen modal around a locally bundled Expo DOM/WebView viewer, reusing the renderer and viewport code. It integrates with both mobile Markdown rendering paths. Diagram generation requires no CDN or rendering service. This is a client presentation change with no database migration, provider adapter change, or new WebSocket contract; remote environments continue supplying the original Markdown.

Mitigations informed by Horde

The viewer design draws on my Horde implementation. It already addresses several practical costs through lazy component loading, rendering completed fences, reusing the generated SVG for pan/zoom, and preserving readable source on failure. The T3 adaptation adds shared caching, stricter rendering limits, SVG image isolation, and a native mobile integration.

Concern raised in earlier reviews	Implemented mitigation	Remaining cost or validation
Large frontend dependency	Horde loads the diagram component lazily; T3 dynamically imports Mermaid when a diagram needs rendering.	The runtime is still loaded for conversations that display diagrams.
Repeated layout and rendering work	Horde renders when diagram source changes. T3 waits for a completed streaming fence and caches SVG by source and theme, with 32 entries.	First-render layout still runs on the client main thread. Long-history performance and a byte-based cache budget need measurement.
Pan/zoom cost and interference with chat scrolling	Both viewers transform the generated image/SVG without asking Mermaid to lay out the graph again. Pointer and wheel interaction is confined to the expanded viewer.	Diagram paint cost and mobile gesture behavior still need device profiling.
Untrusted or malformed agent output	Horde uses strict mode and displays the failing source and error. T3 additionally disables HTML labels, protects configuration, limits source length and edge count, and displays output in an SVG image.	Mermaid parsing still processes untrusted input. These controls reduce exposure but do not replace security review.
Mobile renders different content	The T3 prototype uses the shared renderer and viewport code in a locally bundled Expo DOM/WebView, with native controls and a fullscreen modal.	Offline DOM rendering and Android export passed; native iOS/Android behavior and memory usage remain unverified.

Horde feeds renderer failures back to the agent, deduplicating repair requests per message and restricting them to the latest assistant message when the session is idle. T3 adapts that recovery path into an explicit Ask agent to fix action that prepares a draft, so the user controls whether another turn runs. Neither approach changes the original diagram silently.

Mobile availability and upstream scope

Mobile support is feasible with the existing stack. T3 Mobile uses React Native and Expo. Mermaid needs browser DOM APIs, so the prototype supplies them through a locally bundled WebView; it does not require a Swift port of Mermaid or an external rendering service. The earlier PRs left mobile unchanged, whereas this fork includes the integration. A desktop nightly does not distribute the native mobile app, which needs its own build and device validation.

The mobile DOM bundle is approximately 5.5 MB, and multiple WebViews can add runtime and memory cost. The prototype disables Expo DOM bundle splitting to work around exported asset references. That packaging choice needs upstream review and native-device testing. Sharing the renderer reduces duplicated implementation work, but the native shell still needs maintenance.

The fork renders diagrams automatically. An upstream variant could start with an explicit Render diagram action, keeping the renderer unloaded until requested. That is a proposed adaptation, not implemented behavior in this prototype. Scope and default behavior should be agreed before preparing a PR.

Before an upstream port, I would measure cold rendering, main-thread stalls, scrolling and remount behavior in virtualized histories, memory retention, and multiple mobile diagram views. Off-screen queued work needs review. Source copy is implemented, but whole-message and selection-to-Markdown copying need explicit regression coverage so rendered diagrams survive as their original fences. Accessibility beyond keyboard navigation, screen-reader behavior, additional Markdown consumers such as file previews and PR bodies, and remote-client checks remain part of the acceptance work.

Evidence and validation

These captures show the fork implementation.

Before	After

Pan and zoom recording · Invalid-source recovery · Offline mobile DOM capture

Validation completed in the fork:

• 54 focused tests passed across the existing chat Markdown suite and shared Mermaid tests, including streaming fences and invalid-source recovery.
• Scoped web, mobile, and shared-client typechecks passed, along with the web production build and Android Expo export. PR and post-merge CI passed.
• Browser checks covered both themes, mouse and keyboard controls, focus restoration, source errors, and repair drafts.
• The exported mobile DOM bundle rendered offline in Chromium and passed touch-pinch checks without HTTP requests. This was not a native app test; iOS and Android device testing remains outstanding.

Related work and contribution

I found #6812, the existing fork write-up linked there, #7497, and #9621. There is substantial overlap, including prior pan/zoom work. The performance objections in #8533 remain relevant to this implementation.

If Mermaid becomes worth revisiting, would an explicitly requested viewer with measured performance and a mobile path be a useful scope? I can prepare focused patches or contribute selected pieces to an existing implementation if that direction is welcome. The fork code and evidence are available for reference in the meantime.

[juliusmarminge]

Closing as a duplicate of the existing Mermaid Ideas thread: #6812

That discussion already tracks Mermaid-in-chat (including fork implementation notes). Please continue there so proposals stay in one place.

Related context already linked from that area: maintainer decision on #6191 (comment), plus prior PRs #7497, #8533, #9621.