Skip to content

Multimodal

Jump to bottom
phinn edited this page Sep 18, 2026 · 2 revisions

🌐 Language: English | 中文

Multimodal (Image + Voice + Screenshot)

Direct engine supports multimodal input: images, voice transcription, and region screenshots.

Image input

📎 button in the composer (or drag-drop / paste):

  • Select one or more images (PNG / JPG / GIF / WebP).
  • Large images are sent as-is (no client-side resize — ponytail: MVP).
  • Images are injected into the prompt as a special marker: \x00IMAGES[{name, dataUrl}]\x00.
  • The Direct engine's AgentLoop parses this marker and converts to OpenAI image_url content parts (for OpenAI-compatible providers) or Anthropic base64 image blocks (for Anthropic).
  • dropTransient() strips base64 data before persisting to directHistory — keeps SQLite small.
User types: "What's wrong with this UI?"
           + attaches screenshot.png
           ↓
Prompt sent: "What's wrong with this UI?\x00IMAGES[{"name":"screenshot.png","dataUrl":"data:image/png;base64,..."}]\x00"
           ↓
Provider receives: [{type:"text", text:"What's wrong with this UI?"},
                    {type:"image_url", image_url:{url:"data:image/png;base64,..."}}]

Voice input (Whisper transcription)

🎤 button in the composer:

  1. Click → starts recording (browser MediaRecorder API).
  2. Click again → stops → audio encoded as base64.
  3. Sent to OpenAI Whisper API (whisper-1 model) for transcription.
  4. Transcribed text fills the composer input.
  5. Requires an OpenAI-compatible endpoint (Whisper is OpenAI-only).

For realtime voice conversation (bidirectional, natural TTS, Agent tool execution), see Voice-Chat.

TTS (Text-to-Speech):

  • Answers can be read aloud via the system speechSynthesis API.
  • Auto-detects language (Chinese vs English) from the text.
  • Strips markdown before reading (code blocks, links, formatting).
  • 2000-character limit per utterance.

Screenshot

📸 button in the chat area:

  1. Click → a full-screen overlay appears (semi-transparent dark layer).
  2. Drag to select a rectangular region.
  3. Release → the selected region is cropped from the screen capture.
  4. Cropped image is injected into the prompt as an image attachment.
  5. If selection is < 5px in either dimension, the screenshot is auto-cancelled (treat as a click, not a drag).

The overlay uses mouse events bound to document (not the overlay element) to prevent listener leaks and ensure mouseup is captured even outside the overlay.

Platform notes

  • Image multimodal requires a provider that supports vision (GLM-4V, GPT-4o, Claude 3.5, etc.).
  • Voice transcription requires whisper-1 model access on the configured endpoint.
  • Screenshots use Electron's desktopCapturer + canvas cropping.
  • Claude Code and Codex engines do not support in-app multimodal — use their native capabilities instead.

Key source files

  • src/main/AgentLoop.ts — \x00IMAGES[...]\x00 marker parsing → ContentPart[]
  • src/main/glm.ts — image_url (OpenAI) ↔ base64 image (Anthropic) conversion; dropTransient()
  • src/renderer/app.ts — wireVoice() (recording + Whisper), wireScreenshot() (overlay + crop), file attachment handling

Clone this wiki locally