feat: add multimodal support (voice, camera, screen, MCP)

[JacobFV]

Multimodal Support - AGI CLI Updates

This update adds comprehensive multimodal support to the AGI CLI.

New Features

Voice Mode (--voice)

• Audio input from microphone
• Automatic turn detection
• Text-to-speech output
• Requires: OPENAI_API_KEY environment variable

Camera Mode (--camera)

• Webcam video feed
• 30-second rolling buffer
• Agent can see you

Screen Mode (--screen)

• Screen recording
• 30-second rolling buffer
• Agent can see your screen

MCP Support (--mcp)

• Load MCP servers from config
• Default config: ~/.agi/mcp.json
• Custom config: --mcp-config /path/to/mcp.json

Usage Examples

Voice Mode

agi --voice "What's the current time?"

Voice + Screen

agi --voice --screen "What's on my screen?"

Full Multimodal

agi --voice --camera --screen "Can you see me and my screen?"

MCP Servers

# Set up MCP config
mkdir -p ~/.agi
cat > ~/.agi/mcp.json << 'EOF'
{
  "filesystem": {
    "command": "npx",
    "args": ["-y", "@modelcontextprotocol/server-filesystem", "/Users/you/Documents"]
  }
}
EOF

# Use MCP
agi --mcp "List my documents"

Everything Combined

agi --voice --camera --screen --mcp "Help me with my work"

Configuration

Environment Variables

• AGI_API_KEY: Your AGI API key (required)
• OPENAI_API_KEY: OpenAI key for voice features (required for --voice)

MCP Config Format

{
  "server-name": {
    "command": "executable",
    "args": ["arg1", "arg2"],
    "env": {
      "ENV_VAR": "value"
    }
  }
}

CLI Options

Option	Description
--voice	Enable voice input/output
--camera	Enable camera video
--screen	Enable screen recording
--mcp	Load MCP servers from config
--mcp-config PATH	Custom MCP config path
-m, --model	Model to use (default: claude-sonnet)
-v, --verbose	Show agent thinking
--no-confirm	Auto-approve confirmations

Implementation

Changes made:

• Updated src/cli.ts to add multimodal options
• Updated src/hooks/useAgent.ts to pass multimodal config to driver
• Added UI components for multimodal events
• Updated examples in help text

Testing

# Install dependencies
npm install

# Build
npm run build

# Test voice mode
agi --voice "Hello"

# Test full multimodal
agi --voice --camera --screen --mcp "What do you see?"

Related PRs

• agi-api (driver): https://github.com/agi-inc/agents/pull/344
• agi-python: https://github.com/agi-inc/agi-python/pull/8
• agi-node: feat: add comprehensive multimodal driver support agi-node#11
• agi-csharp: feat: add comprehensive multimodal driver support documentation agi-csharp#8

[JacobFV]

Merge Order

This PR depends on @agi_inc/agi-js@0.5.0 which has the multimodal DriverOptions types. CI will fail until the Node SDK PR is merged and published:

1. agi-inc/agents #344 — merge first (driver binary)
2. agi-inc/agi-node feat: redesign TUI and add update command #11 → publish @agi_inc/agi-js@0.5.0 to npm
3. agi-inc/agi-cli feat: add multimodal support (voice, camera, screen, MCP) #36 — merge after agi-js 0.5.0 is published
4. agi-inc/agi-python feat(auth): add device code login flow and credential management #8 and agi-inc/agi-csharp feat(auth): add device code login flow and credential management #8 — can merge independently

The import.meta typecheck error is a pre-existing tsconfig issue (tsup outputs ESM correctly, tsc --noEmit false positive).