中文 · 日本語 · Español · Français

A funny and easy AI — bring the most powerful tools to the most curious minds

▶ Watch the demo on YouTube

OpenBuddy is a Claude Code desk pet system built on M5Stack ESP32-S3 devices. It brings AI-powered voice interaction to your desktop through a cute animated companion that listens, thinks, and speaks — all driven by Claude Code under the hood.

Two hardware variants are supported: Cardputer (rectangular LCD) and StopWatch (circular AMOLED). A three-layer architecture — ESP32 firmware ↔ Python backend ↔ React WebUI — connects everything in real time via WebSocket.

• 🗣️ Voice-First Interaction — press a button, speak naturally, get a spoken response. Full STT → Agent → TTS pipeline
• 🐾 Desk Pet Animation — 6 animated states: idle, listening, thinking, speaking, error, disconnected
• 🔌 Claude Code Integration — hooks into Claude Code lifecycle events (Stop, PreToolUse, PostToolUse, Notification) in real time
• 🌐 Web Dashboard — configure settings, view live chat transcripts, explore files, manage prompts
• 📡 Auto-Discovery — mDNS service discovery via openbuddy.local, zero manual IP configuration
• 🎯 Dual Device Support — one server powers both Cardputer and StopWatch simultaneously
• 🔊 Bilingual Voice — auto-selects Chinese or English voice based on text content

┌─────────────┐       WebSocket        ┌─────────────────┐       WebSocket       ┌─────────────┐
│  ESP32       │◄──── /ws/openbuddy ───►│  Python Server   │◄──── /ws/webui ──────►│  React       │
│  Device      │      (binary+JSON)     │  (FastAPI)       │      (JSON events)    │  WebUI       │
└─────────────┘                         └────────┬────────┘                        └─────────────┘
                                                 │
                                          Voice Pipeline
                                                 │
                             ┌───────────────────┼───────────────────┐
                             ▼                   ▼                   ▼
                     ┌──────────────┐   ┌──────────────┐   ┌──────────────┐
                     │ STT          │   │ Agent        │   │ TTS          │
                     │ ElevenLabs   │──►│ LLM via      │──►│ ElevenLabs   │
                     │ Scribe v2   │   │ claude-agent  │   │ Eleven v3    │
                     └──────────────┘   │ -sdk         │   │ PCM16/16kHz  │
                                        └──────────────┘   └──────────────┘

Voice Pipeline (F3 Chain):

🎤 Mic → STT (ElevenLabs Scribe v2) → Qwen Cleanup → Agent → Qwen Cleanup → TTS (ElevenLabs v3) → 🔊 Speaker

git clone https://github.com/lennonkc/openbuddy.git
cd openbuddy

make dev
# Server on :8000 · WebUI on :5173

Open http://localhost:5173 → Settings panel, or use the CLI:

keyring set openbuddy elevenlabs <key>   # STT + TTS
keyring set openbuddy dashscope <key>    # Qwen text cleanup
keyring set openbuddy llm <key>          # Agent LLM

Add the following to ~/.claude/settings.json:

{
  "hooks": {
    "Stop": [{ "matcher": "", "hooks": [{
      "type": "command",
      "command": "curl -m 1 -X POST -H 'Content-Type: application/json' -d @- http://127.0.0.1:8000/hooks/Stop &"
    }]}]
  }
}

Apply the same pattern for UserPromptSubmit, PreToolUse, PostToolUse, Notification, and SessionStart. The Stop hook must be fire-and-forget (short timeout + & background) to avoid blocking.

source ~/esp/esp-idf/export.sh
make fw-stopwatch    # StopWatch

openbuddy/
├── openbuddy_server/    # Python FastAPI backend
│   ├── voice/           #   Voice pipeline (STT, TTS, Qwen)
│   ├── agent/           #   Agent lifecycle & scrubbing
│   ├── ws/              #   WebSocket endpoints
│   └── api/             #   REST API routes
├── openbuddy_webui/     # React WebUI (Vite + TypeScript + Tailwind)
├── stopwatch/           # StopWatch ESP32-S3 firmware
└── openbuddy_fun/       # Landing page (openbuddy.fun)

make lint    # Python: ruff check + format
make test    # pytest + vitest

• Python: ruff (line-length=100)
• TypeScript: ESLint with @/ path alias
• Firmware: ESP-IDF CMake, C++ namespaced by app

MIT

• 🌐 Website: openbuddy.fun
• 📦 GitHub: lennonkc/openbuddy