DesktopWallE

A LAN-resident desktop robot that combines an M5Stack CoreS3 + StackChan hardware platform with a Python bridge daemon and an LLM agent. You wake it with a hotword, talk to it, and it talks back through Doubao (ByteDance Volcengine) ASR/TTS. Heading, eyes, LEDs, motors, camera, sensors — everything is steerable from the agent loop.

Inspired by Pixar's Wall‑E — small, expressive, and very chatty.

┌─────────────────────┐         ws://    ┌─────────────────────┐    https      ┌────────────────────┐
│  M5 CoreS3 + Stack  │  ◀──────────────▶│   bridge (Python)    │ ──────────▶ │  Doubao ASR Flash  │
│  Chan firmware      │  ocsc.v2 binary  │   server.py (asyncio)│ ──────────▶ │  Doubao TTS chunked│
│  (Arduino C++)      │   audio + RPC    │                      │             └────────────────────┘
└─────────────────────┘                  │  + Hermes / OpenClaw│
        │   I2S mic / I2S DAC            │    agent gateway     │
        │   M5.Speaker.playRaw           └──────────┬───────────┘
        ▼                                            │ HTTP
   speaker + screen + LEDs + servos                  ▼
                                              LLM (any provider)

What is in this repo

Path	Language	What it is
src/main.cpp	Arduino C++	Firmware for the M5Stack CoreS3 + StackChan. Single TU. Exposes 34 HTTP endpoints + an ocsc.v2 WebSocket client.
StackChanOpenClaw/	Arduino CLI shim	One-line .ino so Arduino IDE / arduino-cli accepts the project. PlatformIO uses src/ directly.
include/config.example.h	C header	Template for the per-device config (Wi-Fi, bridge host, optional auth token). Copy to include/config.h.
bridge/	Python 3.13	Long-lived daemon that holds the WebSocket connection to the device, runs Doubao Flash ASR + chunked TTS, and forwards transcripts to the agent.
bridge/doubao_asr_flash.py	Python	One-shot Flash ASR client (/api/v3/auc/bigmodel/recognize/flash). Encodes audio as OGG/OPUS @16 kbps to keep uploads small.
bridge/doubao_tts_unidirectional.py	Python	HTTP Chunked unidirectional TTS client (/api/v3/tts/unidirectional). Streams PCM frames as they arrive — first audio chunk lands in ~400–800 ms.
bridge/test_doubao_new.py	Python	Self-test that round-trips text → TTS → PCM → Flash ASR → text and asserts equality.
hermes-stackchan-skill/	Hermes skill	Installs into Hermes Agent so it can drive the device.
hermes-hotdog-skill/ + hermes-hotdog-plugin/	Hermes skill + plugin	Personality skin for the agent — turns the bridge into "热狗" (Hotdog) the desk robot.
openclaw-stackchan-skill/	OpenClaw skill	Same idea for the OpenClaw agent host.
tools/openclaw_stackchan_demo.sh	Bash	Curl-driven smoke test that exercises every HTTP endpoint on the device.

Hardware

Part	Notes
M5Stack CoreS3 (ESP32-S3)	240 MHz dual-core, 8 MB PSRAM, 16 MB flash, GC0308 camera, BMI270 IMU, BMM150 mag, AXP2101 PMU, AW9523 GPIO expander, 320×240 IPS LCD with capacitive touch, NS4168 I2S DAC + speaker, dual MEMS mics, USB-C, microSD.
StackChan base (servo type B)	2× SG90 servos (yaw + pitch), capacitive touch headplate, 12-pixel RGB ring around the neck. The Mongonta stack-chan repo is the upstream of the mechanics.
Optional	M5Stack speaker module if you want louder output; otherwise the onboard NS4168 is fine.

Quick start

1. Create the two local config files

Both are gitignored — your keys never leave your machine.

# Device-side config (Wi-Fi, bridge host, optional auth token)
cp include/config.example.h include/config.h
$EDITOR include/config.h

# Bridge-side config (Doubao keys, server ports, agent URLs)
cp bridge/config.toml.example bridge/config.toml
$EDITOR bridge/config.toml

You only need to fill in:

• include/config.h — STACKCHAN_WIFI_SSID / STACKCHAN_WIFI_PASSWORD and STACKPROXY_WS_HOST (the LAN IP of whatever machine will run the bridge).
• bridge/config.toml — [doubao] appid and access_token from the Volcengine speech console.

Everything else has sensible defaults.

2. Flash the firmware

arduino-cli compile --fqbn m5stack:esp32:m5stack_cores3 \
    --build-path .arduino-build StackChanOpenClaw
arduino-cli upload  -p /dev/cu.usbmodemXXX --fqbn m5stack:esp32:m5stack_cores3 \
    --input-dir .arduino-build StackChanOpenClaw
arduino-cli monitor -p /dev/cu.usbmodemXXX -c baudrate=115200

PlatformIO works too:

pio run
pio run -t upload --upload-port /dev/cu.usbmodemXXX
pio device monitor -p /dev/cu.usbmodemXXX -b 115200

The device boots, joins Wi-Fi, advertises mDNS as stackchan.local, and opens a WebSocket to your bridge host. Tap its head; you should see head.tap events arrive at the bridge.

3. Run the bridge

The bridge is Python 3.13. It expects ffmpeg on the PATH for the OPUS encoder.

brew install ffmpeg python@3.13          # macOS; apt equivalents on Linux
cd bridge
python3.13 -m pip install -r requirements.txt
python3.13 server.py

You should see:

config: doubao=ok ws=0.0.0.0:8765 http=0.0.0.0:8766 source=bridge/config.toml
[ws]    ocsc.v2 listening on 0.0.0.0:8765
[http]  admin listening on 0.0.0.0:8766
[ws]    hello <device-id> boot=N fw=1.0.0 caps=[...] remote=...
[disp]  idle face -> <device-id>

If bridge/config.toml is missing or has placeholder values, the bridge exits immediately with a one-line fix-up message — no silent crashes mid-call.

4. Smoke-test the audio path

python3.13 bridge/test_doubao_new.py

Expected output:

--- round-trip: '今天天气真不错' ---
  TTS: 52292 B PCM (1.63s audio) in 921ms first=625ms chunks=10
  ASR: text='今天天气真不错。' duration=1634ms utts=1
...
--> 3/3 passed

Then poke the live audio path directly:

curl -X POST http://<bridge-host>:8766/say \
    -H 'Content-Type: application/json' \
    -d '{"text":"你好，我是热狗，今天有什么需要帮忙的吗？","device":"<device-id>"}'

The device should start speaking within ~600 ms of the POST.

5. (Optional) Plug in an agent

The bridge is just a protocol translator. Point it at any chat agent:

• Hermes — copy hermes-stackchan-skill/ into ~/.hermes/skills/. The skill exposes sc commands like sc say "hello", sc face happy, sc move nod.
• OpenClaw — copy openclaw-stackchan-skill/ into ~/.openclaw/workspace/skills/. Same idea via OpenClaw's tool API.
• Anything else — the bridge speaks HTTP. Wire it to your own LLM loop in 50 lines of Python.

Configuration files

This project has exactly two config files — both gitignored, each with a committed .example:

File	What it configures	Loaded by	Template
include/config.h	Device side: Wi-Fi SSID/password, bridge host:port, optional HTTP auth token	Compiled into firmware at build time	include/config.example.h
bridge/config.toml	Bridge side: Doubao credentials, WS/HTTP/MCP ports, agent gateway URLs	Read at bridge startup by bridge/config.py	bridge/config.toml.example

The bridge loader looks for bridge/config.toml first, then falls back to ~/.stackproxy/config.toml for legacy installs. No environment variables are consulted — the whole runtime contract lives in one TOML file. Run python3.13 bridge/config.py to print a masked sanity report (values redacted to first/last 4 chars) and verify your file is being picked up.

To rotate keys: edit the TOML and restart server.py. No code change, no env var dance, no .env files.

Architecture deep dive

Why a bridge daemon (not on-device)?

Earlier iterations tried to run TLS + JSON + base64 + Volcengine signing on the ESP32 directly. The CoreS3 has the headroom but the stack-blow risk and per-call latency made the agent loop feel sluggish. Moving everything to a Mac/Linux box on the same LAN lets us:

• Reuse the Python ecosystem (httpx, websockets, asyncio) for fast iteration.
• Hold a single persistent ocsc.v2 WS to the device — no per-call TCP handshake.
• Stream audio over a binary WS frame (8-byte header + raw int16 LE PCM) at exactly 1× realtime so the device's I2S DMA never underruns.
• Make protocol upgrades (e.g. swapping TTS vendors) a one-line change instead of an OTA.

Why Flash ASR + OGG/OPUS encoding?

The naive "streaming Doubao bidirectional ASR" path turned out to be very fiddle-prone: the server requires you to feed packets at exactly the right rate or it kills the session with the dreaded 45000081 等包超时 error. Flash ASR (/api/v3/auc/bigmodel/recognize/flash) is a synchronous REST call — no pacing, no per-packet timer, no session state. For utterance-length audio (<10 s) it returns the full transcript in ~500–2000 ms.

The one catch: Flash takes the entire audio in one POST body. A 1.6 s WAV is ~70 KB → fine on most networks, but on bandwidth-shaped uplinks (mine maxes out at ~2 KB/s to Doubao) it would time out. OGG/OPUS at 16 kbps brings the same 1.6 s of speech down to ~5 KB JSON — uploads in ~1.5 s end-to-end. We use ffmpeg as the encoder (10–25 ms per call, negligible).

Why HTTP Chunked TTS (not WebSocket)?

The unidirectional endpoint (/api/v3/tts/unidirectional) is Transfer-Encoding: chunked + gzip + NDJSON. Each line is {"code":0,"data":"<base64-PCM>"} plus a final sentence-end marker. httpx.AsyncClient.stream + aiter_lines() does all the work. We never see WebSocket framing, never have to track sequence numbers, never have to gzip-decode by hand. First audio chunk lands ~400 ms after the POST and the entire 5 s reply ships in ~1.5 s.

The bridge then pipes those chunks straight to the device via stream_pcm_async() — see bridge/server.py. The device cushion fills from a 16-chunk (1 s) prebuffer; after that every chunk is sent at exactly t0 + N * 60 ms so the I2S queue never starves.

ocsc.v2 — the device wire protocol

Frame	Direction	Payload
hello / hello.ack	both	Handshake + capability negotiation + boot counter for ghost-session detection.
req / res / err	both	RPC envelope. Methods include say, face, motion.action, led.set, tts_stop, mic.start, cam.capture, show_text, play_pcm.
evt	device → bridge	Async events: head.tap, screen.tap, tts.done, mic.end, imu.shake, wake.fired.
mic.start / mic.end	bridge → device	Bracket a mic streaming session.
tts.start / tts.end	bridge → device	Bracket a TTS stream.
ping / pong	both	30 s heartbeat. Two misses → reconnect.
binary frame	both	8-byte header (version + kind + sid + seq) + raw payload. Kinds: MIC_PCM (device→bridge), TTS_PCM (bridge→device), CAM_JPEG, DISP_IMG, DISP_RGB565.

See bridge/protocol.py for the encoder/decoder.

Firmware HTTP API (legacy / admin path)

The firmware also exposes a full HTTP+JSON API on port 80 — useful for admin work, demos, and pre-bridge testing. All endpoints accept an optional X-StackChan-Token header (set STACKCHAN_TOKEN in config.h to require auth).

• Movement — /head, /head/normalized, /head/rotate, /head/stop, /head/torque, /head/calibrate, /home.
• Choreography — /action (home, nod, shake, yes, no, left, right, look_up, look_down, look_around, dance, surprised, sleep, wake, panic, peek, tilt_left, tilt_right, bow).
• Idle — /idle, /breathing.
• Display — /display, /display/big, /display/marquee, /display/progress, /display/brightness, /display/clear, /display/status, /display/face (16 expressions: neutral, happy, smile, love, sad, angry, surprised, thinking, sleep, wink_l, wink_r, stare, dead, embarrassed, cat, talking).
• LEDs — /led, /led/effect (15 effects: rainbow, breathing, pulse, scanner, wipe, sparkle, police, fire, chase, theater, listening, thinking, talking, solid, off).
• Audio — /speak {url} (background WAV download + playback), /mic/record {seconds, upload_url} (record + auto-POST).
• Sensors — /sensors (IMU + magnetometer + touch + battery + camera), /events (ring buffer of physical events).
• Camera — /camera/capture, /camera/config, /camera/preview.
• System — /status, /battery, /api, /volume, /reboot.

tools/openclaw_stackchan_demo.sh exercises every one of these so you can confirm the firmware is healthy.

Note on /camera/capture: there's a known bug where the esp_camera driver can lock up the device (requires physical power cycle to recover). The Hermes/OpenClaw skills gate this command behind a --force flag for safety.

License

MIT. Doubao / Volcengine credentials are yours and never live in the repo; see bridge/config.toml.example.

Acknowledgements

• The Mongonta stack-chan repo for the mechanical platform and the inspiration.
• M5Stack for the CoreS3.
• ByteDance / Volcengine for the Doubao Flash ASR + unidirectional TTS endpoints — both substantially better than their bidirectional-streaming counterparts for short-turn voice loops.
• Wall‑E. Always Wall‑E.