Repository navigation
Protocol
Version: 0.1. Transport: WebSocket over TLS (wss://) in production, plaintext (ws://) on a trusted LAN. Text frames carry JSON; binary frames carry raw audio.
The client must include Authorization: Bearer <token> in the WS upgrade request. The bridge responds with HTTP 401 on any mismatch. The token is generated by server/install.sh and stored in /etc/hermes-voice-bridge/config.toml (mode 0600).
GET /ws HTTP/1.1
Host: hermes.local:8765
Upgrade: websocket
Authorization: Bearer 0f_RAndOmBASE64URL_32_BYTES
client server
│ hello │
│ ─────────────────────────────────────────────────────────────► │
│ welcome│
│ ◄───────────────────────────────────────────────────────────── │
│ │
│ start_turn │
│ ─────────────────────────────────────────────────────────────► │
│ audio_chunk × N (binary frame following each header) │
│ ─────────────────────────────────────────────────────────────► │
│ or │
│ transcript (final=true) if client-side STT │
│ ─────────────────────────────────────────────────────────────► │
│ end_turn │
│ ─────────────────────────────────────────────────────────────► │
│ thinking│
│ ◄───────────────────────────────────────────────────────────── │
│ response_text × N│
│ ◄───────────────────────────────────────────────────────────── │
│ response_audio_chunk × N + binary │
│ ◄───────────────────────────────────────────────────────────── │
│ response_end│
│ ◄───────────────────────────────────────────────────────────── │
Heartbeats: either side may send {"type": "ping"} at any time; the peer replies with {"type": "pong"}. The desktop client pings every 15 s and reconnects with exponential backoff if no pong arrives within 5 s.
Capabilities advertised in hello / welcome decide where STT and TTS run.
| Client capability | Meaning |
|---|---|
stt_local |
The client has whisper.cpp or an API key. |
tts_local |
The client has Piper or an API key. |
barge_in |
The client can interrupt mid-response. |
streaming_audio |
The client can both send and receive PCM. |
| Server capability | Meaning |
|---|---|
stt_server |
The bridge can transcribe incoming audio. |
tts_server |
The bridge can synthesise outgoing audio. |
streaming_text |
Server emits incremental response_text. |
streaming_audio |
Server emits response_audio_chunk frames. |
Default for the v0.1 bridge: only streaming_text. STT and TTS therefore happen on the client. Adding server-side STT is one adapter swap.
type ClientMessage =
| { type: "hello"; client_version: string; capabilities: string[] }
| { type: "start_turn"; turn_id: string; lang?: string }
| { type: "audio_chunk"; turn_id: string; seq: number }
// followed by a single binary frame: mono PCM16 LE @ 16 kHz
| { type: "end_turn"; turn_id: string }
| { type: "interrupt"; reason: "user_barge_in" | "user_cancel" }
| { type: "transcript"; turn_id: string; text: string; final: boolean }
| { type: "ping" };type ServerMessage =
| { type: "welcome"; session_id: string; server_version: string; capabilities: string[] }
| { type: "transcript"; turn_id: string; text: string; final: boolean }
| { type: "thinking"; turn_id: string }
| { type: "response_text"; turn_id: string; text: string; final: boolean }
| { type: "response_audio_chunk"; turn_id: string; seq: number;
format: "pcm16_24khz" | "mp3" }
// followed by a single binary frame in the declared format
| { type: "response_end"; turn_id: string }
| { type: "error"; code: string; message: string; turn_id?: string }
| { type: "pong" };audio_chunk and response_audio_chunk are two-frame envelopes: first a text JSON header carrying turn_id + seq, immediately followed by a single binary frame holding the bytes. Implementations MUST send the two frames atomically and MUST NOT interleave headers from different turns.
A binary frame with no preceding header is a protocol violation; the receiver SHOULD emit:
{ "type": "error", "code": "WS_INVALID_MESSAGE", "message": "binary frame without preceding header" }| Code | When |
|---|---|
WS_AUTH_FAILED |
Token rejected (HTTP 401 / error frame). |
WS_INVALID_MESSAGE |
Malformed JSON or unknown type. |
HERMES_UPSTREAM |
Bridge couldn't reach the Hermes HTTP API. |
TIMEOUT |
Server-side timeout waiting on Hermes. |
UNKNOWN |
Anything else. |
→ {"type":"hello","client_version":"0.1.0","capabilities":["stt_local","tts_local","barge_in"]}
← {"type":"welcome","session_id":"f3a9","server_version":"0.1.0","capabilities":["streaming_text"]}
→ {"type":"start_turn","turn_id":"t-001"}
→ {"type":"transcript","turn_id":"t-001","text":"que horas são","final":true}
→ {"type":"end_turn","turn_id":"t-001"}
← {"type":"thinking","turn_id":"t-001"}
← {"type":"response_text","turn_id":"t-001","text":"São","final":false}
← {"type":"response_text","turn_id":"t-001","text":" três da tarde.","final":false}
← {"type":"response_text","turn_id":"t-001","text":"São três da tarde.","final":true}
← {"type":"response_end","turn_id":"t-001"}
← {"type":"response_audio_chunk","turn_id":"t-002","seq":0,"format":"pcm16_24khz"}
← <binary: 480 bytes>
← {"type":"response_audio_chunk","turn_id":"t-002","seq":1,"format":"pcm16_24khz"}
← <binary: 480 bytes>
→ {"type":"interrupt","reason":"user_barge_in"} ← user starts speaking
→ {"type":"start_turn","turn_id":"t-003"} ← new turn already begins
The bridge is expected to stop sending chunks for the interrupted turn but MAY emit a final response_end for it.
Voice Gateway — VivaldiCode/voice-gateway. MIT licensed. Wiki sources live under docs/; run ./docs/sync-wiki.sh to publish.
Getting started
Architecture
- Architecture — system overview
- State-Machine — conversation FSM
- Conversation-Orchestrator — main glue
- WebSocket-Client — Hermes connection
- IPC-Layer — main ↔ renderer
- Settings-And-Storage — config persistence
Audio
- Audio-Pipeline — capture & playback
- Speech-To-Text — STT adapters
- Text-To-Speech — TTS adapters
- Wake-Word-Detection — openWakeWord
Server
- Hermes-Voice-Bridge — Python bridge
- Protocol — WebSocket spec
Platform
- Build-And-Packaging — electron-builder
- macOS-Permissions — TCC, signing
- Renderer-UI — React components
Project
- Testing-Guide — unit, integration, E2E
- Development-Guide — contributing
- Glossary — terms