# Voice Gateway ↔ Hermes Bridge WebSocket protocol Version: `0.1`. Transport: WebSocket over TLS (`wss://`) in production, plaintext (`ws://`) on a trusted LAN. Text frames carry JSON; binary frames carry raw audio. ## Authentication The client must include `Authorization: Bearer ` in the WS upgrade request. The bridge responds with HTTP 401 on any mismatch. The token is generated by `server/install.sh` and stored in `/etc/hermes-voice-bridge/config.toml` (mode `0600`). ``` GET /ws HTTP/1.1 Host: hermes.local:8765 Upgrade: websocket Authorization: Bearer 0f_RAndOmBASE64URL_32_BYTES ``` ## Lifecycle ``` client server │ hello │ │ ─────────────────────────────────────────────────────────────► │ │ welcome│ │ ◄───────────────────────────────────────────────────────────── │ │ │ │ start_turn │ │ ─────────────────────────────────────────────────────────────► │ │ audio_chunk × N (binary frame following each header) │ │ ─────────────────────────────────────────────────────────────► │ │ or │ │ transcript (final=true) if client-side STT │ │ ─────────────────────────────────────────────────────────────► │ │ end_turn │ │ ─────────────────────────────────────────────────────────────► │ │ thinking│ │ ◄───────────────────────────────────────────────────────────── │ │ response_text × N│ │ ◄───────────────────────────────────────────────────────────── │ │ response_audio_chunk × N + binary │ │ ◄───────────────────────────────────────────────────────────── │ │ response_end│ │ ◄───────────────────────────────────────────────────────────── │ ``` Heartbeats: either side may send `{"type": "ping"}` at any time; the peer replies with `{"type": "pong"}`. The desktop client pings every 15 s and reconnects with exponential backoff if no pong arrives within 5 s. ## Capability negotiation Capabilities advertised in `hello` / `welcome` decide where STT and TTS run. | Client capability | Meaning | |--------------------|--------------------------------------------| | `stt_local` | The client has whisper.cpp or an API key. | | `tts_local` | The client has Piper or an API key. | | `barge_in` | The client can interrupt mid-response. | | `streaming_audio` | The client can both send and receive PCM. | | Server capability | Meaning | |--------------------|--------------------------------------------| | `stt_server` | The bridge can transcribe incoming audio. | | `tts_server` | The bridge can synthesise outgoing audio. | | `streaming_text` | Server emits incremental `response_text`. | | `streaming_audio` | Server emits `response_audio_chunk` frames.| Default for the v0.1 bridge: only `streaming_text`. STT and TTS therefore happen on the client. Adding server-side STT is one adapter swap. ## Message reference ### Client → server ```ts type ClientMessage = | { type: "hello"; client_version: string; capabilities: string[] } | { type: "start_turn"; turn_id: string; lang?: string } | { type: "audio_chunk"; turn_id: string; seq: number } // followed by a single binary frame: mono PCM16 LE @ 16 kHz | { type: "end_turn"; turn_id: string } | { type: "interrupt"; reason: "user_barge_in" | "user_cancel" } | { type: "transcript"; turn_id: string; text: string; final: boolean } | { type: "ping" }; ``` ### Server → client ```ts type ServerMessage = | { type: "welcome"; session_id: string; server_version: string; capabilities: string[] } | { type: "transcript"; turn_id: string; text: string; final: boolean } | { type: "thinking"; turn_id: string } | { type: "response_text"; turn_id: string; text: string; final: boolean } | { type: "response_audio_chunk"; turn_id: string; seq: number; format: "pcm16_24khz" | "mp3" } // followed by a single binary frame in the declared format | { type: "response_end"; turn_id: string } | { type: "error"; code: string; message: string; turn_id?: string } | { type: "pong" }; ``` ## Binary audio framing `audio_chunk` and `response_audio_chunk` are two-frame envelopes: first a text JSON header carrying `turn_id` + `seq`, immediately followed by a single binary frame holding the bytes. Implementations MUST send the two frames atomically and MUST NOT interleave headers from different turns. A binary frame with no preceding header is a protocol violation; the receiver SHOULD emit: ```json { "type": "error", "code": "WS_INVALID_MESSAGE", "message": "binary frame without preceding header" } ``` ## Error codes | Code | When | |--------------------|-----------------------------------------------| | `WS_AUTH_FAILED` | Token rejected (HTTP 401 / `error` frame). | | `WS_INVALID_MESSAGE` | Malformed JSON or unknown `type`. | | `HERMES_UPSTREAM` | Bridge couldn't reach the Hermes HTTP API. | | `TIMEOUT` | Server-side timeout waiting on Hermes. | | `UNKNOWN` | Anything else. | ## Examples ### Welcome handshake ``` → {"type":"hello","client_version":"0.1.0","capabilities":["stt_local","tts_local","barge_in"]} ← {"type":"welcome","session_id":"f3a9","server_version":"0.1.0","capabilities":["streaming_text"]} ``` ### Client-side STT turn ``` → {"type":"start_turn","turn_id":"t-001"} → {"type":"transcript","turn_id":"t-001","text":"que horas são","final":true} → {"type":"end_turn","turn_id":"t-001"} ← {"type":"thinking","turn_id":"t-001"} ← {"type":"response_text","turn_id":"t-001","text":"São","final":false} ← {"type":"response_text","turn_id":"t-001","text":" três da tarde.","final":false} ← {"type":"response_text","turn_id":"t-001","text":"São três da tarde.","final":true} ← {"type":"response_end","turn_id":"t-001"} ``` ### Server-side TTS with barge-in ``` ← {"type":"response_audio_chunk","turn_id":"t-002","seq":0,"format":"pcm16_24khz"} ← ← {"type":"response_audio_chunk","turn_id":"t-002","seq":1,"format":"pcm16_24khz"} ← → {"type":"interrupt","reason":"user_barge_in"} ← user starts speaking → {"type":"start_turn","turn_id":"t-003"} ← new turn already begins ``` The bridge is expected to stop sending chunks for the interrupted turn but MAY emit a final `response_end` for it.