Skip to content

Protocol

voice-gateway-bot edited this page May 25, 2026 · 1 revision

Voice Gateway ↔ Hermes Bridge WebSocket protocol

Version: 0.1. Transport: WebSocket over TLS (wss://) in production, plaintext (ws://) on a trusted LAN. Text frames carry JSON; binary frames carry raw audio.

Authentication

The client must include Authorization: Bearer <token> in the WS upgrade request. The bridge responds with HTTP 401 on any mismatch. The token is generated by server/install.sh and stored in /etc/hermes-voice-bridge/config.toml (mode 0600).

GET /ws HTTP/1.1
Host: hermes.local:8765
Upgrade: websocket
Authorization: Bearer 0f_RAndOmBASE64URL_32_BYTES

Lifecycle

client                                                          server
  │  hello                                                          │
  │ ─────────────────────────────────────────────────────────────► │
  │                                                          welcome│
  │ ◄───────────────────────────────────────────────────────────── │
  │                                                                 │
  │  start_turn                                                     │
  │ ─────────────────────────────────────────────────────────────► │
  │  audio_chunk × N  (binary frame following each header)          │
  │ ─────────────────────────────────────────────────────────────► │
  │   or                                                            │
  │  transcript (final=true)   if client-side STT                   │
  │ ─────────────────────────────────────────────────────────────► │
  │  end_turn                                                       │
  │ ─────────────────────────────────────────────────────────────► │
  │                                                         thinking│
  │ ◄───────────────────────────────────────────────────────────── │
  │                                                response_text × N│
  │ ◄───────────────────────────────────────────────────────────── │
  │                              response_audio_chunk × N + binary  │
  │ ◄───────────────────────────────────────────────────────────── │
  │                                                     response_end│
  │ ◄───────────────────────────────────────────────────────────── │

Heartbeats: either side may send {"type": "ping"} at any time; the peer replies with {"type": "pong"}. The desktop client pings every 15 s and reconnects with exponential backoff if no pong arrives within 5 s.

Capability negotiation

Capabilities advertised in hello / welcome decide where STT and TTS run.

Client capability Meaning
stt_local The client has whisper.cpp or an API key.
tts_local The client has Piper or an API key.
barge_in The client can interrupt mid-response.
streaming_audio The client can both send and receive PCM.
Server capability Meaning
stt_server The bridge can transcribe incoming audio.
tts_server The bridge can synthesise outgoing audio.
streaming_text Server emits incremental response_text.
streaming_audio Server emits response_audio_chunk frames.

Default for the v0.1 bridge: only streaming_text. STT and TTS therefore happen on the client. Adding server-side STT is one adapter swap.

Message reference

Client → server

type ClientMessage =
  | { type: "hello"; client_version: string; capabilities: string[] }
  | { type: "start_turn"; turn_id: string; lang?: string }
  | { type: "audio_chunk"; turn_id: string; seq: number }
      // followed by a single binary frame: mono PCM16 LE @ 16 kHz
  | { type: "end_turn"; turn_id: string }
  | { type: "interrupt"; reason: "user_barge_in" | "user_cancel" }
  | { type: "transcript"; turn_id: string; text: string; final: boolean }
  | { type: "ping" };

Server → client

type ServerMessage =
  | { type: "welcome"; session_id: string; server_version: string; capabilities: string[] }
  | { type: "transcript"; turn_id: string; text: string; final: boolean }
  | { type: "thinking"; turn_id: string }
  | { type: "response_text"; turn_id: string; text: string; final: boolean }
  | { type: "response_audio_chunk"; turn_id: string; seq: number;
        format: "pcm16_24khz" | "mp3" }
      // followed by a single binary frame in the declared format
  | { type: "response_end"; turn_id: string }
  | { type: "error"; code: string; message: string; turn_id?: string }
  | { type: "pong" };

Binary audio framing

audio_chunk and response_audio_chunk are two-frame envelopes: first a text JSON header carrying turn_id + seq, immediately followed by a single binary frame holding the bytes. Implementations MUST send the two frames atomically and MUST NOT interleave headers from different turns.

A binary frame with no preceding header is a protocol violation; the receiver SHOULD emit:

{ "type": "error", "code": "WS_INVALID_MESSAGE", "message": "binary frame without preceding header" }

Error codes

Code When
WS_AUTH_FAILED Token rejected (HTTP 401 / error frame).
WS_INVALID_MESSAGE Malformed JSON or unknown type.
HERMES_UPSTREAM Bridge couldn't reach the Hermes HTTP API.
TIMEOUT Server-side timeout waiting on Hermes.
UNKNOWN Anything else.

Examples

Welcome handshake

→ {"type":"hello","client_version":"0.1.0","capabilities":["stt_local","tts_local","barge_in"]}
← {"type":"welcome","session_id":"f3a9","server_version":"0.1.0","capabilities":["streaming_text"]}

Client-side STT turn

→ {"type":"start_turn","turn_id":"t-001"}
→ {"type":"transcript","turn_id":"t-001","text":"que horas são","final":true}
→ {"type":"end_turn","turn_id":"t-001"}
← {"type":"thinking","turn_id":"t-001"}
← {"type":"response_text","turn_id":"t-001","text":"São","final":false}
← {"type":"response_text","turn_id":"t-001","text":" três da tarde.","final":false}
← {"type":"response_text","turn_id":"t-001","text":"São três da tarde.","final":true}
← {"type":"response_end","turn_id":"t-001"}

Server-side TTS with barge-in

← {"type":"response_audio_chunk","turn_id":"t-002","seq":0,"format":"pcm16_24khz"}
← <binary: 480 bytes>
← {"type":"response_audio_chunk","turn_id":"t-002","seq":1,"format":"pcm16_24khz"}
← <binary: 480 bytes>
→ {"type":"interrupt","reason":"user_barge_in"}      ← user starts speaking
→ {"type":"start_turn","turn_id":"t-003"}            ← new turn already begins

The bridge is expected to stop sending chunks for the interrupted turn but MAY emit a final response_end for it.

Voice Gateway

Getting started

Architecture

Audio

Server

Platform

Project


Clone this wiki locally