Repository navigation
Agents
This document is the architectural contract for plugging a new agent backend into Voice Gateway. The desktop app already speaks a single protocol (the WebSocket flow described in Protocol) — we just need a small "adapter" service per agent that bridges that protocol to whatever the agent's native API looks like.
Voice Gateway shouldn't know about Hermes, Claude Code, ChatGPT, Cursor, or any other agent. The desktop app does the audio, the STT, the TTS, the FSM, and the UX. What it asks of the agent is narrow:
- "Here is a turn id and a user transcript. Stream me text back."
- "Tell me when you're done so I can clear the THINKING state."
- "Surface errors with a code I can render in a toast."
Anything the agent needs beyond that (file-system access, browser sessions, OS automation) is the agent's own problem and stays in the agent's process. The bridge is intentionally dumb plumbing.
A "bridge" is a small long-lived process that:
- Listens on a WebSocket endpoint (
/ws) using Voice Gateway's own protocol. - Authenticates the desktop client via a single bearer token (the one the user pastes into the pairing wizard).
- For each incoming turn (
transcriptframe,final=true), calls the agent backend and streams the textual reply back asresponse_textframes followed by aresponse_end.
The reference implementation is hermes-voice-bridge, which talks to a local Hermes daemon via its OpenAI-compatible chat-completions endpoint. New agents follow the same shape.
Each agent ships its own adapter Python module that implements a tiny interface:
from typing import AsyncIterator, Optional
class AgentAdapter:
"""Minimum surface every agent needs to expose to the bridge."""
async def stream_chat(
self, *, text: str, session_id: Optional[str] = None
) -> AsyncIterator[str]:
"""Yield text deltas for one user turn, keeping session
history. Closes the upstream on cancellation."""
...
def forget(self, session_id: str) -> None:
"""Drop any cached state for a session — called on WS close."""
...hermes_voice_bridge.hermes_adapter.HermesAdapter is the canonical
example. A new agent ships its own equivalent.
The intent (round-12 I4) is for the same server/install.sh
script to work for every agent. The user picks the agent via a flag:
# Hermes (default — current behaviour)
curl -fsSL ${INSTALL_SCRIPT_URL} | bash
# Future agents:
curl -fsSL ${INSTALL_SCRIPT_URL} | bash -s -- --agent=claude-code
curl -fsSL ${INSTALL_SCRIPT_URL} | bash -s -- --agent=chatgpt
curl -fsSL ${INSTALL_SCRIPT_URL} | bash -s -- --agent=cursor
curl -fsSL ${INSTALL_SCRIPT_URL} | bash -s -- --agent=openclawInternally each agent is a small set of bash functions in
server/install.sh that the dispatcher calls based on the flag:
-
agent_${name}_validate()— preflight checks (binary on PATH, API key reachable, etc.) -
agent_${name}_install_deps()— apt / dnf / brew incantations -
agent_${name}_write_config()— append agent-specific keys to/etc/hermes-voice-bridge/config.toml -
agent_${name}_systemd_unit()— produce the systemd unit text
The dispatcher (run_install()) always handles the bits that are
shared across all agents:
- The base bridge package + venv install
- The bearer token generation
- The
hermes-voicesystem user - Enabling + starting the systemd unit
- Printing the pairing block (URL + token) at the end
| Agent | Flag | Status | Notes |
|---|---|---|---|
| Hermes | --agent=hermes |
✅ shipping | Default. Local Hermes daemon on :8642. Uses hermes_adapter.HermesAdapter. |
| Claude Code | --agent=claude-code |
🚧 placeholder | Adapter file scaffolded but the actual SDK wiring is TBD. See #claude-code below. |
| ChatGPT | --agent=chatgpt |
🚧 placeholder | Will wrap the OpenAI chat-completions API; needs OPENAI_API_KEY in /etc/hermes-voice-bridge/secrets. |
| Cursor | --agent=cursor |
🚧 placeholder | Cursor exposes no public API yet; deferred until Cursor Agents ships. |
| OpenClaw | --agent=openclaw |
🚧 placeholder | Awaits OpenClaw's API spec; treat as "OpenAI-compatible" until proven otherwise. |
When you choose an unimplemented agent today the installer prints a clear "not yet supported — see docs/Agents.md" message and exits non-zero rather than producing a broken install.
Claude Code's native interface is the Anthropic SDK.
A claude_code_adapter would:
- Receive
text+session_id. - Call
client.messages.create(stream=True, ...)with the accumulated history. - Yield each
content_block_delta.delta.textchunk.
Pre-reqs the install script must surface:
-
pip install anthropicinto the bridge venv. -
[claude_code]section in the config TOML withapi_keyand optionalmodel(defaultclaude-sonnet-4-5).
Conceptually identical to Hermes — Hermes already speaks the
OpenAI-compatible chat-completions endpoint, so the existing
HermesAdapter would handle ChatGPT too. The only delta is the
config:
-
[openai]section withapi_keyandmodel. -
[openai].base_urldefaults tohttps://api.openai.com.
Both currently lack a stable third-party API surface. The agents stay registered in the dispatcher so the install command can fail gracefully with a pointer to this doc, instead of an opaque "unknown flag" error.
Round-12 ships only the dispatcher scaffolding (the flag, the table, the placeholder error). Future PRs that wire a real agent must add:
-
server/hermes-voice-bridge/tests/test_<agent>_adapter.pywith the same shape astest_server.py::test_hermes_*— scripted fake upstream, assert deltas + history. - A bash integration test that runs
bash server/install.sh --agent=<name> --dry-runand diff-asserts the generated systemd unit + config file.
This document is the source of truth for those expectations — if
the API of AgentAdapter changes, update this file first.
Voice Gateway — VivaldiCode/voice-gateway. MIT licensed. Wiki sources live under docs/; run ./docs/sync-wiki.sh to publish.
Getting started
Architecture
- Architecture — system overview
- State-Machine — conversation FSM
- Conversation-Orchestrator — main glue
- WebSocket-Client — Hermes connection
- IPC-Layer — main ↔ renderer
- Settings-And-Storage — config persistence
Audio
- Audio-Pipeline — capture & playback
- Speech-To-Text — STT adapters
- Text-To-Speech — TTS adapters
- Wake-Word-Detection — openWakeWord
Server
- Hermes-Voice-Bridge — Python bridge
- Protocol — WebSocket spec
Platform
- Build-And-Packaging — electron-builder
- macOS-Permissions — TCC, signing
- Renderer-UI — React components
Project
- Testing-Guide — unit, integration, E2E
- Development-Guide — contributing
- Glossary — terms