[Feature]: Real-time Voice Conversation Mode (voice-in, voice-out, low-latency)

[maxonliu]

Problem

Current interaction with Hermes Agent is text-only (CLI/TUI/IM platforms). There's no way to have a natural, real-time voice conversation with the agent — the kind where you speak, it listens, thinks, and responds verbally with low latency, like talking to a person.

With the rise of Thinking Machines Lab's Interaction Model, GPT-5 Realtime, and Google Gemini Omni, voice-first agent interaction is becoming a major trend in 2026. Hermes Agent, as the fastest-growing open-source agent framework, should lead in this direction too.

Proposed Feature

A real-time voice conversation mode in Hermes Agent's terminal/TUI that enables:

1. Microphone input → Capture user's speech
2. Streaming STT → Real-time speech-to-text (e.g., Whisper live / Deepgram / Realtime API)
3. Low-latency LLM response → Agent processes input and generates response with minimal delay
4. Streaming TTS → Natural-sounding voice output (e.g., ElevenLabs / edge-tts / OpenAI TTS)
5. Turn-taking — natural conversation flow with interruption handling

Use Cases

• Hands-free coding assistance (while debugging, reading code)
• Voice-driven research and Q&A
• Accessibility for users who prefer speech over typing
• Mobile-focused interactions (voice in = voice out)
• Natural conversation for brainstorming / thinking out loud with the agent

Suggested Architecture (High-Level)

┌─────────────┐     ┌──────────────┐     ┌─────────────┐
│ Microphone   │────▶│ STT Engine   │────▶│ Agent       │
│ (pyaudio/    │     │ (Whisper     │     │ (LLM)       │
│  portaudio)  │     │  live/RT API)│     │             │
└─────────────┘     └──────────────┘     └─────────────┘
                                               │
                                               ▼
┌─────────────┐     ┌──────────────┐     ┌─────────────┐
│ Speaker      │◀────│ TTS Engine   │◀────│ Agent       │
│ (pyaudio)   │     │ (ElevenLabs/ │     │ Response    │
│              │     │  edge-tts)   │     │ Text        │
└─────────────┘     └──────────────┘     └─────────────┘

Implementation Ideas

• Option A: Built-in voice mode in the TUI — toggle between text and voice with a hotkey
• Option B: Plugin/skill-based — voice is a standalone skill that wraps existing agent capabilities
• Option C: WebRTC-based voice gateway — connect via SIP/WebRTC for phone-like experience
• Latency optimization: Use streaming LLM responses + chunked TTS to achieve sub-500ms end-to-end latency

Related

• Architecture v0: control plane, brain, memory, voice, client, workers #32820 Architecture v0: control plane, brain, memory, voice, client, workers
• Hide or configure voice transcript status rows in editable dictation mode #33898 Hide or configure voice transcript status rows in editable dictation mode (TTS/STT infra already exists)
• [Feature]: Allow tts over ssh if pulseadio is reacheable #35622 Allow tts over ssh if pulseaudio is reachable

Why Now

2026 is being called "the year of voice agents" on X/Twitter. GPT-5 Realtime, Thinking Machines Lab's Interaction Model, and Google Gemini all point to voice-first interaction as the next frontier. Hermes Agent has a unique advantage — it's already the most-used open agent on OpenRouter. Adding real-time voice would be a killer differentiator.

[yungalgo]

Why voice

Voice is a powerful UX surface from first principles, because it lets you communicate information continuously and in duplex. Information via text is one-way and discrete. It's also much faster in words per second, so it's denser in the information communicated, and faster. This is why it makes sense to seriously double down and focus long-term engineering effort on it.

What this is

A working implementation of Option C from this issue, real-time WebRTC voice, as a Hermes gateway platform plugin. Posting here before opening a draft PR, per the discuss-before-PR convention.

The voice loop runs in-process with the agent session (a platform plugin, peer to plugins/platforms/telegram), not as an external orchestrator. For one audio stream the orchestration is thin: STT in, flush on end-of-turn, stream tokens out through TTS, cancel on barge-in. What matters is where that loop lives. In-process is the only place it can do things external harnesses can't:

• Barge-in that cancels generation. Interrupting calls agent.interrupt() and stops the LLM mid-generation. Pipecat, LiveKit Agents, and Vapi run the loop outside the agent and treat it as an LLM endpoint, so they can mute the audio but the model keeps generating server-side, wasting tokens and out of sync with what was actually heard. In-process, the interrupt reaches the model.
• Tool-aware speech. The loop sees the agent's tool events, so it can say "one sec, checking" while a tool runs instead of going silent.
• Voice as a native channel, not a separate client. Same agent, memory, and session whether you text it or call it.

I didn't pull in Pipecat or LiveKit Agents. They're good frameworks, but one audio stream doesn't need an SFU or SIP routing on day one, and a dependency that size has to earn its place. Transports are adapters around the one loop, so a heavier transport can slot in later without touching it.

The stack

• Transport: Daily (daily-python, BSD-2). Inexpensive, reliable WebRTC; per-device sample rates mean no resampling.
• STT and turn-taking: Deepgram Flux. One fused model does transcription and conversation-native turn detection, emitting start/eager-end/resumed/end-of-turn events the loop reacts to directly. That replaces a separate VAD, endpoint model, speculative-start, and energy barge-in with one component, so I made it opinionated rather than a swappable seam.
• TTS: Cartesia, over one persistent socket per call (a fresh connect is ~0.6-1.4s; reusing it keeps per-turn latency at synthesis time).

These aren't default picks. They're what's worked best for me after building a fair amount of voice, and I have a high bar for how it should sound. Daily, Cartesia, and Deepgram are all cheap, and together they give you good quality, low latency, and good prosody out of the box for browser communication. Interruption handling is solid and the prosody holds up. There's polish I'd still add, but it sounds good today. The model side stays swappable (below), so you're not locked to one provider.

The voice_model: slot

I added a top-level voice_model: block, a sibling to model: and fallback_model:. Real-time voice is latency-critical, and you usually don't want your main (possibly reasoning) model answering a call, so this points voice turns at a fast, non-reasoning model while the main model stays the same. Unset or auto falls back to the main model, and it resolves through the same provider surface as everything else (it mirrors _try_resolve_fallback_provider). I think a dedicated top-level slot is right, since voice is a full agentic turn in a different modality, not an auxiliary task, and the fallback_model precedent shows Hermes already has non-main sibling slots. But it touches core, so I'd want a maintainer's read on whether it belongs at the top level or somewhere else.

Scope and what's next

This is a small first cut. It's browser-only (WebRTC), with no telephony or SIP trunking yet, so you can't dial your agent from a phone number. There's no frontend in the Hermes client: standalone mode creates its own private Daily room and prints the join URL on startup (the token in the URL is the join permission), which you open in a browser. Bare-bones, but it works today.

Follow-ups, order open for discussion:

• A keyless, fully self-hosted harness: a keyless transport (Pipecat's SmallWebRTC or a raw aiortc peer) connects the agent and browser serverlessly. You run your own signaling and a TURN server, but need no third-party transport key. With local STT/TTS it becomes a no-API-key stack.
• SIP/PSTN trunking, the phone-like experience from this issue.
• A small "call your agent" surface in the web client, behind a flag so it's hidden unless voice is configured.

Each is another transport adapter or a flagged UI; none touch the loop.

Caller trust (#11063)

Right now this is the narrow case: owner-initiated WebRTC with single-use, short-lived tokens, so caller == owner. The design keeps #11063's model as the contract for the first non-owner transport (SIP/PSTN, shared rooms): per-call session isolation, a scoped tool list, per-call turn limits, and inbound allowlists, as gateway-level call policy rather than per-transport code.

Questions for maintainers

1. Is Option C as a gateway platform plugin the accepted lane?
2. Does a top-level voice_model: slot make sense, or would you prefer it elsewhere? (It touches core.)
3. Dependency posture: is an optional voice-platform extra with bounded pins (daily-python, websockets) acceptable, or would you prefer the first cut ship transport-less (loop + abstract transport + tests) with Daily as a follow-up?
4. Would you prefer it split (loop + fake transport, then the real transport)?

[teknium1]

Thanks for the detailed voice-mode proposal. This is an automated hermes-sweeper review; the report's core premise is no longer reproducible on current main because Hermes now has microphone-driven, voice-in/voice-out conversation support in both the CLI and Ink TUI.

• cli.py:11121-11302 captures microphone audio, detects silence, transcribes it, and feeds the resulting transcript into the next agent turn.
• ui-tui/src/app/createGatewayEventHandler.ts:603-630 routes voice.transcript events into normal TUI prompt submission.
• website/docs/user-guide/features/voice-mode.md:110-168 documents the continuous CLI/TUI loop: speak, silence detection, transcription, spoken response, and automatic re-listening.
• The initial CLI implementation landed in 1a6fbef8a9c046ee2d45da8534663b64453b6502 (feat: add voice mode with push-to-talk and TTS output for CLI).

The specialized Daily/Deepgram/Cartesia WebRTC direction discussed in the comments remains represented by open PR #51827, so it can be reviewed independently of this now-obsolete “text-only” feature request.

---

Closed as cannot_reproduce: the reported absence of voice conversation mode does not match current main.