Voice calls with a thread: hear what the agent is doing and steer it by voice

[d0dge-dev]

What I have in mind

A voice call with a thread: I start a call, talk, the agent works, I hear what it's doing, and I can keep talking to it, without the keyboard or reading the screen. Like the voice mode in ChatGPT, but with my own T3 Code threads, tools and environments.

I use T3 Code as my general agent workspace, not only for repo coding: homelab ops, planning, research, mail and admin work. In a lot of those threads I'm thinking out loud more than writing a precise prompt, and talking would be much more natural. Right now I dictate into T3 Code on my iPad, which is honestly frustrating: it gets the words in, but it's still dictate, send, read the reply on screen, repeat. For a real back-and-forth I switch to a separate voice assistant and lose my tools, MCP servers and thread history.

I don't know the codebase well, so this is the experience I'm after rather than an implementation plan.

How I picture it

• It feels like a call. Start it on a thread, it listens, you can mute or hang up, and it keeps going with the screen locked or in the car. ChatGPT's CarPlay app works this way. Everything said lands in the thread as normal messages.
• While the agent works, it keeps me in the loop. Agent turns take much longer than a chat reply, so the call shouldn't just go silent. Short spoken updates on what it's doing and thinking as it goes, and a spoken summary when it's done, skipping code blocks, paths and tool output.
• I can steer by voice. The mic stays open while the agent works, and what I say goes in like a steer message, so I can correct course mid-turn instead of waiting for it to finish. That's how I already use T3 Code by typing; I'd just like to do it by voice.
• Not only English. I'd use this in German, which iPhone dictation doesn't handle well yet ([Bug]: iPhone 1.2.0 voice input recognizes Korean speech as unrelated English words #12743, [Bug]: Mobile dictation has no usable German transcription #19091).

Related threads

• iPhone dictation has shipped, along with the Settings → Microphone routing that already includes CarPlay. Ideally the call builds on those rather than being a separate voice feature.
• Read-aloud: [Feature]: Read AI responses aloud via built-in TTS (per-message "read aloud" action) #6762, [Feature]: Read agent responses aloud on mobile with background TTS #12622. Web/desktop dictation: Would Codex-backed live dictation for web and desktop be in scope? #12212, Voice input in the web composer, using the browser's speech recognition #14814, Push-to-talk dictation: hold a key, talk, release — the interaction none of the voice PRs implement #7417. Android: Android: microphone dictation into the composer #13603.
• CarPlay / Siri: Idea: Voice conversations in CarPlay #14394, Hands-free thread dispatch from CarPlay and Siri #18286. Several people there and in [Feature]: Read agent responses aloud on mobile with background TTS #12622 say what they want is a conversation with their agents.

Questions

• Is a voice call on a thread something you'd want in T3 Code, or should voice stay limited to dictation and read-aloud?
• If it is, should this be the place to collect the conversation side of [Feature]: Read agent responses aloud on mobile with background TTS #12622 and Idea: Voice conversations in CarPlay #14394?

Happy to test it.

---

This idea came out of a T3 Code thread on my cloud instance: I talked it through by dictation, and the agent looked up the related threads, drafted this, and posted it for me.

[cramt]

+1, this is exactly what i want. the claude code mobile app has this now and its suuuuper good, its basically just a phone call with claude. you just talk and it talks back while it works in the session. no dictate, send, read, repeat loop

i've used voice mode a bunch while driving to get a lot of work done, but obviously i'm stuck with what plain claude chat can do, which is like nothing compared to an actual agent with my tools and repos

what i'd love is that exact ux but on a t3 thread, so i keep my envs, mcp servers and history instead of switching apps

one thing tho: both attempts so far (#13214 and #26393) are tied to openai realtime / a chatgpt codex login. i feel like the voice layer should be separate from whatever provider the thread is using, so a call on a claude thread still works, and the voice model just relays to the thread and reads back what the agent's doing (basically the steer-by-voice + spoken updates stuff from the original post)

i'd love to help out with this, happy to prototype or test if its in scope

[nglessner]

Just a heads up - this is mostly possible already via the T3 Code MCP. I'm using Codex in their conversational mode, connected to T3 Code. I suspect it's not efficient with token usage, but it does work and isn't completely painful. I was able to pick up an existing thread and use the voice mode to "interview" myself to fill in details on a project.

[jobsingles]

+1 from us, with the phone as the main case. We run T3 Code as the control room for a small team of agents (one master thread, sub-threads, a second machine), and on the road the T3 mobile app is the only way in. Dictation works, but it's still dictate, send, read.

What we'd need most is the subscription mode from the closed #15544 ("Talk to T3", fork elliptic1/t3code, branch feat/voice-agent-mode): the client transcribes, an agent answers from a thread using the subscription we already pay for, and the reply is read aloud. No extra realtime API key, approvals stay in the UI. Running that fork isn't an option for us because it's ~3,600 commits behind and the mobile app wouldn't have it anyway.

Until then we work around it: the agent writes its reply as an MP3 and drops it into the message, and the inline audio player plays it. That's fine for listening, but it isn't a conversation.

Would you take this in the official app if someone rebased #15544 onto main and kept it behind a setting?

[brauliobo]

Following up here instead of opening a duplicate: I've built this for the Codex/ChatGPT-subscription case and would like to know whether maintainers want it in T3 Code, and in what shape, before I send another PR.

My earlier attempts (#26393, then #27799) were closed, the second one for missing prior approval, which is why I'm asking here first.

What exists (branch feat/voice-calls on brauliobo/t3code, 4 commits on current main)

• A wave button in the composer starts and ends a live voice call with GPT through the Codex app-server's thread/realtime/*. It uses the signed-in ChatGPT login, so there is no API key. Audio goes over WebRTC between the browser and OpenAI; the T3 server only relays the SDP offer and answer.
• A call started from a thread opens its own chat as a subagent of that thread (Voice call: <thread>). The voice agent works there through T3's MCP tools, so it keeps the thread's tools and history, and what you and the voice say is saved into that chat.
• Options (right-click, or press and hold on touch): account, voice, and the model and effort of the call's chat. While live, the mute button also switches microphone and speaker.
• The wave icon follows the real loudness of both voices in the composer, the chat list and the Lineage panel.

Before/after screenshots and a 34 s recording of the flow are in the closed PR: #27799

Limits, so expectations are clear

• Codex only. It depends on Codex's realtime API, so a call on a Claude thread would not use it. @cramt's point that the voice layer should be separate from the thread's provider is right; the server side is behind a small ProviderRealtimeVoice interface so another provider could implement it, but I haven't built one.
• Web only. There is no mobile entry point yet (WebRTC on mobile needs a native stack).
• It is voice-to-voice through Codex's realtime model, not the "transcribe, answer from the thread, read aloud" design in feat(clients): Talk to T3, a voice conversation that operates the app #15544 that @jobsingles mentions. The two could coexist, but I haven't tried to unify them.
• Spoken updates while the agent works come from the realtime model's own behavior; T3 doesn't add them.

Questions for maintainers

1. Is a live voice call something you want in T3 Code, as opposed to dictation and read-aloud only?
2. If yes, is a Codex-only first version acceptable, or should the voice layer be provider-agnostic from the start?
3. Should it be on by default for eligible accounts (it is today, behind a voiceCalls setting), or opt-in?

If you approve a direction, I'll open one PR that links your approval. If the answer is no or not now, I'll keep it on my fork.

[samnetic]

+1 to this idea. I'd like to propose a concrete architecture and ask for approval of direction and scope before anyone writes code. I use T3 Code all day, and I want to build this if you agree.

In short: a voice layer in the client that talks to you and runs the thread. It is not a new provider. It uses ElevenLabs Agents (bring your own key) for speech. It controls the thread only through T3 commands that already exist, so it works the same on Claude, Codex, Cursor, Grok, or any other thread. This answers cramt's point above: the closed attempts (#13214, #26393, #27799) were tied to OpenAI Realtime and a ChatGPT login.

Prior art: Happy

Happy (MIT) ships this today on top of Claude Code and Codex. Their voice architecture doc describes it. The parts worth copying:

• The voice agent runs on ElevenLabs over WebRTC, directly between the client and ElevenLabs. Their backend only mints a short-lived conversation token.
• The voice agent has a few client tools: send a message to a session, and allow or deny a permission request.
• App events reach the voice agent in two ways:
  • Silent context (sendContextualUpdate): new messages, focus changes. The agent knows, but does not speak.
  • Spoken prompts (sendUserMessage): "the turn finished", "an approval is waiting". These wait while the user or the agent is speaking. Then they go out as one batch.
• The voice agent follows the session the user is looking at.

How it maps onto T3 Code

 Mic/speaker ◄──WebRTC──► ElevenLabs agent (STT + LLM + TTS, user's account)
     │                          │ client tools / context updates
     ▼                          ▼
 Web client: VoiceCallController (packages/client-runtime, platform-neutral)
     │  existing thread commands             ▲ existing thread atoms
     ▼                                       │
 T3 environment ── orchestration ── provider (Claude / Codex / …)
     │
     └─ voice.createConversationToken (new RPC; holds the ElevenLabs key)

Server: one small service, no audio

• Settings. A new environment setting voice.elevenLabsApiKey (SettingsSecretValue, so it goes to ServerSecretStore or a 1Password reference and is redacted for clients). Also a non-secret voice.elevenLabsAgentId.
• RPC. voice.createConversationToken, gated on AuthOrchestrationOperateScope and listed in CLIENT_GUARDED_RPC_SCOPES. The service resolves the key and calls GET /v1/convai/conversation/token?agent_id=…. It returns only the token, which is single-use and short-lived. The key never leaves the environment.
• Agent provisioning. An optional RPC, voice.provisionAgent, creates the ElevenLabs agent from T3's tool definitions (POST /v1/convai/agents/create), with prompt overrides allowed. This keeps the tool schema and the system prompt versioned in this repo, not in someone's ElevenLabs dashboard. The prompt is model-facing text, so it gets an eval under evals/.
• The server never sees audio. This keeps the "performance without compromise" promise and works the same locally, over Tailscale, and through T3 Connect.

Client: a controller in client-runtime, a thin web adapter

VoiceCallController lives in packages/client-runtime so mobile can reuse it later. It takes an injected VoiceSession interface (start, end, sendContext, sendPrompt, mode events). The web adapter wraps @elevenlabs/client, which is MIT and lazy-loaded only when a call starts.

Each client tool calls a command that already exists:

Voice tool	T3 call	Notes
list_threads	readThreadShells()	id, title, run status, pending approval or question
focus_thread(threadId)	router navigation	the call follows the visible thread, like Happy
get_thread_status()	deriveThreadRuntime + derivePendingThreadRequests + latest assistant_message	trimmed; code blocks and paths left out
send_message(text, mode)	threadEnvironment.startTurn	dispatchMode auto / steer / queue; default from followUpBehavior; uses the thread's own model, runtime and interaction mode
interrupt()	threadEnvironment.interruptTurn
answer_approval(requestId, decision)	threadEnvironment.respondToApproval	voice offers only accept / decline, never acceptAlways; the agent reads the request back first
answer_question(requestId, answers)	respondToUserInput

Context feed. The controller subscribes to the focused thread's atom with subscribeAtomChanges:

• A finished assistant message becomes silent context, trimmed. Stream chunks are not sent.
• The end of a run becomes a spoken prompt: "summarize what changed".
• A new pending approval or question becomes a spoken prompt.
• A focus change becomes silent context with the new thread's status.

Prompts wait while anyone is speaking, as in Happy.

Every message the voice agent sends lands in the thread as a normal user message. The thread history stays the source of truth. The ElevenLabs call transcript is not stored in phase 1.

UI

• An opt-in toggle on the Labs settings panel.
• Key and agent setup on the environment's settings page.
• One call button in the composer footer, next to attach. It is hidden when Labs is off or the environment has no key.
• A static "in call" indicator in the sidebar row. There is no continuous animation.
• Hang up from the button or the Stop control. Mute is a reverse state: a muted call shows as muted.

Scope

Phase 1 (this proposal): web and desktop. One call at a time. It follows the focused thread. It uses only existing thread commands. No creating, archiving, or deleting threads by voice.

Later, only if phase 1 is accepted:

• mobile through @elevenlabs/react-native, reusing the client-runtime controller;
• CarPlay routing (Idea: Voice conversations in CarPlay #14394, Hands-free thread dispatch from CarPlay and Siri #18286), using the existing microphone routing settings;
• starting new threads by voice;
• other voice backends behind the same VoiceSession interface, such as OpenAI Realtime through the Codex subscription (feat(voice): live voice calls with GPT through Codex #26393), for people who do not want an ElevenLabs account.

Hit every surface (from AGENTS.md)

• Clients: web and desktop in phase 1. The mobile decision is "later, same controller".
• Providers: no adapter changes. Voice uses provider-neutral thread commands.
• Agents/MCP: not exposed to agents. A call needs a human with a microphone.
• Contracts: two settings fields, one or two RPCs, one client setting.
• Connection modes: the token RPC goes to the environment that owns the thread. Audio never goes through T3, so relay and tunnel behave the same as local.
• Reverse states: start and hang up, mute and unmute, Labs on and off, key set and removed.
• Docs: a short section in docs/user/. No internal doc unless review finds a non-obvious constraint.

Verification plan

• Unit tests in packages/client-runtime: tool handlers against a fake VoiceSession and fake commands; the prompt queue (held while someone speaks, flushed as one batch); context formatting.
• A server test for the token service with a fake HTTP client: missing key, ElevenLabs error, missing scope.
• An eval for the voice system prompt: it must not approve without reading the request back, and it must say "sent" and not invent results.
• A screen recording of a real call on a Claude thread and a Codex thread: start, steer mid-turn, approval by voice, spoken summary, hang up.

Questions for maintainers

1. Do you want a voice call on a thread in T3 Code at all, behind Labs?
2. Is bring-your-own ElevenLabs key acceptable for phase 1? The alternative is to wait for a subscription-based backend.
3. Should the key belong to the environment (my proposal: it works from any client, including the phone later) or to the client?
4. Is web and desktop first, mobile later, an acceptable split, or must mobile ship together?

If the direction is approved, I'll open one focused PR with the evidence above.

Drafted with Claude Code (Opus 5.5) in T3 Code, from a read of the T3 Code and Happy sources.

[d0dge-dev]

Thanks for all the replies. Since nobody had picked this up yet when I posted, I built it myself as a separate app on top of T3 Code, and I've now published it as a proof of concept: https://github.com/d0dge-dev/yapper (MIT). It's not a fork and doesn't touch T3's code. I've used it daily for about ten days, on my phone, mostly in German.

It's the "transcribe, answer from the thread, read aloud" design that @jobsingles described, so to @cramt's point it doesn't care which provider the thread uses:

• No voice LLM in front. What I say goes into the thread word for word. While the agent is running it goes in as a steer. The agent's prose is read back, without code blocks, paths and tool output. If the agent is quiet for a while, the call says what it's doing ("running git").
• No extra API key. The thread runs on whatever subscription it already uses. Speech is self-hosted: Whisper large-v3-turbo and Qwen3-TTS on a small GPU, or Parakeet and Piper on the CPU. With streaming TTS it's about 0.4 s to the first audio, and I can talk over it to interrupt.
• Approvals by voice: "needs your approval: …, yes or no?"
• Android app (Capacitor) with a real call: lock screen, Bluetooth, earpiece and speaker. Audio goes over WebRTC straight to my server, with a WebSocket fallback.
• Callbacks turned out to be the best part. I say "call me back when you're done" and hang up. When the run ends, or it needs an approval or an answer, my phone rings with a real incoming-call screen and the agent tells me the result. There's also an MCP tool, call_me_back, so an agent in any thread can ring me when it needs me.

A few things I learned that might help whichever design lands here:

1. Relay vs. voice agent. @samnetic's and @brauliobo's designs put a second model between you and the thread. That's nicer for small talk and summaries. The relay design is cheaper, keeps the thread as the only source of truth, and never has a voice model "approve" or paraphrase anything. I think both are worth having behind the same VoiceSession-style interface. A transcribe-and-read-aloud backend would also be the one that works without any extra account.
2. The phone and the locked screen are the main use. For me that meant a native call stack (Android ConnectionService), not just a browser tab. Almost all my calls happen while walking or driving.
3. Recognition needs the thread's vocabulary. Project names and tools come out garbled. Biasing recognition with words from the thread and its repo helped a lot, which matches Voice input: project-aware vocabulary hints for speech-to-text #27752.
4. What would make external clients like this sustainable: Yapper talks to T3 over the internal WebSocket RPC, so nightly updates can break it (it has a smoke test for that). A small stable surface would be enough: list threads, read history, subscribe to events, send a message or steer, answer approvals and questions. It would let people build voice, CarPlay or other clients without forking.

I'm not asking for any of this to be merged. It's meant as a working reference and something to try. I'm happy to test any of the proposals above, on Claude threads and in German.

Drafted with Claude Code in a T3 Code thread, from a voice call with Yapper.

[leopiney]

I wanted to share a different take on this: T3Bud, an voice agent for your whole T3code app instead of a single thread.

t3bud-demo-github.mp4

It's a small animal (wolf, frog, capybara, or axolotl) that sits on the window and follows your pointer. It knows every thread across your projects, so you can ask what needs you, start threads, send prompts, approve requests, or have a thread read back. It also speaks up when something finishes or gets stuck (it doesn't get interrupted if a notification arrives).

It builds on ideas from this thread. The voice layer stays separate from the thread's provider (@cramt's split), and every action is a command the app already has (@samnetic's sketch). Unlike the relay in Yapper, it's a second model that summarizes. It only approves one request at a time, after reading it back, and ElevenLabs only sees thread data during a call.

I'm not sure this belongs in T3 itself, since it's experimental and pretty opinionated. Mostly I wanted to share the approach and hear what you think. If there's interest, I'm happy to split it into smaller PRs.

PS: sorry for the sloppiness of the video, just asked Claude to do a demo based on a screen recording I did 😅

Here's my fork jic https://github.com/leopiney/t3code

[cramt]

@juliusmarminge yeah i know youre busy so imma just summarize everything above and give you an mvp, really just looking for a thumbs up (or a no) on the scope so someone can build it without it getting closed for prior approval again

tl;dr of the thread

theres basically 3 directions people have built/proposed:

1. voice model in front, codex only (brauliobo, feat(voice): live voice calls with GPT through Codex #26393/feat(voice): live voice calls with GPT through Codex #27799, realtime via chatgpt login)
2. voice model in front, provider agnostic (samnetic's elevenlabs design, leopiney's t3bud)
3. no voice model, just a relay (d0dge-dev's yapper): your speech goes into the thread as a steer, the agents prose gets read back

every voice pr so far (#13214, #15544, #26393, #27799) got closed on prior approval, not on the idea, so i feel like the missing piece is just a scoped yes. and most of the plumbing is already on main: the mobile dictation, the shared voice-input controller in client-runtime, mic routing incl carplay, and the outside agent mcp surface for list/read/send/steer/interrupt

mvp

a "call" on one thread, on mobile, built on the existing dictation:

• continuous mic with vad, each utterance gets transcribed and sent through the normal send path (steer if the agent is running, new turn if not, same send-mode logic as typing)
• the agents prose gets read back, minus code blocks, paths and tool output. talking over it stops playback
• keeps running with the screen locked (background audio session), cause driving/walking is like the whole use case
• works on any provider, cause it only uses existing thread commands. theres no second model in the loop

stt/tts sit behind one small interface with 2 backends:

1. on-device (default): the apple/android transcription mobile already uses + the os tts. no setup, no key
2. openai-compatible endpoint: base url + key as an environment setting, so the key stays on the server. /v1/audio/transcriptions and /v1/audio/speech are basically the de facto standard, so this covers openai, groq etc but more importantly self-hosted stuff. whisper.cpp's whisper-server speaks it with --inference-path /v1/audio/transcriptions, and kokoro/piper have compatible tts servers, so anyone running t3 on a homelab box or an old gaming laptop can just run that next to it. it also fixes non-english (the german issues in [Bug]: iPhone 1.2.0 voice input recognizes Korean speech as unrelated English words #12743/[Bug]: Mobile dictation has no usable German transcription #19091) and lets us pass the thread's vocabulary as the transcription prompt

explicitly not in the mvp

• realtime/voice-llm backends (openai realtime, elevenlabs). no standard api there, could come later behind the same interface
• approvals by voice (the mcp surface intentionally cant do that, feels like its own discussion)
• callkit/carplay "real phone call" ui, multi-thread, starting threads by voice, callbacks, mascots lmao
• t3 bundling/managing a whisper.cpp process, it just talks to a url

the ask

is this scope okay to build? mainly:

1. mobile only first, or do you want web in the same pr (the controller lives in client-runtime so it could be shared)?
2. on by default or behind a setting?

if its a yes i'm happy to build it and open one focused pr linking your approval