Desktop composer dictation with on-device transcription (client-side, like iOS)

[Zacharon]

Problem. Dictation exists on iPhone (Voice input on iPhone) and nowhere else. Desktop users keep asking (#7417, #10489, #12212, #17317). Two attempts were closed: #11278 and #18302. They used private vendor endpoints, bypassed environment selection, duplicated the controller, and had no prior approval. I'd like to agree on direction and scope before opening anything.

Proposed direction. Treat desktop like iOS: the client transcribes locally. No provider, account or environment is involved, so there is nothing to route.

• Move VoiceInputSession, metering and presentation from apps/mobile into @t3tools/client-runtime/voice-input, so web and mobile share one session and one controller.
• Web: a MediaRecorder recorder and a mic beside Attach (cancel / level meter + time / confirm). The transcript goes in at the captured selection, and send is blocked while dictating. The mic is hidden entirely when the client has no transcriber, as on Android today. No disabled placeholder.
• Desktop shell: a DesktopVoiceTranscription service runs a whisper.cpp executable and ggml model named in desktop client settings (two path fields under Settings → Chat → Voice input). It works on a temp 16 kHz mono WAV that is deleted afterward. Nothing leaves the machine.
• macOS already has the microphone usage string and audio-input entitlement, so packaging doesn't change.

Scope of a first PR. The above, plus docs and unit tests for the whisper output parser and WAV encoder. Excluded: a push-to-talk keybinding, live interim text, a bundled engine or model download, Android, and environment-backed transcription. I can split the client-runtime move into its own PR if you prefer.

Questions.

1. Is client-side transcription (the iOS model) the shape you want for desktop? Or would you rather it be environment-backed (docs/internals/voice-input.md names that as the unimplemented alternative)?
2. Is "the user points at a whisper.cpp executable + model" acceptable for v1, leaving bundling or a first-run download as a later product decision? The transcriber is pluggable, so the UI and session wouldn't change.
3. Any objection to moving the session, metering and presentation modules into client-runtime?

Status. I have a working build on Windows 10 (whisper.cpp b5454 CPU, ggml-base.en). A 6.5 s recording lands at the caret about 4 s after confirm. Dictation was tried with a physical microphone. Cancel keeps the draft. A draft edited mid-recording inserts nothing. A silent recording reports "No speech was detected" instead of whisper's hallucinated "you". check:changed passes. Screenshots and a short recording are ready for the PR. If the answer is "not now", that's fine; I'd rather hear it here than after a PR.

The code and this write-up were produced with Claude (Fable 5.1 and Opus 5.5, via Claude Code).

[sfarthin]

On question 2, I have a variant that replaces the path fields with a one-time download. It runs Whisper inside the client's own browser engine, so one implementation covers both web and desktop. It reuses VoiceInputController unchanged and adds only a MediaRecorder recorder and a VoiceTranscriber backed by transformers.js in a web worker (WebGPU, with a CPU fallback). The first mic press downloads whisper-base (about 100 MB) with a "one time only" progress label, and the browser caches it. It doesn't touch a provider, the environment, or the desktop shell, and no audio leaves the machine.

	Path fields (this proposal)	In-browser Whisper	#14882
Runs in	Desktop shell, whisper.cpp	Client, web and desktop	Environment server
Setup	User installs a binary and a model	Download on first mic press	Model catalog in Settings
Audio over remote connections	Desktop only	Stays on the client	Sent to the environment

#14882 does much more (model catalog, dictionary, live previews), but it's about 18.6k lines. This is about 600 lines plus the lockfile, and it doesn't rule #14882 out later. The cost is one new dependency (@huggingface/transformers, with its Node-only deps removed by an override), plus a 548 KB worker chunk and a 27 MB ONNX Runtime wasm. Both load only on the first mic press. On a Mac with WebGPU, a 5-second clip transcribes in about 1 second.

1. Would you take this as the transcriber in Zacharon's plan, instead of the path fields?
2. Should it ship behind a build flag like T3CODE_FEATURE_SECRETS, or as a Labs toggle?

[Alex-DashNC]

+1 for desktop. I'm on the macOS desktop app (0.1.6) and dictate most of my prompts. macOS system dictation works, but it means a keyboard shortcut and its own popup. A mic button in the composer, like the iPhone app's, would make it one click. On-device transcription as proposed here would suit me; I'd rather not send audio to another vendor.

[MikaMcFlurry]

A Windows data point that might help question 2, as a zero-dependency first step.

Windows already has OS-level dictation: voice typing (Win+H). It transcribes into whatever field has focus, in the language set under Settings → Time & language → Speech, with punctuation. In my fork, a mic button next to Attach focuses the composer and opens that panel through a small desktop bridge method (the main process presses Win+H via user32 keybd_event, because Electron can't send the Windows key). That's about 100 lines and needs no model or binary. One caveat from use: voice typing listens in the window's input language, not the Windows display language. On a German Windows it first transcribed in English, because English (UK) was also installed with a German keyboard layout and was active in the T3 window, so the keyboard looked the same. After switching that window to German, and downloading the German speech files under Settings → Accessibility → Speech, it starts listening on one click and transcribes German. An OS-engine path should point at this, or switch the input language before it opens the panel. The trade-offs: it's Windows only, the transcription runs through Microsoft's speech service rather than on-device, and the composer's own controller (selection capture, send blocking) isn't involved.

Branch: mika/desktop-dictation. Since then I've used it with a real microphone; see the caveat above about the language download. I'd still prefer the in-client Whisper variant upstream, since it covers macOS and web with one implementation. This is just a cheap fallback if that needs more time.

[SwitchOCE]

Written and posted by Claude (Anthropic's AI assistant) on behalf of this account's owner, at their request.

+1 from a Windows 11 desktop user. Since macOS desktop and iPhone already have a composer mic, the main gap is Windows and Linux parity. What I'd like:

1. A mic button beside Send on every desktop OS, using the OS's own dictation first. Windows voice typing (Win+H) or Voice Access, macOS dictation (already shipped), and on Linux the best available local engine. This needs no account or extra download.
2. A "Voice provider" setting with the OS engine as the default, plus the options from [Feature]: Choose transcription engine (BYOK — Gemini/OpenAI/Groq) for macOS voice dictation #28150 and Feature Request: External Self Hosted Speech to Text Integration for T3 Code #28636: on-device Whisper, an OpenAI-compatible /v1/audio/transcriptions endpoint (OpenAI, Groq, a self-hosted server), and Gemini.
3. A configurable push-to-talk shortcut. Win+H is awkward, and it can't be changed in Windows.

The transcript goes into the draft for editing and is never sent automatically, same as on macOS.

[MikaMcFlurry]

Agreed on point 1. For Windows, that's what my fork does today: mika/desktop-dictation adds the mic next to Attach, and it opens Windows voice typing for the composer. Since my comment above I've confirmed it with a real microphone. One catch for Windows: voice typing uses the window's active input language. With English (UK) also installed (same keyboard layout), it transcribed English until the T3 window was switched to German. A "voice provider" setting with the OS engine as the default would keep that path and still leave room for the Whisper or BYOK engines from #28150 and #28636.