feat: voice-typing using external Whisper / ALM API

[heimoshuiyu]

This pull request implements the voice typing feature. It uses external Whisper API or multimodal large models (such as gpt-4o, qwen3-omni) for voice input. It uses the last message as a prompt (context) to improve contextual recognition accuracy (key!).

Related issues and PR:

• Add voice-to-text transcription feature #3827
• Fixes: feat: first party support for voice conversing #2425
• Fixes: [FEATURE]: Speech-to-Text Voice Input for Lazy People in OpenCode #4695

This feature follows the frontend-backend separation design of opencode.

• TUI: Uses tools like ffmpeg, sox, rec, arecord to record the microphone (I only tested ffmpeg on Linux, as I don't have other devices)
• APP: Uses the browser's native microphone recording interface

Two types of speech recognition services can be configured, with whisper as the default:

• whisper: Compatible with the OpenAI /v1/audio/transcriptions interface
• alm: Uses the speech input capability of multimodal large models for transcription

It uses the last assistant message in the current session + the text in the input box as a prompt for speech transcription. This contextual understanding ability allows you to directly voice input special terms like code paths, variable names, etc.!

For testing convenience, I have set up a web frontend with voice input functionality at https://d3ir6x3lfy3u68.cloudfront.net. And replaced the hardcoded https://app.opencode.ai in opencode (in the third commit Add web deploy skill and configurable web proxy)

Disclaimer: Most of the code was vibe coded and then roughly checked by me. So this might be a relatively rough implementation (or even just a POC). Welcome to provide various improvement suggestions, or take this idea and implement it yourself.

External resources:

• I used https://github.com/heimoshuiyu/whisper-fastapi to set up a local whisper API interface
• I used the Qwen/Qwen3-Omni-30B-A3B-Instruct model on https://cloud.siliconflow.cn / https://cloud.siliconflow.com

Here is the demo

tui.mp4

app.mp4

[github-actions]

Thanks for your contribution!

This PR doesn't have a linked issue. All PRs must reference an existing issue.

Please:

1. Open an issue describing the bug/feature (if one doesn't exist)
2. Add Fixes #<number> or Closes #<number> to this PR description

See CONTRIBUTING.md for details.

[github-actions]

The following comment was made by an LLM, it may be inaccurate:

Potential Duplicate Found

PR #3827: Add voice-to-text transcription feature

• Add voice-to-text transcription feature #3827

Why it's related: This PR is already referenced in the current PR's description as a related issue. It appears to be a previous attempt or related work on voice-to-text transcription functionality. The current PR (9264) may be an updated implementation or continuation of this feature.

[calebdw]

See also https://github.com/goodroot/hyprwhspr, which I've started using for system wide STT, not just in opencode

[heimoshuiyu]

@calebdw Great project. However, integrating voice input in OpenCode is necessary because it can capture context to significantly improve transcription accuracy.

[heimoshuiyu]

rebased, ready to merge

[telnet2]

I cannot wait to see it working

[Mikec78660]

This works great. Set it up with a local speaches server and it works perfect. Using model Systran/faster-distil-whisper-large-v3.

[heimoshuiyu]

@Mikec78660 Glad you got it working! Could you share what issues you encountered during configuration? We should show a toast message when configuration errors occur.

[Mikec78660]

@heimoshuiyu I had https instead of http in my url for my stt server like an idiot. I just coped and pasted my server name into the url field and didn't even notice. But it was weird because the Record "button" was just ghosted out and would not let me press it like I had not setup voice at all. I was expecting it would give me some sort of error. Like let me hit the button and then get an error that the server was unreachable or something. But it was completely my mistake that I even had an issue.

Here is my entry in opencode.json

  "voice": {
    "type": "whisper",
    "whisper": {
      "url": "http://speaches.lan:8000/v1/audio/transcriptions",
      "apiKey": "1234",
      "model": "Systran/faster-distil-whisper-large-v3",
      "language": "en"
    }
  }

[mohamedbouddi7777-dev]

اوكي مرحبا بسم الله الرحمن الرحيم توكلنا على الله انشر في المكان يعني هو هنا يعني لا تنشر هذه الامور في جيميل

[heimoshuiyu]

@Mikec78660 Thank you for sharing. I just fixed an issue where clicking the gray RECORD button in the TUI was unresponsive.

[Mikec78660]

@heimoshuiyu Have you considered adding wake word capability?

[heimoshuiyu]

@Mikec78660 That might be another PR. Whisper and ALM do not have wake word functionality. To support wake word, it may be necessary to use other smaller models to continuously monitor the microphone.