Voice Input Local STT EN

Voice Input (Local STT)

Speak instead of typing: press Alt+V in the side panel input, watch the draft appear as you talk, and let a local model add punctuation and normalize numbers when you stop. Nothing is sent automatically.

Recognition runs entirely on your machine — no audio ever leaves the device, no cloud ASR service, no extra API key. The price is a one-time setup of two things: optional dependencies and a model pack.

← Back to Wiki Home · Related: Agent Service · Configuration · FAQ

---

1. What it looks like

Once enabled, a 🎤 button appears in the input toolbar; while recording, a recorder strip expands above the input:

Element	Description
24 volume bars	Live waveform that scrolls with your voice
Timer	00:12 style; single take is capped at 5 minutes by default
Status text	Listening / Transcribing / Limit reached / Failed
Cancel	Discard this take and roll the draft back (Esc does the same)
Done	Finish and finalize (Alt+V again does the same)

Two behaviours differ from typing:

• The draft is provisional. The streaming model writes as you speak. The moment you edit the input by hand, later drafts only append what is genuinely new — and if a new hypothesis is a rewrite rather than an extension, it is skipped entirely instead of overwriting what you already see.
• The result is never sent for you. Finalizing only writes text into the input; sending is still your call.

---

2. Three-part enablement (all required)

The 🎤 button only appears when dependencies + models + the switch are all satisfied; otherwise it stays hidden rather than failing after a click.

2.1 Install the optional dependency (once)

The Agent needs sherpa-onnx-node plus its platform binary packages. Pick one entry point:

# 1) CLI (recommended: check first, then install)
aha asr detect
aha asr install

# 2) Opt in while installing the Agent (off by default)
bash install.sh --with-asr

# 3) Options → Voice tab → one-click install (progress polled every 2 s)

On intranet machines the npm mirror must carry the sherpa-onnx platform packages, otherwise the install stalls at "missing platform binary". The full package list lives in docs/Voice-Input-Local-STT.md.

2.2 Provide model packs

Models live in a fixed directory, one sub-directory per pack, each containing a manifest.json:

~/.ai-helper-agent/asr/models/
└── sense-voice-zh-en-ja-ko-yue-2024-07-17/
    ├── manifest.json
    ├── model.int8.onnx
    └── tokens.txt

Three kinds cooperate; you may install only some of them (offline alone works, you just lose the live draft):

Kind	Role	If missing
online (streaming)	Draft as you speak	No draft; one result after you stop
offline (final pass)	High-accuracy pass + punctuation	Falls back to streaming draft text
vad	Sentence segmentation	Still works, slightly worse punctuation

Ways to get them:

# Machine can reach the internet or an HF mirror: one-click download
# (writes the manifest and verifies hashes for you)
curl -H "Authorization: Bearer <token>" -X POST \
  http://127.0.0.1:18910/api/asr/models/download \
  -H 'Content-Type: application/json' \
  -d '{"kind":"offline"}'

On a fully offline network, download from a connected machine, lay the files out as above with a manifest.json (files plus optional sha256), and copy them over — or import the folder from the Options page.

2.3 Flip the switch

Options → Voice tab, top toggle. It is off by default: having the dependency and models installed does not enable anything, which avoids the "I installed it, why is nothing happening" confusion.

Once all three are ready the panel re-gates itself (🎤 appears) without reopening the side panel.

Microphone permission (tab page only)

In side panel or popup form, Chromium cannot show the microphone prompt. Allow it once in the Options page (tab form) — Voice tab → "Grant permission". After that the side panel records without prompting.

---

3. Using it

1. Put the caret in the input (or anywhere) and press Alt+V (Option+V);
2. Speak — the draft appears in the input and you can keep editing it by hand;
3. Finish any way you like:
   • press Alt+V again,
   • click Done in the recorder strip,
   • or just press Enter to send (recording stops and finalizes first);
4. To abandon: press Esc or click Cancel — the draft rolls back.

Shortcuts come in two tiers:

• Alt+V (default, not remappable) — its point is that it also fires while the focus is on the web page (the extension forwards the key to the panel).
• Right Alt / Right Ctrl (optional, both off by default) — tick them separately under Options → Voice → Quick trigger; ticking both means either one fires.
  • Semantics are "press, then release" = one action. Pressing any other key while it is held voids that press — right Alt is a modifier in AltGr layouts, so this neither starts a stray recording nor swallows AltGr character input.
  • The config lives in chrome.storage.local and takes effect immediately in both the side panel and the page — no extension reload, no panel reopen.
  • It only responds when voice input is ready (dependency × model × switch).

---

4. Configuration

Options → Voice tab

Item	Description
Enable voice input	Master switch; turning it off disables 🎤 and Alt+V
Quick trigger	Right Alt / Right Ctrl single-key toggles, both off by default; Alt+V always works and is unaffected
Language	auto / Chinese / English / Japanese / Korean / Cantonese
Streaming model	Pick one of the scanned online packs (first usable by default)
Final-pass model	Same, for offline packs
Dependency status / install	sherpa-onnx detection result and install progress
Scan / import models	Re-scan the model directory or import a pack from anywhere

Agent config file

~/.ai-helper-agent/asr/asr-config.json (written by the extension; normally not hand-edited):

{
  "enabled": true,
  "language": "auto",
  "streamingModelId": "sense-voice-streaming-zh-en",
  "offlineModelId": "sense-voice-zh-en-ja-ko-yue-2024-07-17",
  "idleUnloadMs": 600000
}

Field	Default	Description
enabled	false	Master switch
language	auto	Expected language; auto lets the model decide
streamingModelId / offlineModelId	null	null = first usable pack by capability
idleUnloadMs	600000	Unload engines after this much idle time (0 = never)

Environment variables

Variable	Default	Description
AI_HELPER_ASR_ROOT	~/.ai-helper-agent/asr	Root for models and config
AI_HELPER_ASR_MAX_RECORDING_MS	300000	Max length of one take
AI_HELPER_ASR_SESSION_IDLE_MS	30000	Idle session reclamation
AI_HELPER_ASR_MAX_SESSIONS	1	Concurrent sessions
AI_HELPER_ASR_RSS_LIMIT	~1.5 GB	Above this RSS, skip the offline final pass
AI_HELPER_ASR_OFFLINE_TIMEOUT_MS	180000	Per-task inference timeout

---

5. API (for developers)

WebSocket streaming session

ws://127.0.0.1:18910/ws/asr/<sessionId>?token=<token>&language=auto

Dir	Frame	Meaning
→	{type:'start'}	Open the session
→	binary	16 kHz / 16-bit / mono PCM
→	{type:'stop'}	Finish and finalize (async; final follows)
→	{type:'cancel'}	Abandon, no final
←	{type:'ready', draft}	Ready; draft:false means no streaming model
←	{type:'partial', text}	Draft, roughly every 150 ms
←	{type:'final', text, fallback}	Final text; fallback:true = streaming draft
←	{type:'error', code}	busy / model_missing / max_duration / rss_pressure / internal

On stop the session frees its concurrency slot immediately; the final pass runs on a background worker thread and delivers final later, so the next take can start right away.

HTTP endpoints

Method	Path	Description
GET	/api/asr/status	Dependency / model / switch / loaded engine state
POST	/api/asr/config	Update config (switch, language, model ids, idle unload)
GET	/api/asr/models	Scanned packs (invalid ones include a reason)
POST	/api/asr/models/scan	Re-scan the model directory
POST	/api/asr/models/pick	Import a pack from a directory
GET	/api/asr/models/catalog	Downloadable model catalog
POST	/api/asr/models/download	Download (resumable + hash verified)
GET	/api/asr/models/download/status	Download progress
POST	/api/asr/models/download/cancel	Cancel a download
POST	/api/asr/transcribe	One-shot recognition of a WAV buffer
POST	/api/asr/warmup	Warm the engines (also runs at startup)
POST	/api/asr/deps/detect	Detect the sherpa-onnx dependency
POST	/api/asr/deps/install	Install it (allow-listed command)
GET	/api/asr/deps/install/status	Install progress

All endpoints require a Bearer token, like the rest of the API.

---

6. Privacy and resource usage

• Audio never leaves the machine — no cloud ASR, nothing written to disk.
• Never auto-sent — finalizing only fills the input.
• Idle unload — engines are released after 10 minutes of disuse, returning a few hundred MB of native memory.
• Non-blocking — the final pass runs on a worker thread, so /health, chat streaming and tool execution stay responsive during long transcriptions.
• Memory guard — when process RSS nears the threshold the offline pass is skipped and the streaming draft is used instead (rss_pressure), so the Agent never gets taken down by one long recording.

Rough sizes: streaming (zh/en, int8) ≈ 100 MB, SenseVoice final pass (zh/en/ja/ko/yue, int8) ≈ 230 MB, VAD ≈ 1 MB.

---

7. FAQ

The 🎤 button never appears. One of the three requirements is missing. Open Options → Voice and read whether it is "dependency not ready" or "models not ready", then use the matching install / scan button; also confirm the switch is on.

Clicking 🎤 says there is no microphone permission. The side panel cannot show the prompt. Go to Options → Voice → "Grant permission" and allow it in the tab page.

I copied a model folder but it says the pack is invalid. The Options page now shows the concrete reason (missing manifest.json, missing fields, missing files, sha256_mismatch, …). Fix the manifest and hit "Scan" again.

Recognition feels slow / takes seconds after I stop. That is the offline final pass doing a full recomputation for better quality. Install only the streaming model if you prefer speed over punctuation and number normalization.

Long takes get cut off. Default cap is 5 minutes. At the limit the Agent stops recording and finalizes with the full audio — nothing you already said is lost, and the strip shows "recording limit reached, transcribing…".

Can I add a sentence while the AI is generating? Yes. Once the text lands in the input, the send button switches to Send instead of Stop.

Can two recordings run at once? Default concurrency is 1 (single-user desktop). A second one receives busy and the UI says it is still processing the previous take.

---

Next

• Full install and model-pack authoring (intranet package list, offline packaging) → docs/Voice-Input-Local-STT.md
• Other Agent capabilities → Agent Service
• Every setting → Configuration