Architecture

Architecture

简体中文 · Home

Process model

The normal desktop deployment has two long-running processes:

1. voice-input daemon owns audio capture, session control, ASR, state, refinement, and output.
2. /usr/bin/qs --no-duplicate --path …/quickshell renders the HUD.

Clients send short commands over $XDG_RUNTIME_DIR/voice-input/control.sock. Runtime state is atomically replaced at $XDG_RUNTIME_DIR/voice-input/state.json; waveform NDJSON is broadcast separately over waveform.sock.

flowchart LR
  Key["Hyprland binding"] -->|control.sock| D["Daemon"]
  PW["pw-record<br/>mono PCM16"] --> D
  D --> RT["Qwen realtime WebSocket"]
  D --> Local["/usr/bin/voxtype"]
  D --> Final["Qwen full-audio HTTP"]
  D --> LLM["OpenAI-compatible LLM"]
  D -->|state.json| HUD["Quickshell HUD"]
  D -->|waveform.sock| HUD
  D --> Out["wtype / wl-copy<br/>xclip / xdotool"]

Loading

Session stages

1. Capture and arming

pw-record produces mono signed 16-bit PCM at audio.sample_rate (16 kHz by default). With pre-roll disabled, each session owns a dedicated recorder. With pre-roll enabled, a capture service keeps one recorder hot, maintains a ring buffer, and attaches the active session to that shared stream. The ring holds at least the configured pre-roll window and the 320 ms capture warm-up.

At start, the daemon also records a Wayland/XWayland target hint and, when enabled, starts focused-agent discovery on a separate thread.

2. Live ASR and waveform

For alibaba-qwen-realtime, PCM is packetized into 2,048-sample chunks and sent to Qwen over WebSocket. Realtime events update committed and unstable transcript text. server-vad supplies speech-start and speech-stop events; manual commits at stop. If Server VAD produces no transcription activity for three seconds while enough audio is buffered, the client forces a buffer commit to recover incremental output.

The waveform analyzer works directly from PCM. It uses a 512-sample window and 256-sample hop, producing 62.5 frames per second at 16 kHz. Thirty bars are mirrored around the center. Visibility is gated by realtime voice activity; waveform cadence is independent of ASR packet cadence.

For local-cli, a background partial thread periodically transcribes the accumulated audio according to audio.partial_interval_ms. On stop, the complete audio is transcribed again.

3. Stop, cancellation, and final ASR

On stop, capture detaches or the dedicated pw-record child is terminated, the final partial ASR packet is flushed, and realtime ASR is asked to finish. A realtime session with no speech event or non-empty transcript after a 350 ms grace period becomes a silent cancellation: no final ASR, LLM call, or text injection occurs.

If final_pass_enabled = true, the daemon writes the complete PCM buffer to a temporary WAV and sends it as a base64 data:audio/wav input to the configured Qwen final model. Success replaces realtime text. Failure uses realtime final text when available, then /usr/bin/voxtype when local fallback is enabled. Without a final pass, usable realtime final text is preferred and local fallback handles remote failure or empty output.

Chinese output runs through OpenCC (t2s or s2t) after recognition.

4. LLM refinement

Refinement is conservative and optional. The default llm.timeout_ms = 5000; implementation clamps the shared total budget to 1,000–5,000 ms. Prompt construction, the contextual request, and any eligible transcript-only fallback all consume the same deadline. A context-free retry occurs only for recognized context/payload errors or an invalid response, and only if at least one second remains.

Any error, timeout, truncated response, missing model/credential, or exhausted budget leaves the ASR transcript unchanged. Timing logs record attempt kind, elapsed milliseconds, outcome category, and final choice; they do not print transcript or context.

5. Output

The daemon resolves the active target again while retaining the target hint captured at start. Direct type mode uses wtype. Text longer than 120 Unicode characters automatically switches to paste. XWayland switches from type to paste by default. Wayland paste uses wl-copy plus a wtype key chord; XWayland uses xclip plus xdotool. Paste mode backs up and restores the relevant clipboard after a 220 ms wait.

When configured, an Fcitx5 guard temporarily closes the active input method before output and restores it afterward.

Concurrency boundaries

Boundary	Why it exists
Capture service or dedicated reader thread	Reads PipeWire continuously without waiting for network or UI work.
Qwen backend worker + realtime event pump	WebSocket I/O and state/transcript updates progress independently.
Local partial-ASR thread	Periodic previews do not run in the control listener.
Agent-discovery thread	Kitty/Hyprland inspection overlaps recording.
Waveform publisher thread	Uses a bounded queue and nonblocking sends; full queues may drop visual frames instead of blocking capture.
Serialized state update lock	Keeps concurrent capture and ASR updates ordered; each JSON snapshot uses temporary-file rename.
Quickshell process	Polls state every 50 ms and consumes waveform socket frames independently. HUD failure does not own recognition.

The control listener processes one accepted connection at a time and holds the daemon mutex through finalization, refinement, and output. Toggle requests include a timestamp; a queued toggle older than 750 ms is ignored so a delayed second press cannot start a new session after processing completes.

Runtime state machine

idle → arming → recording → transcribing → refining → outputting → idle, with error available from any failed control/session operation. Refining is skipped when disabled. Silent cancellation returns directly to idle.

See also: Configuration · Agent Context · Desktop Integration