Architecture

Architecture

简体中文 · Home

Process model

The normal desktop deployment has two long-running processes:

1. voice-input daemon owns audio capture, session control, ASR, state, refinement, and output.
2. /usr/bin/qs --no-duplicate --path …/quickshell keeps the HUD resident.

Settings is not resident. voice-input settings first tries a non-secret Quickshell IPC activation call for an existing window. If none is available, it starts /usr/bin/qs --daemonize --no-duplicate --path …/quickshell-settings. That independent Quickshell configuration creates a regular FloatingWindow and starts voice-input settings-backend --stdio as its dedicated child.

Dictation clients send short commands over $XDG_RUNTIME_DIR/voice-input/control.sock. Runtime state is atomically replaced at $XDG_RUNTIME_DIR/voice-input/state.json; waveform NDJSON is broadcast separately over waveform.sock.

flowchart LR
  Key["Hyprland binding"] -->|control.sock| D["Daemon"]
  PW["pw-record<br/>mono PCM16"] --> D
  D --> RT["Qwen realtime WebSocket"]
  D --> Local["/usr/bin/voxtype"]
  D --> Final["Qwen full-audio HTTP"]
  D --> LLM["OpenAI-compatible LLM"]
  D -->|state.json| HUD["Quickshell HUD"]
  D -->|waveform.sock| HUD
  D --> Out["wtype / wl-copy<br/>xclip / xdotool"]
  SettingsCmd["voice-input settings"] -->|activate or start| Settings["Quickshell FloatingWindow"]
  Settings <-->|versioned NDJSON over stdio| SB["Rust settings backend"]
  SB -->|validate and atomically write| Config["config.toml"]
  SB -->|secret over stdin| Creds["systemd-creds"]

Loading

Session stages

1. Capture and arming

pw-record produces mono signed 16-bit PCM at audio.sample_rate (16 kHz by default). With pre-roll disabled, each session owns a dedicated recorder. With pre-roll enabled, a capture service keeps one recorder hot, maintains a ring buffer, and attaches the active session to that shared stream. The ring holds at least the configured pre-roll window and the 320 ms capture warm-up. Both paths enforce audio.max_duration_secs (five minutes by default); reaching the limit triggers the normal stop and finalization pipeline automatically.

At start, the daemon also records a Wayland/XWayland target hint and, when enabled, starts focused-agent discovery on a separate thread.

2. Live ASR and waveform

For alibaba-qwen-realtime, PCM is packetized into 2,048-sample chunks and offered to a bounded realtime queue without blocking capture. A WebSocket worker sends those packets to Qwen. Realtime events update committed and unstable transcript text. server-vad supplies speech-start and speech-stop events; manual commits at stop. If Server VAD initially produces no transcription activity for three seconds while enough audio is buffered, the client forces one recovery commit; it does not repeat forced commits after server activity begins. If the realtime queue fills, the session marks the remote stream incomplete instead of freezing capture or waveform delivery.

The waveform analyzer works directly from PCM. It uses a 512-sample window and 256-sample hop, producing 62.5 frames per second at 16 kHz. Thirty bars are mirrored around the center. Visibility is gated by realtime voice activity; waveform cadence is independent of ASR packet cadence.

For local-cli, a background partial thread periodically transcribes the accumulated audio according to audio.partial_interval_ms. On stop, the complete audio is transcribed again.

3. Stop, cancellation, and final ASR

On manual stop or automatic duration-limit stop, capture detaches or the dedicated pw-record child is terminated, the final partial ASR packet is flushed, and realtime ASR is asked to finish. A realtime session with no speech event or non-empty transcript after a 350 ms grace period becomes a silent cancellation: no final ASR, LLM call, or text injection occurs.

If final_pass_enabled = true, the daemon writes the complete PCM buffer to a temporary WAV and sends it as a base64 data:audio/wav input to the configured Qwen final model. Success replaces realtime text. Failure uses realtime final text when available, then /usr/bin/voxtype when local fallback is enabled. Without a final pass, usable realtime final text is preferred and local fallback handles remote failure or empty output. One exception is realtime queue overflow: because the remote service did not receive complete audio, its transcript is rejected and recovery must use the complete buffered recording through an enabled final pass or local fallback.

Chinese output runs through OpenCC (t2s or s2t) after recognition.

4. LLM refinement

Refinement is conservative and optional. The default llm.timeout_ms = 15000; implementation clamps the shared total budget to 1,000–30,000 ms. Prompt construction, the contextual request, and any eligible transcript-only fallback all consume the same deadline. For budgets of at least 10 seconds, the contextual attempt receives all but the final five seconds so transcript-only recovery still has a chance. A context-free retry can follow a transport error, recognized context/payload error, invalid response, truncation, or contextual budget exhaustion, and only if at least one second remains. Non-retryable HTTP/provider errors keep the original ASR text.

Any error, timeout, truncated response, missing model/credential, or exhausted budget leaves the ASR transcript unchanged. Timing logs record attempt kind, elapsed milliseconds, outcome category, and final choice; they do not print transcript or context.

5. Output

The daemon resolves the active target again while retaining the target hint captured at start. Direct type mode uses wtype. Text longer than 120 Unicode characters automatically switches to paste. XWayland switches from type to paste by default. Wayland paste uses wl-copy plus a wtype key chord; XWayland uses xclip plus xdotool. Paste mode backs up and restores the relevant clipboard after a 220 ms wait.

When configured, an Fcitx5 guard temporarily closes the active input method before output and restores it afterward.

Settings boundary

The Settings QML edits a full configuration draft, but it does not parse, validate, or write TOML itself. Every backend request and response is one versioned JSON object per line. Requests go to the child process's inherited standard input, and responses return on standard output.

Rust exclusively loads and validates configuration. A load response includes an opaque revision derived from the exact source that was read. Save sends that revision and the complete supported configuration; Rust rejects the save if the exact source changed in the meantime, preventing one Settings window from silently overwriting another editor. Successful saves preserve all supported fields, create the configuration directory with mode 0700, write the file with mode 0600, and atomically replace it.

Credential actions are keep or replace. Password inputs travel only through inherited stdin to Rust and then through stdin to systemd-creds; they do not enter TOML, process arguments, environment variables, logs, or backend responses. QML clears each credential field immediately after submitting Save or Test LLM. Because QML/JavaScript strings are managed memory, this clearing is best effort rather than a guaranteed zeroization primitive.

Test LLM can use either the newly entered LLM credential or the encrypted-store credential. Save may request a daemon restart. A restart failure is reported separately from persistence, so a successfully written configuration is not misreported as an unsuccessful write.

The read-only runtime.get method probes voice-input.service and parses the canonical runtime snapshot independently from save requests. Its response is constructed from an allowlist: bounded service state plus phase, update time, language, engine, and model. Transcript variants, tooltips, output-target data, runtime error text, and credentials are never returned. Probe failure produces an unknown/unavailable Overview state and does not block editing or replace a save error.

Concurrency boundaries

Boundary	Why it exists
Capture service or dedicated reader thread	Reads PipeWire continuously without waiting for network or UI work.
Bounded Qwen audio queue + backend worker + realtime event pump	Capture offers packets without waiting for WebSocket I/O. Queue overflow marks the stream incomplete and drops remote transcript recovery to the complete audio buffer; state/transcript events still progress independently.
Local partial-ASR thread	Periodic previews do not run in the control listener.
Agent-discovery thread	Kitty/Hyprland inspection overlaps recording.
Waveform publisher thread	Uses a bounded queue and nonblocking sends; full queues may drop visual frames instead of blocking capture.
Serialized state update lock	Keeps concurrent capture and ASR updates ordered; each JSON snapshot uses temporary-file rename.
Resident HUD Quickshell process	Polls state every 50 ms during active phases and every 100 ms while idle, and consumes waveform socket frames independently. HUD failure does not own recognition.
On-demand Settings + Rust child	Uses request IDs and versioned NDJSON; exact-source revision checks prevent lost updates from concurrent editors.

The control listener processes one accepted connection at a time and holds the daemon mutex through finalization, refinement, and output. Toggle requests include a timestamp; a queued toggle older than 750 ms is ignored so a delayed second press cannot start a new session after processing completes.

Runtime state machine

idle → arming → recording → transcribing → refining → outputting → idle, with error available from any failed control/session operation. Refining is skipped when disabled. Silent cancellation returns directly to idle.

See also: Configuration · Agent Context · Desktop Integration