Architecture

Architecture

简体中文 · Home

Process model

The normal desktop deployment has two long-running processes:

1. voice-input daemon owns audio capture, session control, ASR, state, refinement, and output.
2. /usr/bin/qs --no-duplicate --path …/quickshell keeps the HUD resident.

Settings is not resident. voice-input settings first tries a non-secret Quickshell IPC activation call for an existing window. If none is available, it starts /usr/bin/qs --daemonize --no-duplicate --path …/quickshell-settings. That independent Quickshell configuration creates a regular FloatingWindow and starts voice-input settings-backend --stdio as its dedicated child.

Dictation clients send short commands over $XDG_RUNTIME_DIR/voice-input/control.sock. Runtime state is atomically replaced at $XDG_RUNTIME_DIR/voice-input/state.json; waveform NDJSON is broadcast separately over waveform.sock.

flowchart LR
  Key["Hyprland binding"] -->|control.sock| D["Daemon"]
  PW["pw-record<br/>mono PCM16"] --> D
  D --> RT["Qwen realtime WebSocket"]
  D --> Local["/usr/bin/voxtype"]
  D --> Final["Qwen full-audio HTTP"]
  D --> LLM["OpenAI-compatible LLM"]
  D -->|state.json| HUD["Quickshell HUD"]
  D -->|waveform.sock| HUD
  D --> Out["wtype / wl-copy<br/>xclip / xdotool"]
  SettingsCmd["voice-input settings"] -->|activate or start| Settings["Quickshell FloatingWindow"]
  Settings <-->|versioned NDJSON over stdio| SB["Rust settings backend"]
  SB -->|validate and atomically write| Config["config.toml"]
  SB -->|secret over stdin| Creds["systemd-creds"]

Loading

Session stages

1. Capture and arming

pw-record produces mono signed 16-bit PCM at audio.sample_rate (16 kHz by default). With pre-roll disabled, each session owns a dedicated recorder. With pre-roll enabled, a capture service keeps one recorder hot, maintains a ring buffer, and attaches the active session to that shared stream. The ring holds at least the configured pre-roll window and the 320 ms capture warm-up. Both paths enforce audio.max_duration_secs (five minutes by default); reaching the limit triggers the normal stop and finalization pipeline automatically.

At start, the daemon also records a Wayland/XWayland target hint. The effective recording clock starts only when capture and ASR readiness move the session from arming to recording; accepting stop freezes the duration before finalization begins. The runtime snapshot carries the start timestamp while recording and the frozen duration afterward so every HUD surface shows the same timer.

2. Live ASR and waveform

For alibaba-qwen-realtime, PCM is packetized into 2,048-sample chunks and offered to a bounded realtime queue without blocking capture. A WebSocket worker processes a bounded number of outgoing packets before reading server events, preventing either direction from starving the other. Realtime events update committed and unstable transcript text. server-vad supplies speech-start and speech-stop events; only manual mode sends a commit at stop.

If recent local speech continues but Qwen produces no transcription event for eight seconds, the worker reconstructs the realtime session once. It opens a fresh connection and replays every retained raw PCM packet from the beginning while capture continues feeding the same bounded control queue. The replacement uses a fresh transcript assembler, so its complete replayed transcript replaces the interrupted preview instead of being appended to it. A new speech burst after a long pause receives its own eight-second observation window.

If the replacement also stalls, or if either connection closes prematurely, Voice Input stops trusting realtime text and switches to stop-time full-audio recovery. A full queue and a disconnected worker are diagnosed separately. In every degraded case, local capture, the HUD waveform, and the daemon's complete PCM buffer continue. The client does not send a manual recovery commit in Server VAD mode because the server may reject it and close the connection.

The waveform analyzer works directly from PCM. It uses a 512-sample window and 256-sample hop, producing 62.5 frames per second at 16 kHz. It publishes thirty mirrored bars plus twelve frequency bands and aggregate voice metrics. During recording, the HUD passes those live bands through a cubic B-spline and maps the result to the capsule's top edge. Before speech is detected, Listening shows a broad, calm virtual spectrum with the same maximum geometric reach as processing; voice onset crossfades to the live spectrum with a roughly 150 ms attack, while pauses decay back over roughly 800 ms. That release blends the measured spectrum directly into the full-height standby envelope and never falls back through a separate procedural perimeter wave. Arming uses virtual bands for its preparation cue. Finalizing, Refining, and Sending share one broad virtual-band profile, one continuously accumulated motion phase, one cadence, and one brightness range. The HUD retains the last live spectrum when the daemon closes the waveform session, then performs a roughly 360 ms breath-like handoff: the halo falls to about five percent visibility while geometry and color move to the processing profile, then rises to full visibility. It clears the retained frame only after the runtime snapshot reaches idle and preserves processing geometry across later phase changes. Those changes only start a roughly 650 ms whole-halo color crossfade, so processing never appears to restart and no spatial color boundary crosses the waveform. The HUD initializes geometry and color transition progress synchronously with each phase change, preventing a target-state frame from rendering before the crossfade begins. Waveform cadence is independent of ASR packet cadence. While recording, the same visible animation heartbeat advances the elapsed clock and provides a low-rate fallback state refresh, so delayed QML timers cannot leave the transcript or clock frozen while the halo still moves.

For local-cli, a background partial thread periodically transcribes the accumulated audio according to audio.partial_interval_ms. On stop, the complete audio is transcribed again.

3. Stop, cancellation, and final ASR

On manual stop or automatic duration-limit stop, the daemon first captures the focused Kitty Pi/Codex process when agent context and refinement are enabled. Session resolution runs in parallel with final ASR, so focus changes during finalization do not change the selected session. An explicit record cancel skips this lookup.

Capture then detaches or the dedicated pw-record child is terminated, the final partial ASR packet is flushed, and realtime ASR is asked to finish. A realtime session with no speech event or non-empty transcript after a 350 ms grace period becomes a silent cancellation: no final ASR, LLM call, or text injection occurs.

If final_pass_enabled = true, the daemon writes the complete PCM buffer to a temporary WAV and sends it as a base64 data:audio/wav input to the configured Qwen final model. Success replaces realtime text. Failure uses realtime final text when available, then /usr/bin/voxtype when local fallback is enabled. Without a final pass, usable realtime final text is preferred and local fallback handles remote failure or empty output. A failed one-time reconstruction, queue overflow, or disconnected worker means the remote service did not process a complete recording; its transcript is rejected and recovery uses the complete buffered recording through an enabled final pass or local fallback.

Chinese output runs through OpenCC (t2s or s2t) after recognition.

4. LLM refinement

Refinement is conservative and optional. The default llm.timeout_ms = 15000; implementation clamps the shared total budget to 1,000–30,000 ms. Prompt construction, the contextual request, and any eligible transcript-only fallback all consume the same deadline. For budgets of at least 10 seconds, the contextual attempt receives all but the final five seconds so transcript-only recovery still has a chance. A context-free retry can follow a transport error, recognized context/payload error, invalid response, truncation, or contextual budget exhaustion, and only if at least one second remains. Non-retryable HTTP/provider errors keep the original ASR text.

Any error, timeout, truncated response, missing model/credential, or exhausted budget leaves the ASR transcript unchanged. Timing logs record attempt kind, elapsed milliseconds, outcome category, and final choice; they do not print transcript or context.

5. Output

The daemon resolves the active target again while retaining the target hint captured at start. Direct type mode uses wtype. Text longer than 120 Unicode characters automatically switches to paste. XWayland switches from type to paste by default. Wayland paste uses wl-copy plus a wtype key chord; XWayland uses xclip plus xdotool. Paste mode backs up and restores the relevant clipboard after a 220 ms wait.

When configured, an Fcitx5 guard temporarily closes the active input method before output and restores it afterward.

Settings boundary

The Settings QML edits a full configuration draft, but it does not parse, validate, or write TOML itself. Every backend request and response is one versioned JSON object per line. Requests go to the child process's inherited standard input, and responses return on standard output.

Rust exclusively loads and validates configuration. A load response includes an opaque revision derived from the exact source that was read. Save sends that revision and the complete supported configuration; Rust rejects the save if the exact source changed in the meantime, preventing one Settings window from silently overwriting another editor. Successful saves preserve all supported fields, create the configuration directory with mode 0700, write the file with mode 0600, and atomically replace it.

Credential actions are keep or replace. Password inputs travel only through inherited stdin to Rust and then through stdin to systemd-creds; they do not enter TOML, process arguments, environment variables, logs, or backend responses. QML clears each credential field immediately after submitting Save or Test LLM. Because QML/JavaScript strings are managed memory, this clearing is best effort rather than a guaranteed zeroization primitive.

Test LLM can use either the newly entered LLM credential or the encrypted-store credential. Save may request a daemon restart. A restart failure is reported separately from persistence, so a successfully written configuration is not misreported as an unsuccessful write.

The read-only runtime.get method probes voice-input.service and parses the canonical runtime snapshot independently from save requests. Its response is constructed from an allowlist: bounded service state plus phase, update time, language, engine, and model. Transcript variants, tooltips, output-target data, runtime error text, and credentials are never returned. Probe failure produces an unknown/unavailable Overview state and does not block editing or replace a save error.

Concurrency boundaries

Boundary	Why it exists
Capture service or dedicated reader thread	Reads PipeWire continuously without waiting for network or UI work.
Bounded Qwen audio queue + backend worker + realtime event pump	Capture offers packets without waiting for WebSocket I/O. Queue overflow marks the stream incomplete and drops remote transcript recovery to the complete audio buffer; state/transcript events still progress independently.
Local partial-ASR thread	Periodic previews do not run in the control listener.
Agent-discovery thread	Kitty/Hyprland inspection overlaps recording.
Waveform publisher thread	Uses a bounded queue and nonblocking sends; full queues may drop visual frames instead of blocking capture.
Serialized state update lock	Keeps concurrent capture and ASR updates ordered; each JSON snapshot uses temporary-file rename.
Resident HUD Quickshell process	Polls state every 50 ms during active phases and every 100 ms while idle, and consumes waveform socket frames independently. HUD failure does not own recognition.
On-demand Settings + Rust child	Uses request IDs and versioned NDJSON; exact-source revision checks prevent lost updates from concurrent editors.

The control listener processes one accepted connection at a time and holds the daemon mutex through finalization, refinement, and output. Toggle requests include a timestamp; a queued toggle older than 750 ms is ignored so a delayed second press cannot start a new session after processing completes.

Runtime state machine

idle → arming → recording → transcribing → refining → outputting → idle, with error available from any failed control/session operation. Refining is skipped when disabled. Silent cancellation returns directly to idle.

See also: Configuration · Agent Context · Desktop Integration