Architecture

Architecture

简体中文 · Home

Process model

The normal desktop deployment has two long-running processes:

1. voice-input daemon owns audio capture, session control, ASR, state, refinement, and output.
2. /usr/bin/qs --no-duplicate --path …/quickshell keeps the HUD resident.

Settings is not resident. voice-input settings first tries a non-secret Quickshell IPC activation call for an existing window. If none is available, it starts /usr/bin/qs --daemonize --no-duplicate --path …/quickshell-settings. That independent Quickshell configuration creates a regular FloatingWindow and starts voice-input settings-backend --stdio as its dedicated child.

Dictation clients send short commands over $XDG_RUNTIME_DIR/voice-input/control.sock. Runtime state is atomically replaced at $XDG_RUNTIME_DIR/voice-input/state.json; waveform NDJSON is broadcast separately over waveform.sock.

flowchart LR
  Key["Hyprland binding"] -->|control.sock| D["Daemon"]
  PW["pw-record<br/>mono PCM16"] --> D
  D --> RT["Qwen Realtime WebSocket"]
  D --> A3S["Audio3 Streaming WebSocket"]
  D --> Local["/usr/bin/voxtype"]
  D --> Final["Qwen full-audio HTTP"]
  D --> A3N["Audio3 Native HTTP"]
  StartContext["Pi / Codex focused at start"] -->|immutable terminology snapshot| A3S
  StartContext -->|same snapshot| LLM["OpenAI-compatible Refine"]
  D --> LLM
  D -->|state.json| HUD["Quickshell HUD"]
  D -->|waveform.sock| HUD
  D --> Out["wl-copy + hyprctl<br/>xclip + xdotool"]
  SettingsCmd["voice-input settings"] -->|activate or start| Settings["Quickshell FloatingWindow"]
  Settings <-->|versioned NDJSON over stdio| SB["Rust settings backend"]
  SB -->|validate and atomically write| Config["config.toml"]
  SB -->|secret over stdin| Creds["systemd-creds"]

Loading

Session stages

1. Capture and arming

pw-record produces mono signed 16-bit PCM at audio.sample_rate (16 kHz by default). With pre-roll disabled, each session owns a dedicated recorder. With pre-roll enabled, a capture service keeps one recorder hot, maintains a ring buffer, and attaches the active session to that shared stream. The ring holds at least the configured pre-roll window and the 320 ms capture warm-up. Both paths enforce audio.max_duration_secs (five minutes by default); reaching the limit triggers the normal stop and finalization pipeline automatically.

At start, the daemon records a Wayland/XWayland target hint. When terminology context is enabled, it also captures the focused Pi/Codex source and begins local extraction from that session's latest completed assistant message. The resulting immutable terminology snapshot is shared with Audio3 Streaming and Refine; later focus changes cannot replace it. The effective recording clock starts only when capture and ASR readiness move the session from arming to recording; accepting stop freezes the duration before finalization begins. The runtime snapshot carries the start timestamp while recording and the frozen duration afterward so every HUD surface shows the same timer.

2. Live ASR and waveform

For alibaba-qwen-realtime, PCM is packetized into 2,048-sample chunks and offered to a bounded realtime queue without blocking capture. A WebSocket worker processes a bounded number of outgoing packets before reading server events, preventing either direction from starving the other. Realtime events update committed and unstable transcript text. server-vad supplies speech-start and speech-stop events; only manual mode sends a commit at stop.

In Server VAD mode, the Qwen Realtime worker may reconstruct the session at most once. The watchdog triggers when either a server-confirmed speech segment remains active without a transcription event for eight seconds, or text has already appeared and sustained pitch-correlated local speech continues while no server event arrives for eight seconds. The local path is not a pure RMS threshold: ordinary silence and stationary broadband microphone noise cannot consume the reconstruction attempt. A qualifying pre-finish transport failure can use the same one-attempt budget. The worker opens a fresh connection and replays every retained raw PCM packet from the beginning while capture continues feeding the bounded audio-control queue. The replacement starts with a fresh transcript assembler, so replayed text replaces the interrupted preview rather than being appended to it. If the server never emits its first speech event and no qualifying local path exists, Voice Input keeps recording and defers the empty-audio decision until stop-time finalization or full-audio recovery.

If the replacement stalls or disconnects, Voice Input stops trusting Qwen Realtime text and switches to stop-time full-audio recovery. A full audio-control queue and a disconnected worker are diagnosed separately. In every degraded case, local capture, the HUD waveform, and the daemon's complete PCM buffer continue. The client does not send a manual recovery commit in Server VAD mode because the server may reject it and close the connection.

For experimental Audio3, Streaming supplies live and final text over WebSocket, while Native is an HTTP complete-audio pass. One recoverable disconnect before finish-task creates a distinct replacement task, resets the authoritative transcript so the old task's text is discarded, and replays the complete retained PCM prefix at 4× realtime while capture continues. The replacement receives the same start-time terminology snapshot. Prefix retention is capped by the minimum of audio.max_duration_secs, 300 seconds, and 10 MiB of PCM; crossing the cap disables reconnect rather than retaining or replaying an incomplete prefix. A second disconnect, any disconnect after finish-task, or a failed replacement proceeds to Native or local complete-audio recovery rather than opening a third task.

Audio3 Native has three modes. streaming-only never invokes Native. adaptive invokes it for overload, worker interruption, empty/failed/degraded streaming, missing explicit Finished, or recordings of at least 30 seconds; however, a healthy explicitly finished Streaming result that actually sent Session Context is not replaced solely because of the 30-second rule. always invokes Native for every non-cancelled, nonempty recording. Native accepts at most 10 MiB of raw WAV audio, and usable streaming text remains available when a Native request fails.

The waveform analyzer works directly from PCM. It uses a 512-sample window and 256-sample hop, producing 62.5 frames per second at 16 kHz. It publishes thirty mirrored bars plus twelve frequency bands and aggregate voice metrics. During recording, the HUD passes those live bands through a cubic B-spline and maps the result to the capsule's top edge. Before speech is detected, Listening shows a broad, calm virtual spectrum with the same maximum geometric reach as processing; voice onset crossfades to the live spectrum with a roughly 150 ms attack, while pauses decay back over roughly 800 ms. That release blends the measured spectrum directly into the full-height standby envelope and never falls back through a separate procedural perimeter wave. Arming uses virtual bands for its preparation cue. Finalizing, Refining, and Sending share one broad virtual-band profile, one continuously accumulated motion phase, one cadence, and one brightness range. The HUD retains the last live spectrum when the daemon closes the waveform session, then performs a roughly 360 ms breath-like handoff: the halo falls to about five percent visibility while geometry and color move to the processing profile, then rises to full visibility. It clears the retained frame only after the runtime snapshot reaches idle and preserves processing geometry across later phase changes. Those changes only start a roughly 650 ms whole-halo color crossfade, so processing never appears to restart and no spatial color boundary crosses the waveform. The HUD initializes geometry and color transition progress synchronously with each phase change, preventing a target-state frame from rendering before the crossfade begins. Waveform cadence is independent of ASR packet cadence. State refresh is owned centrally by StateStore: it polls the atomically replaced snapshot every 50 ms in active phases and every 100 ms while idle, with no per-surface animation fallback refresh. It strictly validates the snapshot's updated_at_ms/revision version before replacing the UI snapshot. Waveform messages must have a valid session ID and increasing sequence, exactly 30 bars and 12 bands, and finite scalar and array values in [0,1]; a valid frame replaces the complete waveform state atomically. The halo shader outputs premultiplied alpha and returns transparent immediately for fragments wholly inside the capsule.

For local-cli, a background partial thread periodically transcribes the accumulated audio according to audio.partial_interval_ms. On stop, the complete audio is transcribed again.

3. Stop, cancellation, and final ASR

On manual stop or automatic duration-limit stop, the daemon captures the focused destination category when refinement is enabled. This stop-time focus chooses only the refinement style; it does not recapture or replace the terminology snapshot frozen at start. An explicit record cancel skips this lookup.

Capture then detaches or the dedicated pw-record child is terminated, the final partial ASR packet is flushed, and realtime ASR is asked to finish. If no speech event or non-empty transcript has arrived after the 350 ms grace period, the daemon does not cancel immediately: it continues through realtime Finish/manual commit and the normal complete-audio decision. Only a final selected transcript that is still empty returns directly to idle without LLM refinement or text injection.

If final_pass_enabled = true, the daemon writes the complete PCM buffer to a temporary WAV and sends it as a base64 data:audio/wav input to the configured Qwen final model. Success replaces realtime text. Failure uses realtime final text when available, then /usr/bin/voxtype when local fallback is enabled. Without a final pass, usable realtime final text is preferred and local fallback handles remote failure or empty output. A failed one-time reconstruction, queue overflow, or disconnected worker means the remote service did not process a complete recording; its transcript is rejected and recovery uses the complete buffered recording through an enabled final pass or local fallback.

Chinese output runs through OpenCC (t2s or s2t) after recognition.

4. LLM refinement

Refinement is conservative and optional. The destination captured at stop selects the style even when agent session context is disabled: Pi and Codex receive compact Markdown that preserves explicit sequences as ordered lists, sibling enumerations as unordered lists, and distinct parts as paragraphs; installed messaging clients receive conversational punctuation; other targets use lightly formal prose. The model is instructed not to force a simple request into a list or invent headings, hierarchy, or content.

The default llm.timeout_ms = 15000; implementation clamps the shared total budget to 1,000–30,000 ms. Prompt construction, the contextual request, and any eligible transcript-only fallback all consume the same deadline. For budgets of at least 10 seconds, the contextual attempt receives all but the final five seconds so transcript-only recovery still has a chance. A context-free retry can follow a transport error, recognized context/payload error, invalid response, truncation, or contextual budget exhaustion, and only if at least one second remains. Non-retryable HTTP/provider errors keep the original ASR text.

Any error, timeout, truncated response, missing model/credential, or exhausted budget leaves the ASR transcript unchanged. Timing logs record attempt kind, elapsed milliseconds, outcome category, and final choice; they do not print transcript or context.

5. Output

At delivery time, the daemon re-probes the active window to resolve Wayland versus XWayland while retaining the target hint captured at start. Every transcript uses clipboard paste: Wayland writes with wl-copy and asks Hyprland to send the configured shortcut to the active window through dispatch sendshortcut; XWayland uses xclip plus xdotool. The output path creates no per-character synthetic keymap. It backs up the relevant clipboard, sends the paste shortcut, waits 220 ms, and restores the previous content. Native Wayland marks both the temporary transcript and restored payload as sensitive.

When configured, an Fcitx5 guard temporarily closes the active input method before output and restores it afterward.

Settings boundary

The Settings QML edits a full configuration draft, but it does not parse, validate, or write TOML itself. Every backend request and response is one versioned JSON object per line. Requests go to the child process's inherited standard input, and responses return on standard output.

QML applies a 30-second deadline to each request and rejects response lines larger than 2 MiB. A response is accepted only after both the exact protocol envelope and the method-specific result or error payload pass strict shape, type, enum, and size validation. A backend process or protocol failure triggers exponential restart delays from 250 ms up to 8 seconds, with at most six automatic attempts; only a valid response resets the failure count.

Rust exclusively loads and validates configuration. A load response includes an opaque revision derived from the exact source that was read. Save sends that revision and the complete supported configuration; Rust rejects the save if the exact source changed in the meantime, preventing one Settings window from silently overwriting another editor. Successful saves preserve all supported fields, create the configuration directory with mode 0700, write the file with mode 0600, and atomically replace it.

Credential actions are keep or replace. Password inputs travel only through inherited stdin to Rust and then through stdin to systemd-creds; they do not enter TOML, process arguments, environment variables, logs, or backend responses. QML clears each credential field immediately after submitting Save or Test LLM. Because QML/JavaScript strings are managed memory, this clearing is best effort rather than a guaranteed zeroization primitive.

Test LLM can use either the newly entered LLM credential or the encrypted-store credential. Save may request a daemon restart. A restart failure is reported separately from persistence, so a successfully written configuration is not misreported as an unsuccessful write.

The read-only runtime.get method probes voice-input.service and parses the canonical runtime snapshot independently from save requests. Its response is constructed from an allowlist: bounded service state plus phase, update time, language, engine, and model. Transcript variants, tooltips, output-target data, runtime error text, and credentials are never returned. Probe failure produces an unknown/unavailable Overview state and does not block editing or replace a save error.

Concurrency boundaries

Boundary	Why it exists
Capture service or dedicated reader thread	Reads PipeWire continuously without waiting for network or UI work.
Bounded ASR audio-control queue + backend worker	Capture offers packets without waiting for WebSocket I/O. Queue overflow marks the stream incomplete and moves recovery to the complete audio buffer. ASR events currently use a standard unbounded Rust mpsc channel; only the audio-control queue is bounded.
Local partial-ASR thread	Periodic previews do not run in the control listener.
Start-time terminology worker	Pi/Kitty/Hyprland inspection and local extraction can overlap capture; consumers share one immutable result.
Waveform publisher thread	Uses a bounded queue and nonblocking sends; full queues may drop visual frames instead of blocking capture.
Serialized state update lock	Keeps concurrent capture and ASR updates ordered; each JSON snapshot uses temporary-file rename.
Output child-process runner	Drains piped stdout and stderr concurrently, caps each at 16 MiB, applies one deadline to stdin/I/O/process completion, and kills the child process group on timeout, size-limit violation, or I/O failure.
Resident HUD Quickshell process	StateStore polls state every 50 ms during active phases and every 100 ms while idle, and consumes validated waveform socket frames independently. HUD failure does not own recognition.
On-demand Settings + Rust child	Uses request IDs and versioned NDJSON; strict response validation and exact-source revision checks protect the boundary.
Control connection threads + daemon mutex	Accepts bounded concurrent local clients without allowing command execution itself to overlap.

The control socket permits at most 32 active connections. Commands are capped at 4 KiB, responses at 64 KiB, and server-side read/write I/O has a two-second timeout. Each connection may wait for the daemon mutex, which still serializes command handling through final ASR, refinement, and output. Recording controls are stamped with the current idle generation when accepted; a command is ignored as stale if the daemon reaches a newer idle generation before that command acquires the mutex. Staleness is not based on a 750 ms age cutoff.

Build-time shader validation

CI compiles the HUD shader with QSB and verifies all six packaged targets: SPIR-V 100, GLSL ES 100, GLSL 120, GLSL 150, HLSL 50, and MSL 12. It also parses QSB reflection metadata and checks the uniform block binding plus the reflected qt_Matrix and qt_Opacity layout.

Runtime state machine

idle → arming → recording → transcribing → refining → outputting → idle, with error available from any failed control/session operation. Refining is skipped when disabled. A cancelled session or a final empty/no-words result returns directly to idle without output.

See also: Configuration · Agent Context · Desktop Integration