How It Works

How It Works

A contributor's tour of the YazSes internals. The goal here is orientation — enough to find the file you need and understand how a spoken phrase becomes typed text. For per-module depth, read the code; each module is small and focused.

Architecture at a glance

flowchart LR
    subgraph UI["Control surface"]
        CLI["yazses CLI"]
        TRAY["tray icon"]
    end
    CLI -- "JSON-RPC / IPC" --> D
    TRAY -- "JSON-RPC / IPC" --> D

    subgraph D["yazses-daemon · core/daemon.py (state machine)"]
        direction LR
        HK["1· Hotkey<br/>keyboard / EMG"] --> AUD["2· Audio<br/>capture · VAD · padding"]
        AUD --> STT["3· Speech-to-text<br/>faster-whisper (CPU int8)"]
        STT --> PP["4· Post-process<br/>cleanup · disfluency"]
        PP --> CMD{"5· Command<br/>or dictation?"}
    end

    CMD -- "dictate" --> INJ["Inject into focused app<br/>xdotool / ydotool / wtype"]
    CMD -- "command" --> KEYS["Send key sequence<br/>undo · save · go to line…"]
    INJ -. "SSH tunnel" .-> REMOTE["remote host"]

    subgraph PLAT["Platform abstraction · platform/ (Protocols)"]
        LNX["linux"]
        MAC["macos"]
        WIN["windows"]
    end
    PLAT -. "implements" .-> D

    CFG[("config.toml")] -. "reads" .-> D

    classDef offline fill:#ede7f6,stroke:#5e35b1,color:#311b92;
    class D,STT offline;

Loading

Everything above runs on your machine. There are no network calls in the dictation path — audio is transcribed locally and never uploaded. The full, always-current diagram (with the opt-in learning loop and v2 cognitive layer) is on the documentation site.

The pipeline, end to end

When you hold the hotkey and speak, this is the journey of your audio:

Hotkey held (keyboard hook, or a USB EMG squeeze sensor)
   → Audio captured (sounddevice → numpy buffer)
   → Voice-activity gate (calibrated RMS threshold) + pre-speech padding
   → faster-whisper transcribes locally  ←── optional context prompt (vocabulary, editor)
   → Text cleanup (strip Whisper artefacts) + disfluency filter
   → Command grammar: is this dictation, or a command like "undo" / "new line"?
   → Dispatch:
        • DICTATE → (optional offline LLM cleanup) → inject into the focused app
        • COMMAND → translate to a key sequence and send it
   → Injection: type locally, or forward over SSH to a remote host

The whole thing is orchestrated by one long-lived process — the daemon — which talks to the CLI and the tray icon over a small JSON-RPC IPC channel.

The daemon is a state machine

src/yazses/core/daemon.py is the orchestrator. It moves through:

LOADING → IDLE ↔ RECORDING → TRANSCRIBING → INJECTING → IDLE

plus side-states for remote setup, enrollment, meeting capture, and errors. It owns the IPC server, signal handling, and all the pipeline wiring above. If you're changing behaviour, you'll almost always touch this file or one of the modules it wires together.

The module map

Area	Where	What it does
Orchestration	core/daemon.py	The state machine + all pipeline wiring.
OS abstraction	platform/	base.py declares Protocol interfaces (hotkey, injector, lifecycle, IPC, permissions, tray). linux/, macos/, windows/ each implement them. factory.py returns the right bundle.
Hotkey	hotkeys/, platform/*/	"Key held ≥ N ms → start/stop recording." Linux uses evdev; an optional USB EMG squeeze sensor is an alternate hotkey backend.
Audio	audio/	Capture, voice-activity gating, pre-speech padding, and input-device resilience (auto-heal when the OS default mic changes).
Speech-to-text	stt/	Wraps faster_whisper.WhisperModel; optional streaming decode; a disfluency filter.
Post-processing	postprocess/	Text cleanup, continuation spacing, spoken-punctuation, and optional offline LLM cleanup.
Commands	commands/	A fast regex grammar classifies dictation vs. commands; an optional small-model router handles the ambiguous cases.
Injection	inject/	Types into the focused window (xdotool / ydotool / wtype / clipboard, auto-selected) — with a guard that avoids typing when no text field is focused.
IPC	ipc/	JSON-RPC 2.0 over a Unix socket (Linux/macOS) or named pipe (Windows) — how the CLI and tray talk to the daemon.
CLI	cli.py	The yazses command (start/stop/status/doctor/features/…).
Tray	tray/, platform/linux/tray.py	A top-bar status icon whose colour tracks the daemon state and whose menu picks/pins the mic and restarts the daemon.

Design principles you'll see everywhere

1. On-device only. No network calls in the dictation path. If a feature seems to need the cloud, it's either off-by-default-and-clearly-labelled or it doesn't ship.
2. Protocols over if platform ==. New OS support means implementing the Protocol interfaces in platform/base.py and registering the platform in factory.py — the daemon and CLI need no changes. See Contributing → adding a platform.
3. Off by default. Advanced features are dormant until enabled (yazses features enable <name>), so a fresh install is simple and safe.
4. Pure cores, injected backends. The heavy/OS-specific parts are injected so the logic can be unit-tested without hardware. That's why the test suite is large and fast.

Where to start reading

• Want to fix dictation behaviour? → core/daemon.py + the relevant audio/, stt/, or postprocess/ module.
• Want to add a voice command? → commands/grammar.py.
• Want to add OS/injection support? → platform/base.py + a new platform/<os>/ or inject/ backend.
• Want a docs/UX fix (great first PR)? → docs/, cli.py, or system/doctor.py.

Then head to Contributing — Start Here.