Repository navigation
Testing Guide
Three layers, each fast and focused:
| Layer | Runner | What it covers | Where |
|---|---|---|---|
| Unit |
vitest (tests/unit/) |
Pure functions: FSM, protocol parser, URL utils | tests/unit/*.test.ts |
| Integration |
vitest (tests/integration/) |
Services with fakes for I/O — HermesClient, STT, TTS, orchestrator | tests/integration/*.test.ts |
| Bridge | pytest |
Python bridge /ws handshake, auth, SSE parsing |
server/hermes-voice-bridge/tests/ |
| E2E | @playwright/test |
Packaged app, mic permission flow, pairing wizard, PTT | tests/e2e/*.spec.ts |
Run the full TS suite:
npm test # vitest run (unit + integration)
npm run test:e2e # Playwright (slow)Run a single file:
npm test -- tests/unit/state-machine.test.tsLightweight, no Electron. Imports the pure modules from
src/shared/ and asserts directly.
Drives the State-Machine through every transition listed in the diagram. 28 cases as of writing, covering:
- Initial state per activation mode.
- Every valid transition (PTT_PRESS, PTT_RELEASE, WAKE_DETECTED, …).
- Every invalid event-in-state returns the same context reference
(the orchestrator uses
===to skip emit). -
RESETfrom ERROR clears all transient fields. -
PTT_PRESSfrom ERROR is the auto-recovery path. - Mode switches honoured only in quiet states.
- Turn IDs increment across consecutive turns via the State-Machine#determinism-the-reducerenv.
A typical case:
it('PTT_PRESS from IDLE creates a new turn', () => {
const env = makeEnv(); // returns 'turn-1', 'turn-2', ...
const ctx = initialContext('PUSH_TO_TALK');
const next = reduce(ctx, { type: 'PTT_PRESS' }, env);
expect(next.state).toBe('CAPTURING');
expect(next.turnId).toBe('turn-1');
});Round-trips every message shape through parseClientMessage /
parseServerMessage. Asserts:
- valid payloads parse back to the original;
- missing required fields return
null(no exceptions); - forward-compatibility: extra fields are ignored;
- the binary-after-header pairing on the WS layer.
normalizeBridgeUrl covers the user-typed URL space:
| Input | Normalized | pathWasAdded |
|---|---|---|
ws://host:8765 |
ws://host:8765/ws |
true |
ws://host:8765/ |
ws://host:8765/ws |
true |
ws://host:8765/ws |
ws://host:8765/ws |
false |
wss://host:8765/ws/ |
wss://host:8765/ws |
false |
Also rejects invalid schemes (http://, tcp://).
Vitest with fakes for every external dependency. No real network, no real audio, no Electron — they run in 200 ms each.
Drives HermesClient against a fake WebSocket (tests/integration/__mocks__/fake-ws.ts).
Asserts:
- hello/welcome handshake.
- Reconnect backoff: attempt N delay is correct, no reconnect on explicit disconnect.
- Pong timeout terminates the socket and reconnects.
- Binary-after-header pairing.
- Status emission semantics (connecting → connected → error).
Timers are injected so the heartbeat can be stepped without
vi.useFakeTimers():
const setT = vi.fn(setTimeout) as any;
const client = new HermesClient({ setTimeout: setT, ... });The orchestrator is wired against fakes for the WS client, STT, TTS. Each test drives a sequence of events and asserts on:
- emitted events (
state,transcript_final,response_text,tts_chunk), - FSM state at each step,
- buffer clearing on cancel,
- barge-in doesn't race SPEAKING → CAPTURING.
This is the regression net for Conversation-Orchestrator changes.
Use a fake spawn to simulate subprocess behaviour:
const spawnImpl = vi.fn((bin, args, opts) => {
const proc = new FakeChildProcess();
setTimeout(() => proc.stdout.emit('data', Buffer.from('hello\n')), 10);
setTimeout(() => proc.emit('close', 0), 20);
return proc;
});
const adapter = new WhisperLocalAdapter({ spawnImpl, ... });
const result = await adapter.transcribe({ pcm: buf, language: 'en' });
expect(result.text).toBe('hello');Same pattern for ElevenLabs streaming via fetchImpl: vi.fn().
Tests electron-store against a temp directory:
beforeEach(() => {
app.setPath('userData', mkdtempSync(...));
});Asserts defaults, deep merge, migration, listener add/remove.
testPairing() against fake WebSocket responses — exercises the
friendly-error mapping for every failure mode (401, 404, ECONNREFUSED,
ENOTFOUND, certificate errors, timeout).
This one talks to a real running bridge. It's describe.skip'd
unless both VG_BRIDGE_URL and VG_BRIDGE_TOKEN are set:
VG_BRIDGE_URL=ws://10.0.19.1:8765/ws \
VG_BRIDGE_TOKEN=kug4fJKR... \
npm test -- tests/integration/bridge-text-roundtrip.test.tsThe test sends the literal text "oi" as a final transcript and waits
for at least one non-empty response_text delta back. Times out at
90 s by default — override with VG_BRIDGE_TIMEOUT_MS=120000 if your
Hermes is slow.
Use this when:
- You change the bridge protocol — confirms desktop ↔ bridge ↔ Hermes is still end-to-end intact.
- A user reports "no response" — gives you a deterministic reproducer outside the desktop UI.
Pytest, from inside the bridge's venv:
cd server/hermes-voice-bridge
source venv/bin/activate
pytest -qCoverage:
-
/wsupgrade with valid + invalid Bearer tokens. - The hello/welcome handshake shape.
-
_run_turndispatching against a fake adapter that yields chosen deltas (with and without final). - The non-stream fallback when the streaming adapter yields zero.
-
_iter_sse_deltasboundary cases (split across TCP chunks, missing trailing newline,[DONE]sentinel).
Slow but high-confidence. Launches the packaged .app and drives
the real UI:
npm run build:mac # produce release/*.dmg
npm run test:e2eCurrently covers 65 tests across 19 spec files (~3 min wall-clock, single worker):
- Pairing wizard happy path + 3-step navigation + cancel-on-step-2 + whitespace-trim on token paste.
- PTT button click →
CAPTURINGorb → release →THINKING→ either empty-transcript IDLE recovery or full reply playback. - Wake-word path (openww + custom phrase), barge-in mid-utterance,
brief
data-just-wokevisual flash on the main window. - Settings panel opens in a separate BrowserWindow, edits persist across windows and across app restart; STT/TTS provider live-swap rebuilds the orchestrator.
- Output device live-switch during SPEAKING; Voz Testar with
custom text streamed through
AudioPlayback. - Window-chrome UX: window-title FSM suffix, Cancel X during CAPTURING, hotkey hint per activation mode, "Guardado" indicator flash, ReadinessPill, error-toast Copy Diagnostics + Escape, tray menu with Cmd+, opening Settings, Cmd+L wipes transcript, Cmd+R retries after ERROR, transcript copy + counter chips, click connection indicator to reconnect, mute toggle, VU meter visible during CAPTURING, system notification on reply when window hidden.
- Settings → Avançado: factory reset, auto-launch toggle, "Abrir registo de eventos" reveals the log file path.
- PairingWizard: wizard cancel link, token paste whitespace trim, datalist + chip list of recent bridge URLs, draft URL persistence across launches.
- Re-pair to a different bridge mid-session and recover; reconnect
after the server bounces the socket; capability negotiation
(
hello+welcomeshape).
The Playwright launcher uses _electron.launch({ executablePath: appPath })
so the test boots the same signed bundle a user installs — catching
afterPack hook regressions (missing Info.plist key would surface as a
mic-permission timeout here).
The test pyramid is sized to fit a free GitHub Actions tier:
| Job | Matrix | Time |
|---|---|---|
| Unit + integration |
ubuntu-latest, Node 22 |
<30 s |
| Bridge |
ubuntu-latest, Python 3.10/3.11 |
<20 s |
| E2E (smoke) |
macos-latest, Node 22 |
~3 min |
| Build artefacts | matrix of macos/linux/windows | ~5 min |
A full live round-trip is not part of CI — it requires a Hermes
instance and a real network. It lives in tests/integration/ for
on-demand use only.
-
Pure logic (no Electron, no IO) →
tests/unit/. Vitest. Be strict about purity — if you find yourself mocking modules, you're in the wrong layer. -
Service with IO →
tests/integration/with the dependency injected as an option. Mirror the pattern inhermes-client.test.ts(fake WS) orstt-service.test.ts(fake spawn). -
End-to-end visible-to-user behaviour →
tests/e2e/. Don't be shy about adding here for important flows; the slow runtime is worth catching whole-system regressions.
The full suite must stay green before merging — npm test is the
gatekeeper.
Every Playwright spec talks to the packaged .app (not the dev
build) so we exercise the exact entitlements, signature, and
extra-resources layout a user installs. Almost everything they need is
in
tests/e2e/helpers/rig.ts.
import { expect, test } from '@playwright/test';
import { MOCK_DEFAULT_TOKEN, startMockBridge } from '../integration/__mocks__/mock-bridge-server';
import { scriptedTextReply } from './helpers/mock-bridge-presets';
import { launchPackaged, packagedAppExists, type TestRig } from './helpers/rig';
test.describe('my new behaviour', () => {
let rig: TestRig | null = null;
test.beforeAll(() => {
if (!packagedAppExists()) test.skip(true, 'run `npm run build:mac` first');
});
test.afterEach(async () => { await rig?.dispose(); rig = null; });
test('does the thing', async () => {
const bridge = await startMockBridge({
onClientMessage: scriptedTextReply('hello from the mock'),
});
rig = await launchPackaged({
bridgeUrl: bridge.url,
bridgeToken: MOCK_DEFAULT_TOKEN,
});
// ... drive the UI via rig.mainWindow, assert what you need
await bridge.close();
});
});rig.dispose() is idempotent and tears down both the Electron process
and the temp userData directory.
-
mkdtempa freshuserDataso tests don't see each other's state. -
writeSeedSettings(...)— pre-populates electron-store with a valid pairing pointing at your mock bridge, so the app skips the wizard and lands on the main screen. -
electron.launch({...})with sane defaults:-
--autoplay-policy=no-user-gesture-requiredsoAudioContextcan start synthesising without a real click; -
--use-fake-device-for-media-stream+--use-file-for-fake-audio-capture=<wav>if you passfakeAudioFile. Chromium loops the WAV through everygetUserMediacall.
-
- Console forwarding (errors + warnings) onto the Playwright stdout.
Set
VG_E2E_VERBOSE=1to also forwardinfo/debugand the main-process stderr — useful when a connection silently fails.
For the PairingWizard specs — same as launchPackaged but without
seeding a pairing. The app boots straight to step 1 of the wizard.
Triggers the same IPC the gear icon uses and returns the freshly-opened
BrowserWindow Page, with the same log forwarding installed.
Subscribes (in-page) to vg.conversation.onTtsChunk,
onResponseText, onState, onWarning, onError and stashes them on
window.__vg_*. readVgStats returns the accumulated counts so
assertions like expect.poll(() => readVgStats(page).chunks) work.
tests/e2e/helpers/mock-bridge-presets.ts
ships the common onClientMessage recipes:
| Preset | Behaviour |
|---|---|
captureTranscripts(sink) |
Push every final transcript into the array. |
scriptedTextReply(text) |
On end_turn: thinking → response_text(final) → response_end. |
scriptedError({code,message}) |
On end_turn: send an error frame. Used by the auto-recovery spec. |
sendServerAudio(ws, {…}) |
One-shot helper for the binary-after-header audio path. |
composeBridge(a, b, …) |
Stack handlers — observe AND script. |
The behaviour specs that test the orchestrator pipeline (cancel,
barge-in, error recovery, multi-turn, …) don't want to depend on a
working whisper.cpp install. Set
extraEnv: { VG_E2E_FAKE_TRANSCRIPT: 'olá' } and the main process
swaps in a tiny in-process fake STT adapter that always returns the
env var as the transcript. Production code stays untouched when the
var isn't set; the branch lives in
src/main/index.ts → bootstrapConversation.
Same idea for the wake-word path. With VG_WAKE_E2E_FAKE=1, main
spawns
resources/python/fake_wake_runner.py
instead of wake_word_runner.py. The fake speaks the same JSON
stdout protocol but emits ready → wake on a fixed timeline without
touching the microphone or any model.
Pre-recorded WAVs live in tests/e2e/fixtures/:
| File | Content | Use |
|---|---|---|
hi-how-are-you.wav |
"HI! How are you today?" | audio-conversation full turn |
wake-phrase.wav |
"hey hermes wake up please" | reserved for future wake spec |
Recipe to regenerate on macOS:
say -v Samantha -o /tmp/x.aiff "HI! How are you today?"
afconvert /tmp/x.aiff -d LEI16 -c 1 -r 16000 -f WAVE tests/e2e/fixtures/hi-how-are-you.wavFormat: mono PCM16 @ 16 kHz, which is what whisper-cli expects.
All files | 86.87 % stmts | 84.23 % branches | 91.89 % funcs
src/shared | 97.89 % stmts
src/main/services | 79.78 % stmts
conversation-orchestrator.ts | 86.17 % stmts
hermes-client.ts | 89.76 % stmts
settings-store.ts | 100 % stmts
stt-service.ts | 78.11 % stmts
tts-service.ts | 73.78 % stmts
wake-word-service.ts | 60.90 % stmts
protocol.ts (shared) | 98.23 % stmts
The tts-service and wake-word-service auto-install pipelines
(venv creation, pip install, Hugging Face downloads) deliberately
remain uncovered by unit tests — they require elaborate spawn
sequencing and real filesystem state. The Playwright suite exercises
the happy paths against the packaged app instead.
The shared layer (FSM, protocol, helpers) is essentially fully covered.
The remaining gap in src/main/services/ is mostly subprocess
fan-out paths — Piper venv auto-install, brew install, openWakeWord
PortAudio init — that are exercised by the Playwright E2E suite (not
counted in vitest coverage) and by the live bridge round-trip. Pure
unit coverage above ~90% is not realistic for those without a much
heavier mock layer; the E2E suite is the right place to catch them.
The types.ts declaration file is intentionally excluded from the
coverage report — it has no runtime statements and would appear as 0 %.
tests/e2e/
├── audio-conversation.spec.ts real-audio full turn (fake mic → whisper → bridge → Piper)
├── connection.spec.ts WS reconnect, wizard URL validation (2)
├── conversation-advanced.spec.ts wake→full turn, MP3 server audio,
│ transcript rendering, re-pair mid-session (4)
├── conversation-extras.spec.ts cancel, persistence, provider swap,
│ server PCM audio, factory reset (5)
├── conversation-flows.spec.ts short tap, barge-in, error recovery,
│ multi-turn, settings broadcast (5)
├── mic-capture.spec.ts getUserMedia probe, real-mic RMS (2)
├── pairing.spec.ts wizard happy path, friendly error (2)
├── runtime-extras.spec.ts output-device live-switch, error-toast
│ contents, wizard server_version (3)
├── runtime-protocol.spec.ts error frame, audio backpressure (50 chunks),
│ capability negotiation, hotkey persist,
│ wake-event safety from non-rest states (5)
├── settings-audio.spec.ts speaker selector, custom-text TTS test (2)
├── settings-deep.spec.ts STT language, Piper voices, OpenAI key,
│ Re-emparelhar wizard surface (4)
├── ux-round8.spec.ts window title FSM suffix, cancel X during
│ CAPTURING, data-just-woke wake flash,
│ hotkey-hint mode-aware, "Guardado" flash,
│ wizard step label + cancel link (7)
├── ux-round9.spec.ts Cmd+L clears transcript, Cmd+R retry-from-
│ ERROR, copy/clear chips + turn counter,
│ click-to-reconnect indicator, mute toggle (5)
├── ux-round10.spec.ts auto-launch toggle persists, log file reveal,
│ VU meter visibility, system notification on
│ reply when window hidden, wizard URL chips
│ + draft persistence (5)
├── ux-shortcuts.spec.ts Escape dismisses error toast,
│ Cmd+, opens Settings, "Copiar diagnóstico"
│ button, ReadinessPill, token paste trim (5)
├── visual-states.spec.ts warning toast lifecycle, StateOrb attr (2)
├── wake-phrase-validation.spec.ts validation hint + Testar enable/disable (1)
├── wake-word.spec.ts openww + phrase + tester reset (3)
└── wizard-nav.spec.ts back navigation preserves URL/token,
token field is multi-line monospace (2)
65 specs totalling ~3 minutes for a full local run. The real-audio one is the only slow case (~20 s — STT + Piper warmup); everything else is <3 s each thanks to the fake STT / fake wake runner / mock bridge stack.
The rig allocates a fresh mkdtemp(userData) per spec — on macOS the
default os.tmpdir() is /var/folders/... on the system volume. A
full 43-spec Playwright run plus electron-builder's DMG staging burns
through ~3-5 GB of system tmp; on low-disk systems this kills the run.
Set VG_E2E_TMPDIR=/Volumes/<roomy-disk>/.vg-e2e and the rig will
allocate userData directories there instead (with a fallback to
os.tmpdir() if the override isn't writable). Falls back silently —
no harm in setting it always.
tests/e2e/helpers/rig.ts
ships these in addition to launch helpers:
| Helper | Purpose |
|---|---|
holdPtt(page, ms) |
Press → hold → release the call button via pointer events. |
waitForState(page, [s]) |
Poll __vg_state_log for one of the desired FSM states; throws an actionable error on timeout. |
instrumentTtsCounter |
Subscribe in-page to TTS / state / warning / error events. |
readVgStats(page) |
Snapshot the accumulated counters + event log. |
sttReady(page) |
Ask main if Whisper is wired (skip cleanly otherwise). |
ttsReady(page) |
Same, for Piper. Blocks while the venv auto-install runs on a fresh userData. |
tests/e2e/helpers/driver.ts
exposes a fluent ConversationDriver:
const driver = await ConversationDriver.attach(rig.mainWindow);
await ttsReady(rig.mainWindow); // skip if Piper isn't installed
const stats = await driver.runTurn({ holdMs: 200, until: ['IDLE'] });
expect(stats.chunks).toBeGreaterThan(0);driver is just sugar over holdPtt + waitForState + readVgStats,
but turns each spec body into a sequence of intent rather than
orchestration plumbing. Use driver.pressPtt() / releasePtt() for
barge-in / interrupt tests.
tests/integration/settings-store.test.ts
covers the on-disk persistence + schema-migration path:
- v1 → v2 migration adds the new
wakeMode+wakePhrasefields with defaults while keeping every other user-set field intact. - v2 files round-trip untouched.
-
set()writes survive a store re-creation against the samecwd. -
reset()wipes pairing and revertsschemaVersion. -
onChangelisteners unregister cleanly.
The store gained a createSettingsStore({ cwd }) option so tests can
point it at a mkdtemp() directory without spinning up Electron.
Playwright's
playwright.config.ts
defaults to trace: 'retain-on-failure', screenshot: 'only-on-failure',
video: 'retain-on-failure'. After a failed local run:
ls test-results/
# pick the failing test's directory, then:
npx playwright show-trace test-results/<dir>/trace.zipThe trace viewer shows every network call, IPC payload, and DOM state at each step — usually enough to spot the broken assumption.
Voice Gateway — VivaldiCode/voice-gateway. MIT licensed. Wiki sources live under docs/; run ./docs/sync-wiki.sh to publish.
Getting started
Architecture
- Architecture — system overview
- State-Machine — conversation FSM
- Conversation-Orchestrator — main glue
- WebSocket-Client — Hermes connection
- IPC-Layer — main ↔ renderer
- Settings-And-Storage — config persistence
Audio
- Audio-Pipeline — capture & playback
- Speech-To-Text — STT adapters
- Text-To-Speech — TTS adapters
- Wake-Word-Detection — openWakeWord
Server
- Hermes-Voice-Bridge — Python bridge
- Protocol — WebSocket spec
Platform
- Build-And-Packaging — electron-builder
- macOS-Permissions — TCC, signing
- Renderer-UI — React components
Project
- Testing-Guide — unit, integration, E2E
- Development-Guide — contributing
- Glossary — terms