whaley

Voice-first macOS assistant for research, document creation, and email actions.

Electron desktop app, OpenAI Realtime voice loop, approval-gated tools, and deterministic verification.

Overview

whaley is a local macOS voice assistant built for fast spoken interaction and controlled tool execution. You hold Command, speak naturally, receive spoken responses, and let the app perform bounded tasks such as web research, .docx generation, and email sending.

Unlike a generic chat wrapper, this project is structured around a desktop runtime:

• Electron main process for lifecycle, permissions, IPC, and tool boundaries
• Preload bridge with a minimal typed API surface
• Renderer overlay for transcript, session state, onboarding, and approvals
• OpenAI Realtime session flow for low-latency voice interaction
• Local persistence for configuration and execution history
• Test and governance scripts for repeatable phase-based verification

Vision

whaley is designed as a practical voice operating layer for macOS rather than an unbounded autonomous agent.

• At the interaction layer: it should feel immediate, conversational, and interruptible
• At the execution layer: side effects should be explicit, reviewable, and auditable
• At the engineering layer: behavior should be testable through deterministic scripts and E2E checks

The project goal is a voice-first desktop assistant that is useful in daily work without hiding what it is doing.

Core Capabilities

• Push-to-talk voice control using the Command key
• Live transcript updates while listening
• Spoken assistant responses with interruption support
• Approval-gated external actions for safer execution
• Web research through the browser tool with summarized output
• Local .docx document creation
• Email sending through Gmail credentials or gogcli
• First-run onboarding for user name, OpenAI API key, and optional Gmail setup
• Local config and SQLite-backed runtime records

Workflow

1. Onboarding: collect the user name, validate the OpenAI API key, and optionally save Gmail credentials.
2. Voice Activation: the operator holds Command to enter listening mode and streams speech into the realtime session.
3. Realtime Session: the renderer connects to OpenAI Realtime, updates transcript/UI state, and receives tool events.
4. Tool Handling: tool requests are routed through the main process so filesystem, email, and command execution stay outside the renderer.
5. Approval Gate: actions with external impact require explicit confirmation before execution.
6. Result Delivery: the app speaks the response, updates the UI, and records artifacts for later verification or debugging.

Architecture

Runtime

• Main process: app lifecycle, single-instance lock, permissions, IPC handlers, tool execution
• Preload: safe renderer bridge for config, files, tools, and session events
• Renderer: overlay UI, onboarding controller, waveform, transcript, state machine
• Agent layer: OpenAI Realtime agent plus tool definitions for search, document creation, and email
• Persistence: config storage under ~/.jarvis/config.json and SQLite-backed local records

Repository Layout

• src/main/ - Electron lifecycle, hotkeys, permissions, document writer, tool executor
• src/renderer/ - overlay UI, onboarding flow, transcript/audio/session state
• src/agents/ - realtime agent configuration and tool definitions
• src/shared/ - shared types, constants, config, debug logging, SQLite helpers
• scripts/ - build helpers, checkpoints, E2E governance, release verification
• tests/ - automated validation
• CONSTITUTION_FOR_PROJECT/ - product spec, plan, contracts, quickstart, research, and governance material

Quick Start

Prerequisites

• macOS 13 or later
• Bun
• OpenAI API key with Realtime access
• Microphone permission granted to your development runner or packaged app
• Accessibility permission granted for global hotkey capture
• Optional Gmail CLI fallback: brew install gogcli

Installation

git clone <your-repo-url> whaley
cd whaley
bun install
bun run build
bun run dev

First Launch

On first launch, the app opens onboarding when no config is present. The current implementation stores local config in ~/.jarvis/config.json, so the internal storage path still uses the legacy project name.

You will be asked for:

• your name
• OpenAI API key
• optional Gmail address
• optional Gmail app password

Development Commands

bun run dev
bun run build
bun run test
bun run typecheck
bun run lint

E2E-Friendly Launch

ELECTRON_ENABLE_LOGGING=1 bunx electron . --remote-debugging-port=9222

Permissions

Accessibility

Required for global hotkey handling.

1. Open System Settings
2. Go to Privacy & Security
3. Open Accessibility
4. Enable access for Terminal, iTerm, your IDE, or the packaged app bundle

Microphone

Required for the realtime voice session.

1. Open System Settings
2. Go to Privacy & Security
3. Open Microphone
4. Enable access for Terminal, iTerm, your IDE, or the packaged app bundle

Restart the app after granting permissions.

Verification and Quality Gates

This repo is built with a strict verification model documented in CONSTITUTION_FOR_PROJECT/.

• TypeScript checks via bun run typecheck
• Unit and integration coverage via bun run test
• Phase checkpoints via bash scripts/run-phase-checkpoint.sh <phase>
• Governance E2E checks via bash scripts/e2e-governance-check.sh <phase>
• Release validation via startup benchmark and soak scripts in scripts/release/

The project documentation defines happy, edge, and error validation lanes for every major phase.

Troubleshooting

• Hotkey does not respond: recheck Accessibility permission and restart the app
• Microphone capture fails: verify microphone permission for the process launching Electron
• Realtime session fails: confirm the OpenAI API key has Realtime access
• Email sending fails: configure Gmail credentials in onboarding or install gogcli
• App closes or misbehaves on launch: inspect the local debug log and run bun run typecheck

Notes

• The public project name in this README is whaley
• Some implementation details still use the legacy jarvis name internally, including config and document storage paths such as ~/.jarvis and ~/Jarvis/documents
• If you want a full codebase rename, that should be handled separately across package metadata, app labels, storage paths, and runtime strings

Acknowledgments

whaley builds on:

• Electron
• Bun
• @openai/agents
• sql.js
• docx