Voice-first iOS assistant for hands-free Gmail and Google Calendar — built for use while driving.
- Capture — on-device VAD (Picovoice) and STT (Whisper.cpp) turn speech into text.
- Plan — the server uses Gemini / OpenRouter to pick a tool and extract parameters from the transcript.
- Execute — Gmail, Calendar, and People APIs are called directly from the device using the user's OAuth token. Results are summarized server-side and spoken back via Kokoro TTS.
The server never touches Google data — it only plans and summarizes. All API calls happen on-device.
| Layer | Stack |
|---|---|
| Mobile | React Native 0.83, React 19, TypeScript |
| Voice | Picovoice (VAD), Whisper.cpp (STT), Kokoro (TTS) — native TurboModules |
| Backend | Node.js, Koa 3, PostgreSQL 16 |
| LLM | Gemini, OpenRouter, OpenAI |
| Auth | Google Sign-In, JWT |
Requires Node 20+, Docker, Xcode, Ruby/Bundler, and a Google Cloud project with OAuth + Gmail/Calendar/People APIs enabled.
Backend — create server/.env:
POSTGRES_HOST=localhost
POSTGRES_PORT=5432
POSTGRES_USER=...
POSTGRES_PASSWORD=...
POSTGRES_DB=workfromcar
JWT_SECRET=...
GEMINI_API_KEY=...
OPENAI_API_KEY=...
OPEN_ROUTER_API_KEY=...
GOOGLE_AUTH_CLIENT_ID=... # web client id
GOOGLE_AUTH_CLIENT_ID_APP=... # ios client idcd server
make run-db
npm install && make run-server # serves on :3000App — create app/.env:
API_BASE_URL=http://localhost:3000
GOOGLE_WEB_CLIENT_ID=...
GOOGLE_AUTH_CLIENT_ID_APP=...cd app
npm install && bundle install
cd ios && bundle exec pod install && cd ..
npm run ios| Endpoint | Description |
|---|---|
POST /api/auth/google |
Exchange Google ID token for JWT |
POST /api/agent/plan |
Transcript → structured tool plan |
POST /api/agent/executePermission |
Validate intent for sensitive actions |
POST /api/agent/summarize |
Summarize tool results as natural speech |
Gmail: createDraft, summarizeEmails, readEmail, replyToEmail, forwardEmail, resolveContact
Calendar: createEvent, updateEvent, deleteEvent, getEvents, respondToEvent