stob — Voice-driven Web Automation with Stagehand

This project is a voice-controlled web automation agent built on top of Stagehand (Playwright-based). It can:

• Run a demo that automates a browser session and showcases Stagehand’s extract/act/observe and agent capabilities
• Run an interactive voice loop that listens to your microphone, transcribes speech, interprets intent via LLMs, and executes actions in the browser

Quick Start

npm install
npm start        # Demo run (Browserbase session)
# or
npm run voice    # Interactive voice control (local browser)

---

Prerequisites

• Node.js 18+ and npm
• macOS/Linux/WSL. For voice mode on macOS, ensure microphone permissions are granted to your terminal.
• For voice capture, SoX is required by node-record-lpcm16:
  • macOS: brew install sox
  • Linux: sudo apt-get install sox (or your distro’s package manager)

Install

npm install

Environment Variables

Create a .env file in stob/ and add API keys:

touch .env
open -a TextEdit .env  # or use your editor

Required keys (use at least one per category where applicable):

• OPENAI_API_KEY — used for Whisper fallback (speech-to-text)
• CARTESIA_API_KEY — used for primary speech-to-text via Cartesia (ink-whisper)
• GOOGLE_API_KEY — used for Gemini 2.5 Flash (LLM fallback via OpenAI-compatible endpoint)
• CEREBRAS_API_KEY — used for fast LLM command interpretation

Example .env snippet:

OPENAI_API_KEY=sk-...
CARTESIA_API_KEY=ct-...
GOOGLE_API_KEY=ya29-...
CEREBRAS_API_KEY=cb-...

Notes:

• Voice mode prefers Cartesia for STT and falls back to OpenAI Whisper.
• Command interpretation prefers Cerebras and falls back to Gemini 2.5 Flash.

Scripts

• npm start → Runs index.ts (Browserbase session) to demo Stagehand basics
• npm run voice → Runs enhanced-voice-main.ts for local, voice-controlled automation
• npm run build → TypeScript build (outputs to dist/)

Running the Demo (Browserbase)

npm start

What happens:

• Creates a Browserbase-backed Stagehand session
• Navigates to https://stagehand.dev
• Demonstrates page.extract, page.act, page.observe, and an agent.execute
• Prints a watch URL to view the live session

Running Voice Mode (Local Browser)

npm run voice

What happens:

• Launches Stagehand locally with a visible browser and a persistent user profile (./browser-sessions)
• Listens through your mic for a few seconds, transcribes speech, interprets it, and executes actions
• Say “stop”, “exit”, or “quit” to end the loop

Example commands you can speak:

• “Go to YouTube”
• “Search for funny cat videos”
• “Click the first result”
• “What can I click on this page?”
• “Login to Instagram” / “Logout”

Browser Environments

• Demo (index.ts) runs with env: "BROWSERBASE" and prints a Browserbase session URL.
• Voice mode (enhanced-voice-automation.ts) runs with env: "LOCAL", headless: false, debugDom: true, persistentContext: true, and userDataDir: ./browser-sessions so logins persist across sessions.

File Overview

• index.ts — Minimal Stagehand demo on Browserbase
• enhanced-voice-main.ts — Entry point for voice mode
• enhanced-voice-automation.ts — Voice capture, STT, LLM interpretation, and action execution
• cerebras-config.ts — Cerebras client and structured command interpretation
• tsconfig.json — TypeScript configuration
• package.json — Scripts and dependencies

Permissions and Audio Setup (macOS)

• Grant your terminal microphone access in System Settings → Privacy & Security → Microphone
• Install SoX: brew install sox
• If audio capture fails, quit and restart the terminal after granting permissions

Troubleshooting

• Missing mic input or immediate “Listening...” then “Error”:
  • Ensure SoX is installed and mic permissions are granted
  • Try another terminal app if permissions don’t stick
• STT fails (speech-to-text):
  • Verify CARTESIA_API_KEY; if missing, ensure OPENAI_API_KEY is set for Whisper fallback
• LLM interpretation fails:
  • Verify CEREBRAS_API_KEY; if missing or failing, ensure GOOGLE_API_KEY is set for Gemini fallback
• Browser doesn’t open during voice mode:
  • Voice mode runs locally with a visible browser; check for OS-level blocks or Playwright errors in logs
• Slow actions or flakiness:
  • Voice mode uses debugDom and a long domSettleTimeoutMs for reliability; network-dependent pages may still vary

Security

• API keys are sensitive. Keep .env out of source control.
• The local profile in ./browser-sessions may contain login state. Treat it as sensitive.

License

See repository license if provided. Otherwise, all rights reserved.