Interruptible Voice Agent

A portfolio project demonstrating a real-time voice agent with configurable turn-taking, mid-response interruption, retrieval-augmented generation, and live observability.

The browser captures microphone audio over Daily WebRTC. A Pipecat pipeline transcribes speech with Deepgram, generates a response with OpenAI, synthesizes audio with Cartesia, and streams it back to the browser. Bot state, latency, and interruption events are sent to the dashboard over Daily's data channel.

Highlights

• Real-time browser voice sessions over WebRTC
• Configurable system prompt, LLM settings, voice, speech speed, and interruptibility
• Explicit LISTENING → THINKING → SPEAKING state machine
• End-of-user-speech to first-bot-audio latency measurement
• Mid-response barge-in and interruption tracking
• Qdrant-backed help-center RAG with OpenAI embeddings
• Fail-open retrieval, bounded provider retries, rate limiting, structured logs, and graceful idle-session shutdown
• Production-mode Docker images with health checks and non-root users
• Deterministic backend and frontend tests for the core conversation behavior

Interruption as a conversation design problem

Interruption is treated as part of the agent's social behavior, not as a simple mute button. A useful voice agent must react when the user genuinely wants to take the floor without stopping because of every breath, keyboard sound, or short acknowledgment.

The interruptibility control expresses that tradeoff in human terms:

• At the low end, the agent holds the floor and waits for clearer evidence that the user is beginning a real turn.
• At the high end, the agent yields quickly and feels more conversational, but becomes more sensitive to accidental speech and background audio.
• At zero, the agent is intentionally allowed to finish its response.

Once an interruption is accepted, the current response is abandoned, playback stops, and the agent returns to listening. This works both while the answer is being prepared and while it is being spoken, so the interface does not force the user to wait for generated audio to finish. Every accepted interruption is also reflected in the dashboard, making the behavior visible rather than hidden inside the audio pipeline.

Architecture

Browser microphone
      │
      ▼
Daily WebRTC transport
      │
      ▼
Silero VAD → Deepgram STT → context + Qdrant retrieval → OpenAI LLM
                                                        │
                                                        ▼
Daily speaker ← Cartesia TTS ← output guard ←───────────┘

StateTracker ──→ state / latency / interruption events ──→ browser dashboard

The frontend never receives provider API keys. It calls POST /session, and the backend creates a private Daily room, returns a user token, and starts one asynchronous Pipecat task for that session.

Technology

• Frontend: Next.js App Router, TypeScript, Daily.js, TanStack Query
• Backend: Python 3.11, FastAPI, Pipecat, Pydantic, structlog
• Voice: Deepgram STT, OpenAI LLM, Cartesia TTS
• Realtime: Daily WebRTC and data channel
• Retrieval: Qdrant and OpenAI embeddings
• Runtime: Docker Compose

Launch locally

Option A: Preview the interface without API keys

This mode renders the complete portfolio interface but does not start a voice session:

git clone https://github.com/azizayan/vagent.git
cd vagent/frontend
npm ci
NEXT_PUBLIC_API_URL=/api npm run dev

Open http://localhost:3000. The configuration and telemetry interface will render normally; starting a live voice session still requires the provider credentials listed below. Stop the preview with Ctrl+C.

Option B: Run the complete voice stack

Prerequisites

• Docker Desktop or Docker Engine with Compose
• API credentials for OpenAI, Deepgram, Cartesia, and Daily
• A Cartesia voice ID

1. Configure the environment

git clone https://github.com/azizayan/vagent.git
cd vagent
cp .env.example .env

Open .env and set at least:

OPENAI_API_KEY=your_openai_key
DEEPGRAM_API_KEY=your_deepgram_key
CARTESIA_API_KEY=your_cartesia_key
DAILY_API_KEY=your_daily_key

These credentials remain in your untracked local .env; they are never sent to the frontend bundle or committed to the repository.

Keep NEXT_PUBLIC_API_URL=/api for the default same-origin local setup. Qdrant runs inside Docker Compose and does not require a separate account. The session form includes built-in Cartesia voice IDs and also accepts a custom voice ID.

2. Build and start the stack

docker compose up -d --build

Wait until all three services are healthy:

docker compose ps
curl http://localhost:3000/api/health

The health response should contain "status":"ok".

3. Open the application

Open http://localhost:3000, allow microphone access, choose a voice, and start a session. Browsers treat localhost as a secure context for microphone access.

To exercise the RAG path, ask:

What is the return window?

The fictional Northstar Store knowledge base contains a deliberately specific 37-day return policy.

4. View logs or stop the stack

docker compose logs -f backend frontend qdrant
docker compose down

Use docker compose down -v only when you also want to delete the local Qdrant data volume and force the knowledge base to be re-indexed.

Run tests locally

Backend:

cd backend
uv sync --extra dev
uv run pytest
uv run ruff check .
uv run mypy app

Frontend:

cd frontend
npm ci
npm test -- --runInBand
npm run typecheck
NEXT_PUBLIC_API_URL=/api npm run build

Repository structure

backend/
├── app/
│   ├── api/                 HTTP routes and rate limiting
│   ├── core/                Settings, logging, and domain errors
│   ├── data/                Fictional help-center dataset
│   ├── pipeline/            VAD, prompts, idle handling, and processors
│   ├── schemas/             HTTP and data-channel contracts
│   ├── services/            Daily, agent lifecycle, and Qdrant/OpenAI RAG
│   ├── bot.py               Pipecat pipeline composition root
│   └── main.py              FastAPI application and lifespan
└── tests/                   Backend unit and contract tests

frontend/
├── app/                     Next.js dashboard and configuration UI
├── components/              Reusable configuration controls
├── hooks/                   Daily data-channel state
├── lib/                     API, Daily, environment, and validation helpers
├── tests/                   UI and reducer tests
└── types/                   Frontend copy of the backend contract

docker-compose.yml           Frontend, backend, and Qdrant stack
.env.example                 Environment variable template
DEPLOY.md                    Optional EC2 and HTTPS tunnel notes

Implementation notes

The conversation design above is implemented through coordinated speech detection, recognized-word gating, response cancellation, and shared state events. The same turn boundaries drive the dashboard and latency measurement, so the visual state follows the behavior heard by the user.

The requested stt_temperature field remains in the session contract but is not sent to Deepgram because its streaming STT API does not expose temperature. The tts_temperature control is mapped to Cartesia's supported emotion setting: low values select neutral delivery, high values select excited delivery, and the middle range uses the voice default.

The help-center retriever requests the top three Qdrant matches and inserts them into a cloned LLM context immediately before the latest user message. Retrieval failures pass the original context through, so a vector-store outage does not block the conversation pipeline.

Deployment

The same Docker Compose stack can run on a small Linux host. Remote microphone access requires HTTPS; see DEPLOY.md for a generic EC2 and ngrok example. Never commit the populated .env file.