Aria Voice AI 📞🤖

An AI phone receptionist that answers real phone calls, holds a natural back-and-forth voice conversation in real time, and answers questions from a business knowledge base using RAG.

Call in → Aria picks up → you talk → Aria understands, retrieves relevant knowledge, and replies in a natural voice — all within a live streaming loop.

How it works

 Caller
   │  (phone call)
   ▼
Twilio ──────────► FastAPI webhook  ── opens ──►  Media Stream (WebSocket)
                                                        │
                        ┌───────────────────────────────┼───────────────────────────────┐
                        ▼                                ▼                                ▼
                  Deepgram (STT)                  Claude (reasoning)               ElevenLabs (TTS)
              speech → text in real time     + RAG over knowledge base        text → natural voice
                                                        │
                                                        ▼
                                              Supabase (Postgres + pgvector)
                                        conversations · knowledge · call state

1. Twilio receives the call and opens a bidirectional media stream to the FastAPI backend.
2. Deepgram transcribes the caller's speech to text in real time.
3. Claude generates the response, grounded in relevant documents retrieved from the knowledge base (RAG).
4. ElevenLabs converts the reply to natural speech, streamed back to the caller.
5. Supabase stores conversations, call state, and the embedded knowledge base.

Tech stack

Layer	Technology
API / orchestration	FastAPI, WebSockets, Pydantic
Telephony	Twilio Voice + Media Streams
Speech-to-text	Deepgram
Reasoning	Anthropic Claude
Text-to-speech	ElevenLabs
Retrieval (RAG)	Embeddings + vector search
Database	Supabase (Postgres)
Ingestion	pypdf, BeautifulSoup

Features

• 🗣️ Real-time voice conversations over an actual phone line via Twilio media streams
• 📚 RAG knowledge base — upload PDFs / web pages; answers are grounded in your own content
• 🧾 Conversation logging — every call transcribed and stored
• 📊 Stats & settings APIs — call metrics and runtime configuration
• 🔌 Clean modular structure — routers, services, and prompts separated

Project structure

main.py                 FastAPI app + router registration
routers/                twilio_webhook, twilio_stream, conversations, knowledge, stats, settings
services/               claude_client, deepgram_client, elevenlabs_tts, rag, embeddings, db
prompts/                system prompt for the voice agent
models/                 call state models
database/schema.sql     Supabase schema

Getting started

# 1. Install dependencies
pip install -r requirements.txt

# 2. Configure credentials
cp .env.example .env
# Fill in: ANTHROPIC_API_KEY, DEEPGRAM_API_KEY, ELEVENLABS_API_KEY,
#          TWILIO_* , SUPABASE_URL, SUPABASE_KEY

# 3. Run the API
uvicorn main:app --reload

# 4. Expose locally for Twilio (e.g. ngrok) and point your
#    Twilio number's voice webhook at /twilio/webhook

Companion app: aria-dashboard — a Next.js dashboard to manage the knowledge base, review call transcripts, and view stats.

---

Built by Vinit Kumar — AI Engineer, Melbourne.