An AI-powered video avatar that engages visitors in real-time voice conversations at AWS events. Supports content overlays (architecture diagrams, use cases) via tool calls.
| Requirement | Version | Notes |
|---|---|---|
| Node.js | 18+ | Required for both demos |
| npm | 9+ | Comes with Node.js |
| Python | 3.11+ | Pipecat demo only |
| AWS CLI | 2.x | For deployment and Bedrock access |
| AWS credentials | — | Configured via aws configure or env vars |
API keys from AWS Partners (also available on AWS Marketplace):
| Key | Required for | AWS Partner |
|---|---|---|
TAVUS_API_KEY |
Both demos | Tavus |
DEEPGRAM_API_KEY |
Pipecat demo | Deepgram |
CARTESIA_API_KEY |
Pipecat demo | Cartesia |
DAILY_API_KEY |
Pipecat cloud deploy | Daily |
# Tavus Avatar (managed pipeline)
cd tavus-avatar && cp env.example .env.local
# Edit .env.local with your TAVUS_API_KEY and TAVUS_PERSONA_ID
npm install && npm run dev:all
# Pipecat (self-hosted pipeline)
cd tavus-pipecat-example && cp .env.example .env
# Edit .env with your API keys
pip install -r requirements.txt && npm run dev:allOpen http://localhost:3000.
This repo contains two independent implementations of the same demo:
tavus-avatar/ |
tavus-pipecat-example/ |
|
|---|---|---|
| Approach | Tavus hosted CVI¹ | Self-hosted Pipecat pipeline |
| Frontend | Next.js + Daily.js | React + WebRTC |
| Backend | Next.js API routes → Tavus API | Python Pipecat server |
| Pipeline control | Tavus orchestrates STT²/LLM³/TTS⁴ | You orchestrate everything in Python |
| Pipeline modes | Single (Tavus-managed cascaded) | Cascaded + Amazon Nova Sonic toggle |
| When to choose | Quick demos, minimal setup, managed infrastructure | Custom pipelines, multi-model support, full control |
| Best for | Prototyping, simple use cases | Production, complex workflows, experimentation |
Both share the same tool definitions and content overlays. System prompts are isolated per event in prompts/{EVENT_CONFIG}/.
graph LR
subgraph "Tavus Avatar"
TA[Browser] --> TB[Daily.js] --> TC[Tavus API] --> TD[Managed Pipeline]
end
subgraph "Pipecat Demo"
PA[Browser] --> PB[WebRTC] --> PC[Python Server] --> PD[Custom Pipeline]
PD --> PE[Deepgram STT]
PD --> PF[Bedrock LLM]
PD --> PG[Cartesia TTS]
PD --> PH[Nova Sonic]
end
A Next.js app that delegates pipeline orchestration to Tavus. You configure the persona, STT²/LLM³/TTS⁴ engines, and tools via the Tavus API. Daily.js handles the WebRTC transport.
- Node.js 18+ and npm
- A Tavus account with an API key and configured persona
cd tavus-avatar
npm install
cp env.example .env.localExample .env.local:
TAVUS_API_KEY=your_tavus_api_key_here
TAVUS_PERSONA_ID=your_persona_id_here# All-in-one command
npm run dev:all
# Or step by step
npm run dev
curl -X POST http://localhost:3000/api/persona/setup-toolsOpen http://localhost:3000.
# Check API key and persona
curl -H "x-api-key: $TAVUS_API_KEY" \
"https://tavusapi.com/v2/personas/$TAVUS_PERSONA_ID"Uses a cascaded pipeline orchestrated by Pipecat inside Tavus:
Audio In → Noise Cancellation (local) → VAD⁵ → STT² → LLM³ → TTS⁴ → Audio Out
| Layer | Engine | Role |
|---|---|---|
| Noise Cancellation | Krisp (local macOS) | Filters background noise via a virtual microphone device |
| VAD⁵ | Tavus Sparrow | Turn-taking and interruption detection |
| STT² | tavus-advanced (via Tavus) |
Real-time speech-to-text |
| LLM³ | tavus-gpt-oss (via Tavus) |
Reasoning and response generation |
| TTS⁴ | sonic-3 (Cartesia via Tavus) |
Natural-sounding speech synthesis |
STT²/TTS⁴ engines are managed within Tavus. See Update Tavus Persona for details.
A Python Pipecat server with a React frontend. You control the full pipeline in code. Supports two pipeline modes that visitors can toggle at the kiosk:
- Cascaded: Deepgram STT² → Bedrock Claude → Cartesia TTS⁴ (best-of-breed at each layer)
- Nova Sonic: Amazon Nova 2 Sonic speech-to-speech on Bedrock (single model, 15 languages)
Both modes use Tavus for avatar rendering and Pipecat for orchestration.
- Python 3.11+
- Node.js 18+ and npm
- API keys: Deepgram, Cartesia, Tavus, AWS credentials (for Bedrock)
cd tavus-pipecat-example
# Backend
python -m venv env && source env/bin/activate
pip install -r requirements.txt
cp .env.example .envExample .env:
DEEPGRAM_API_KEY=your_deepgram_key_here
CARTESIA_API_KEY=your_cartesia_key_here
TAVUS_API_KEY=your_tavus_key_here
TAVUS_REPLICA_ID=your_replica_id_here
TAVUS_REPLICA_ID_NOVA_SONIC=your_nova_sonic_replica_id_here
AWS_REGION_NOVA_SONIC=ap-south-1
NOVA_SONIC_VOICE_ID=matthew
CARTESIA_VOICE_ID=79a125e8-cd45-4c13-8a67-188112f4dd22
AWS_ACCESS_KEY_ID=your_aws_key_here
AWS_SECRET_ACCESS_KEY=your_aws_secret_here# All-in-one command
npm run dev:all
# Or step by step
python tavus-pipecat.py --transport webrtc --host 0.0.0.0 --port 7860 &
cd frontend && npm startOpen http://localhost:3000. Click Start demo, choose a pipeline mode, and begin.
# Test API keys
curl -H "Authorization: Token $DEEPGRAM_API_KEY" \
"https://api.deepgram.com/v1/projects"
curl -H "X-API-Key: $CARTESIA_API_KEY" \
"https://api.cartesia.ai/voices"
aws bedrock list-foundation-models --region $AWS_REGION_NOVA_SONICCascaded mode:
Audio In -> Deepgram Nova 3 STT -> Bedrock Claude LLM -> Cartesia TTS -> Tavus Avatar -> Audio/Video Out
Nova Sonic mode:
Audio In -> Amazon Nova 2 Sonic (STT+LLM+TTS) -> Tavus Avatar -> Audio/Video Out
| Variable | Description |
|---|---|
DEEPGRAM_API_KEY |
Deepgram API key (cascaded STT²) |
CARTESIA_API_KEY |
Cartesia API key (cascaded TTS⁴) |
TAVUS_API_KEY |
Tavus API key (avatar) |
TAVUS_REPLICA_ID |
Tavus replica ID for cascaded mode (see Available Avatars) |
TAVUS_REPLICA_ID_NOVA_SONIC |
Tavus replica ID for Nova Sonic mode — use a different avatar to visually distinguish the two modes |
DAILY_API_KEY |
Daily.co API key (required for cloud deployment with Daily transport) |
REACT_APP_TRANSPORT |
WebRTC transport (daily or webrtc). Set automatically by CI/CD — not user-configured. |
AWS_REGION_NOVA_SONIC |
AWS region for Nova Sonic (default: ap-south-1) |
NOVA_SONIC_VOICE_ID |
Nova Sonic voice ID (default: matthew; use arjun for Hindi) |
CARTESIA_VOICE_ID |
Cartesia voice UUID (default: 79a125e8-... British Lady) |
AWS_ACCESS_KEY_ID |
AWS credentials (optional if using default credential chain) |
AWS_SECRET_ACCESS_KEY |
AWS credentials (optional if using default credential chain) |
The Pipecat demo deploys to AWS with a GitHub Actions CI/CD pipeline that triggers only on changes to tavus-pipecat-example/ or prompts/.
Architecture:
- Backend: ECS Fargate (Docker container) behind an ALB
- Frontend: S3 + CloudFront (static React build)
- CloudFront: Serves frontend and proxies
/api/*to the ALB (single HTTPS domain) - Secrets: SSM Parameter Store (SecureString)
- Transport: Daily.js for cloud (reliable WebRTC relay), SmallWebRTC for local dev
graph TD
User([User / Browser])
User -->|HTTPS| CF[CloudFront]
subgraph AWS Cloud
CF -->|Static assets| S3[S3 Bucket\nReact Frontend]
CF -->|/api/* and /start| ALB[Application Load Balancer]
ALB --> ECS[ECS Fargate\nPipecat Bot Container]
SSM[SSM Parameter Store\nSecureString secrets] -->|Secrets at startup| ECS
ECR[ECR\nContainer Registry] -->|Pull image| ECS
end
GHA[GitHub Actions\nCI/CD] -->|Push image| ECR
GHA -->|Sync build| S3
ECS -->|WebRTC relay| Daily[Daily.co]
ECS -->|Speech-to-text| Deepgram[Deepgram STT]
ECS -->|LLM inference| Bedrock[Bedrock Claude\nLLM]
ECS -->|Text-to-speech| Cartesia[Cartesia TTS]
ECS -->|Avatar rendering| Tavus[Tavus Avatar]
ECS -->|Speech-to-speech| NovaSonic[Bedrock Nova Sonic\nSpeech-to-Speech]
aws ssm put-parameter --name /tavus-pipecat/DEEPGRAM_API_KEY --type SecureString --value "your-key"
aws ssm put-parameter --name /tavus-pipecat/CARTESIA_API_KEY --type SecureString --value "your-key"
aws ssm put-parameter --name /tavus-pipecat/TAVUS_API_KEY --type SecureString --value "your-key"
aws ssm put-parameter --name /tavus-pipecat/TAVUS_REPLICA_ID --type SecureString --value "your-replica-id"
aws ssm put-parameter --name /tavus-pipecat/TAVUS_REPLICA_ID_NOVA_SONIC --type SecureString --value "your-replica-id"
aws ssm put-parameter --name /tavus-pipecat/DAILY_API_KEY --type SecureString --value "your-daily-api-key"aws cloudformation deploy \
--template-file tavus-pipecat-example/infra/template.yaml \
--stack-name tavus-pipecat \
--parameter-overrides \
VpcId=vpc-xxx \
PublicSubnetIds=subnet-aaa,subnet-bbb \
--capabilities CAPABILITY_NAMED_IAMNote the outputs — you'll need them for GitHub Actions secrets.
Create a GitHub OIDC identity provider and IAM role so GitHub Actions can assume credentials without static keys:
# Create OIDC provider (one-time per account)
aws iam create-open-id-connect-provider \
--url https://token.actions.githubusercontent.com \
--client-id-list sts.amazonaws.com \
--thumbprint-list 6938fd4d98bab03faadb97b34396831e3780aea1
# Create IAM role with trust policy for your repo, then attach
# ECR, ECS, S3, and CloudFront permissions (see infra/template.yaml)| Secret | Value (from CloudFormation outputs) |
|---|---|
AWS_ROLE_ARN |
IAM role ARN for GitHub Actions OIDC |
S3_BUCKET_NAME |
FrontendBucketName output |
CLOUDFRONT_DISTRIBUTION_ID |
CloudFrontDistributionId output |
Also set these repository variables (Settings → Variables → Actions) to configure the event and voices per deployment:
| Variable | Example (Mumbai) | Example (Sydney) |
|---|---|---|
AWS_REGION |
ap-south-1 |
ap-southeast-2 |
EVENT_CONFIG |
aws-summit-mumbai-2026 |
aws-summit-sydney-2026 |
NOVA_SONIC_VOICE_ID |
matthew |
matthew |
CARTESIA_VOICE_ID |
79a125e8-cd45-4c13-8a67-188112f4dd22 |
79a125e8-cd45-4c13-8a67-188112f4dd22 |
The workflow at .github/workflows/deploy-pipecat.yml runs automatically on push to main when files in tavus-pipecat-example/ or prompts/ change. It:
- Builds the Docker image and pushes to ECR
- Forces a new ECS Fargate deployment and waits for stability
- Builds the React frontend (with
REACT_APP_API_URL=''for same-origin CloudFront) - Syncs to S3 and invalidates the CloudFront cache
The frontend and backend deploy in parallel. Access the demo at the CloudFront domain (CloudFrontDomainName output).
Note: Regenerate
frontend/package-lock.jsonwith Node 20 (fnm use 20 && npm install) to match CI. The first CloudFormation deploy is slow (~15 min) as the ECS service waits for a Docker image — subsequent deploys via CI/CD are faster.
Supported Browsers:
- Chrome 88+ (recommended)
- Firefox 85+
- Safari 14.1+
- Edge 88+
WebRTC Requirements:
- Secure context (HTTPS or localhost)
- Microphone permissions
- WebRTC support (enabled by default in modern browsers)
Known Issues:
- Safari may require user gesture before audio playback
- Firefox requires
media.navigator.permission.disabled=truefor localhost development
Latency Optimization:
- Tavus Avatar: ~200-400ms end-to-end latency (managed pipeline)
- Pipecat Cascaded: ~300-500ms (Deepgram + Bedrock + Cartesia)
- Nova Sonic: ~400-600ms (single model, higher quality)
Concurrent Users:
- Local Development: 1-2 concurrent sessions
- AWS Deployment: 10-50 concurrent sessions (auto-scaling ECS)
- Resource Usage: ~1GB RAM, 0.5 CPU per active session
Optimization Tips:
- Use Daily.js transport for production (better WebRTC handling)
- Enable Krisp noise cancellation for cleaner audio input
- Monitor ECS CPU/memory metrics for scaling decisions
API Key Management:
- Store keys in environment variables, never in code
- Use AWS SSM Parameter Store (SecureString) for production
- Rotate keys regularly and monitor usage
CORS Configuration:
// Next.js API routes
const corsHeaders = {
'Access-Control-Allow-Origin': process.env.NODE_ENV === 'production'
? 'https://your-domain.com'
: '*',
'Access-Control-Allow-Methods': 'GET, POST, OPTIONS',
'Access-Control-Allow-Headers': 'Content-Type, Authorization'
}Rate Limiting:
- Implement per-IP session limits
- Use AWS WAF for DDoS protection
- Monitor API usage and set billing alerts
Chrome/Edge:
- Click the microphone icon in the address bar
- Select "Always allow" for the site
- Refresh the page
Firefox:
- Click the shield icon → Permissions → Microphone
- Select "Allow" and check "Remember this decision"
Safari:
- Safari → Settings → Websites → Microphone
- Set your site to "Allow"
# Tavus
curl -H "x-api-key: $TAVUS_API_KEY" "https://tavusapi.com/v2/personas"
# Deepgram
curl -H "Authorization: Token $DEEPGRAM_API_KEY" "https://api.deepgram.com/v1/projects"
# Cartesia
curl -H "X-API-Key: $CARTESIA_API_KEY" "https://api.cartesia.ai/voices"
# AWS Bedrock
aws bedrock list-foundation-models --region us-east-1Symptoms: No audio/video, connection timeouts Solutions:
- Check browser console for WebRTC errors
- Verify HTTPS/localhost secure context
- Test with different networks (corporate firewalls may block WebRTC)
- Use Daily.js transport for better NAT traversal
Symptoms: Network errors, blocked requests Solutions:
- Check browser console for CORS errors
- Verify API endpoints allow your domain
- For local development, use
--disable-web-securityflag (Chrome only) - Ensure proper CORS headers in API responses
Logging:
# Pipecat server logs
python tavus-pipecat.py --log-level DEBUG
# Browser console
# Check Network tab for API calls
# Check Console for WebRTC errorsHealth Checks:
# Tavus Avatar
curl http://localhost:3000/api/health
# Pipecat
curl http://localhost:7860/healthAWS CloudWatch:
- ECS service metrics (CPU, memory, task count)
- ALB target health and response times
- CloudFront cache hit rates and errors
For tavus-avatar only. The persona's system prompt, tools, and engines are managed via the Tavus API.
# Set EVENT_CONFIG to match your event (e.g. aws-summit-mumbai-2026)
SYSTEM_PROMPT=$(python3 -c "import json; print(json.dumps(open('prompts/$EVENT_CONFIG/system-instruction.md').read()))")
curl -X PATCH "https://tavusapi.com/v2/personas/$TAVUS_PERSONA_ID" \
-H "x-api-key: $TAVUS_API_KEY" \
-H "Content-Type: application/json" \
-d "[{\"op\": \"replace\", \"path\": \"/system_prompt\", \"value\": $SYSTEM_PROMPT}]"curl -X POST "http://localhost:3000/api/persona/setup-tools"curl -X PATCH "https://tavusapi.com/v2/personas/$TAVUS_PERSONA_ID" \
-H "x-api-key: $TAVUS_API_KEY" \
-H "Content-Type: application/json" \
-d '[{"op": "replace", "path": "/layers/stt/stt_engine", "value": "tavus-advanced"}]'The agent-kiosk-shell/ directory contains a generic Electron wrapper for running either demo in a borderless full-screen kiosk window. It appends autostart=1&shell=electron so sessions start immediately.
cd agent-kiosk-shell
npm install
cp .env.example .env
# Local dev
npm start -- --target-url=http://localhost:3000
# Deployed app
npm start -- --target-url=https://your-app.vercel.app --display-number=2Keyboard shortcuts:
Cmd+R— ReloadCmd+B— Show disconnected screenCmd+Q— Quit
Noise cancellation runs locally on the kiosk Mac using Krisp, not inside the pipeline:
- Install the Krisp desktop app.
- Enable Noise Cancellation for the microphone.
- In macOS System Settings → Sound → Input, select Krisp Microphone.
- The browser uses the Krisp virtual mic automatically.
Both implementations load the system prompt from prompts/{EVENT_CONFIG}/system-instruction.md. Set EVENT_CONFIG to the event directory name (e.g. aws-summit-mumbai-2026) to switch event context — prompts, greetings, voice IDs, and content items are all isolated per event. Copy prompts/_template/ to add a new event.
Both implementations support three tools:
- show_content — Display content overlays (architecture diagram, voice AI overview, use cases)
- show_schedule — Show summit schedule as markdown tables
- dismiss_content — Return to full-screen avatar view
Static HTML content pages are served from tavus-pipecat-example/frontend/public/content/ and tavus-avatar/src/app/content/.
- Click Start demo on the landing page.
- Grant microphone permissions when prompted.
- Speak with the AI avatar in real-time.
- Ask the avatar to show content (e.g., "show me the architecture diagram").
- Press End or Escape to end the session.
Keyboard shortcuts:
Ctrl+D— Toggle microphone muteCtrl+F— Cycle microphone devices (tavus-avatar only)Ctrl+G— Cycle speaker devices (tavus-avatar only)
Tavus provides stock replicas usable without custom recordings. Set TAVUS_REPLICA_ID and TAVUS_REPLICA_ID_NOVA_SONIC in SSM Parameter Store to change the avatar per deployment.
Female replicas:
| Name | Replica ID | Setting |
|---|---|---|
| Olivia | rc2146c13e81 |
Neutral background |
| Olivia - Office | r9fa0878977a |
Office setting |
| Anna | r6ae5b6efc9d |
Neutral background |
| Anna - Office | r4dcf31b60e1 |
Office setting |
| Julia | rdc96ac37313 |
Neutral background |
| Julia - Home Office | rb43357fb2ee |
Home office |
| Gloria - Conversational | rbe2c395e725 |
Conversational style |
| Rose | r1af76e94d00 |
Neutral background |
| Luna | r9d30b0e55ac |
Neutral background |
| Helen - Casual | r12d3eb75ec2 |
Casual setting |
| Lucy - Studio | r53a461095cf |
Studio setting |
| Samantha - Office V2 | r38e4c3bc562 |
Office setting |
| Gabby | rdf61be0d4e1 |
Neutral background |
| Ruby | re3a705cf66a |
Neutral background |
| Destiny | r38a383b0173 |
Neutral background |
| Celine - Studio | r4f5b5ef55c8 |
Studio setting |
Male replicas:
| Name | Replica ID | Setting |
|---|---|---|
| Charlie | rf4703150052 |
Neutral background |
| Benjamin | r1a4e22fa0d9 |
Neutral background |
| Raj | ra066ab28864 |
Neutral background |
| Raj - Doctor | r18e9aebdc33 |
Doctor setting |
| Daniel - Office | r72f7f7f7c8b |
Office setting |
| Owen | r9458111c64a |
Neutral background |
| Carter | rca8a38779a8 |
Neutral background |
| Liam | r90a0339d496 |
Neutral background |
| Diego - Office V2 | r044d76f4490 |
Office setting |
| Danny | r62baeccd777 |
Neutral background |
| James | r92debe21318 |
Neutral background |
| Nathan - Bookshelf | rfe12d8b9597 |
Bookshelf setting |
| Lucas - Studio | r5f0577fc829 |
Studio setting |
| Victor - Casual | r1d7cf9edbb4 |
Casual setting |
| Kai | r31e11adf1d3 |
Neutral background |
| Zane | r24efb3b9bef |
Neutral background |
To update the avatar for a deployment:
# Example: switch Sydney Nova Sonic to Olivia - Office
aws ssm put-parameter \
--name /tavus-pipecat/TAVUS_REPLICA_ID_NOVA_SONIC \
--type SecureString --value "r9fa0878977a" --overwrite \
--region ap-southeast-2Current per-deployment config:
| Deployment | Pipeline | Avatar | Voice |
|---|---|---|---|
Sydney (ap-southeast-2) |
Cascaded | Olivia (rc2146c13e81) |
Cartesia British Lady |
Sydney (ap-southeast-2) |
Nova Sonic | Olivia - Office (r9fa0878977a) |
olivia (Australian English) |
Bengaluru (ap-south-1) |
Cascaded | (set in SSM) | Cartesia Hindi |
Bengaluru (ap-south-1) |
Nova Sonic | (set in SSM) | arjun (Hindi) |
Available Nova Sonic voices: matthew (polyglot), tiffany (polyglot), amy (British English), olivia (Australian English), ambre (French), beatrice (Italian), arjun (Hindi), and others. See Amazon Nova Sonic docs for the full list.
This demo promotes the Guidance for Voice Agents on AWS reference architecture. Ask the avatar "show me the voice agent guidance" to display it.
- CVI¹ — Conversational Video Interface: Real-time video avatar with voice interaction
- STT² — Speech-to-Text: Converts spoken audio to text transcription
- LLM³ — Large Language Model: AI model for understanding and generating text responses
- TTS⁴ — Text-to-Speech: Converts text responses to natural-sounding speech
- VAD⁵ — Voice Activity Detection: Detects when user starts/stops speaking for turn-taking
prompts/ # Shared system prompts and knowledge base documents
agent-kiosk-shell/ # Generic Electron kiosk wrapper
tavus-avatar/ # Tavus hosted CVI (Next.js)
src/
app/
api/conversation/ # Create/end Tavus conversation sessions
api/persona/setup-tools/ # Patch persona tool definitions
content/ # Embeddable overlay pages
components/ # React components
lib/ # Tavus API client, persona/tool configs
types/ # TypeScript type definitions
tavus-pipecat-example/ # Self-hosted Pipecat pipeline (Python + React)
tavus-pipecat.py # Pipecat bot (dual pipeline: cascaded + Nova Sonic)
requirements.txt # Python dependencies
frontend/ # React frontend
src/
components/ # VideoConversation UI
webrtc-client.js # WebRTC client
public/
content/ # Static content overlay pages
See IMPROVEMENTS.md for tracked enhancements.
- tavus-avatar: Next.js 16 / React 19 / TypeScript / Tailwind CSS v4 / Daily.js / Tavus API
- tavus-pipecat-example: Python / Pipecat / Deepgram / Cartesia / Amazon Bedrock / Amazon Nova Sonic / Tavus / React
- Shared: Tavus avatar rendering, Pipecat orchestration, AWS infrastructure