Palabra - Real-Time Translation for Agora Video Conferencing

Real-time speech translation with optional lip-synced avatar integration for Agora App Builder.

Features

• Translate Audio - Real-time speech translation to 9+ languages
• Avatar Mode - Lip-synced avatar speaks the translated audio
• Persistent Avatar - Avatar can run independently, switching between original and translated audio

Repository Structure

This repository contains customization files that overlay onto Agora App Builder:

palabra/
├── client/                          # Frontend customization (overlay files)
│   └── customization/
│       ├── index.tsx                # Entry point - wraps VideoCall with TranslationProvider
│       └── palabra/
│           ├── TranslationProvider.tsx   # Core translation/avatar logic
│           └── TranslationMenuItem.tsx   # Menu UI components
├── server/                          # Go backend (runs in Docker)
├── docs/                            # Documentation
└── app-builder/                     # GITIGNORED - App Builder cloned here at build time

How customization works:

1. Agora App Builder is a standalone video conferencing app
2. App Builder supports a customization/ folder for extending functionality
3. This repo provides customization files that add Palabra translation features
4. At build time, files from client/customization/ are copied into App Builder's template/customization/
5. The index.tsx entry point wraps the VideoCall component with TranslationProvider

App Builder source: https://github.com/AgoraIO-Community/app-builder-core

Quick Start

Prerequisites

• Ubuntu 20.04+ (x86-64 architecture required)
• Docker and Docker Compose
• Node.js 18+
• Nginx
• Domain name with SSL certificate

1. Clone Repository

cd /home/ubuntu
git clone https://github.com/BenWeekes/palabra.git
cd palabra

2. Configure Backend

cd server
cp .env.example .env
nano .env

Required .env settings:

# Agora Credentials
APP_ID=<your_agora_app_id>
APP_CERTIFICATE=<your_agora_certificate>
CUSTOMER_ID=<your_customer_id>
CUSTOMER_CERTIFICATE=<your_customer_certificate>

# Palabra Credentials
PALABRA_CLIENT_ID=<your_palabra_client_id>
PALABRA_CLIENT_SECRET=<your_palabra_client_secret>

# Database
POSTGRES_USER=appbuilder
POSTGRES_PASSWORD=<strong_password>
POSTGRES_DB=appbuilder

# Server Configuration
PORT=8080
SCHEME=https
ALLOWED_ORIGIN=https://yourdomain.com

# Avatar Mode (set to true to enable Anam avatar)
ENABLE_ANAM=true
ANAM_API_KEY=<your_anam_api_key>
ANAM_BASE_URL=https://api.anam.ai/v1
ANAM_AVATAR_ID=<your_anam_avatar_id>
ANAM_QUALITY=high
ANAM_VIDEO_ENCODING=H264

# Session Protection
PALABRA_SESSION_TIMEOUT_MINUTES=10
PALABRA_IDLE_TIMEOUT_SECONDS=60

Start backend:

sudo docker compose up -d --build

# Verify it's running
curl http://localhost:7080/v1/palabra/tasks
# Should return: {"success":true,"tasks":[]}

3. Build Frontend

cd /home/ubuntu/palabra

# Clone Agora App Builder
git clone https://github.com/AgoraIO-Community/app-builder-core.git app-builder
cd app-builder

# Copy customization files
cp -r ../client/customization/palabra template/customization/
cp ../client/customization/index.tsx template/customization/

# Apply patch to filter translation UIDs from video tiles (REQUIRED)
cp ../client/app-builder-patches/VideoComponent.tsx template/src/pages/video-call/

# Copy config files
cp config.json template/config.json
cp theme.json template/theme.json

# Update config.json with your domain
nano template/config.json

Update these values in template/config.json:

{
  "APP_ID": "<your_agora_app_id>",
  "FRONTEND_ENDPOINT": "https://yourdomain.com",
  "BACKEND_ENDPOINT": "https://yourdomain.com",
  "PALABRA_BACKEND_ENDPOINT": "https://yourdomain.com"
}

Build:

# Install UI Kit
npm run uikit

# Install dependencies
cd template
npm install --legacy-peer-deps
cd ..

# Build for production
npm run web-build

4. Deploy Frontend

sudo mkdir -p /var/www/palabra
sudo cp -r Builds/web/* /var/www/palabra/
sudo chown -R www-data:www-data /var/www/palabra

5. Configure Nginx

sudo nano /etc/nginx/sites-available/palabra

server {
    listen 443 ssl;
    listen [::]:443 ssl;

    ssl_certificate /etc/letsencrypt/live/yourdomain.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/yourdomain.com/privkey.pem;

    # API proxy to backend (Docker bound to localhost only)
    location /v1/ {
        proxy_pass http://localhost:7080/v1/;
        proxy_http_version 1.1;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_read_timeout 300s;
    }

    location /query {
        proxy_pass http://localhost:7080/query;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
    }

    # Frontend static files
    location / {
        root /var/www/palabra;
        index index.html;
        try_files $uri $uri/ /index.html;
    }

    location ~* \.(js|css|png|jpg|jpeg|gif|ico|wasm|mp4|ttf)$ {
        root /var/www/palabra;
        expires 1y;
        add_header Cache-Control "public, immutable";
    }
}

Nginx uses URL path to route requests: /v1/*, /query, /oauth, /pstn are proxied to the backend Docker container on localhost:7080, while everything else serves the frontend static files from /var/www/palabra/.

Enable site:

# Remove default site if it occupies port 443
sudo rm -f /etc/nginx/sites-enabled/default

sudo ln -s /etc/nginx/sites-available/palabra /etc/nginx/sites-enabled/
sudo nginx -t
sudo systemctl reload nginx

6. Get SSL Certificate

sudo certbot --nginx -d yourdomain.com

7. Verify Deployment

# Test backend API
curl https://yourdomain.com/v1/palabra/tasks

# Test avatar endpoints
curl -X POST https://yourdomain.com/v1/avatar/start \
  -H "Content-Type: application/json" \
  -d '{"channel":"test","sourceUid":"123"}'

Usage

1. Join a video call with 2+ participants
2. Click the 3-dot menu on a remote participant's video
3. Select "Start Avatar" to show lip-synced avatar
4. Select "Translate Audio" and choose target language
5. Avatar will speak the translated audio

API Endpoints

Endpoint	Method	Description
/v1/palabra/start	POST	Start translation for a user
/v1/palabra/stop	POST	Stop translation
/v1/palabra/tasks	GET	List active translation tasks
/v1/avatar/start	POST	Start persistent avatar
/v1/avatar/stop	POST	Stop persistent avatar

Operating Modes

Translation Modes

Mode	Config	Description
Audio-Only	ENABLE_ANAM=false	Translation audio only (UID 3000+)
Avatar	ENABLE_ANAM=true	Lip-synced avatar video+audio (UID 4000+)

Avatar Modes

Mode	Flow	Description
Non-Persistent	Call /v1/palabra/start directly	Each translation creates its own avatar. Bot subscribes to Palabra UID.
Persistent	Call /v1/avatar/start first, then /v1/palabra/start	Avatar stays active, seamlessly switches between original and translated audio.

Persistent Avatar Benefits:

• Avatar appears immediately (no wait for translation to start)
• Seamless audio: original audio plays until translation kicks in (no silence gap)
• Fast switching: stop translation returns to original audio instantly

UID Ranges

Range	Purpose
1-2999	Regular users
3000-3999	Palabra translation streams
4000-4999	Anam avatar streams
5000+	Backend bot workers

The frontend VideoComponent.tsx patch filters UIDs 3000-4999 from video tiles so translation/avatar streams don't appear as separate participants.

Updating

cd /home/ubuntu/palabra
git pull

# Rebuild backend
cd server
sudo docker compose down
sudo docker compose up -d --build

# Rebuild frontend
cd ../app-builder
cp -r ../client/customization/palabra template/customization/
cp ../client/customization/index.tsx template/customization/
cp ../client/app-builder-patches/VideoComponent.tsx template/src/pages/video-call/
npm run web-build
sudo cp -r Builds/web/* /var/www/palabra/

Troubleshooting

Backend not responding:

sudo docker logs server --tail 50

Avatar 401 Unauthorized:

• Verify ANAM_API_KEY in .env is correct
• Recreate container after .env changes: sudo docker compose down && sudo docker compose up -d

Frontend not loading customization:

• Ensure template/customization/index.tsx exists
• Rebuild: npm run web-build

CORS errors:

• Verify ALLOWED_ORIGIN in .env matches your domain
• Use same-origin setup (frontend and API on same port via nginx)

Architecture

┌─────────────────┐     ┌──────────────────┐     ┌─────────────────┐
│   Browser       │────▶│   Nginx :443     │────▶│  Backend :7080  │
│   (Frontend)    │     │   (SSL + Proxy)  │     │  (localhost only)│
└─────────────────┘     └──────────────────┘     └─────────────────┘
                                                          │
                                                          ▼
                                                 ┌─────────────────┐
                                                 │   Palabra API   │
                                                 │   Anam API      │
                                                 │   Agora RTC     │
                                                 └─────────────────┘

How nginx routes requests: Nginx listens on port 443 (standard HTTPS) and uses URL path to decide where to send each request. API paths (/v1/*, /query) are proxied to the Go backend on localhost:7080. All other paths serve the frontend static files from /var/www/palabra/. The Docker ports for the backend and database are bound to 127.0.0.1 only, so they are not accessible from the internet.

Production Hardening

The following settings are applied for production:

Setting	File	Value
CORS debug	server/cmd/video_conferencing/server.go	Debug: false
Log level	server/config.json	LOG_LEVEL: "WARN"
Docker ports	server/docker-compose.yml	Bound to 127.0.0.1 only
Nginx port	/etc/nginx/sites-available/palabra	443 (standard HTTPS)

Documentation

• palabra-integrate.md - Detailed integration guide
• anam-integrate.md - Anam avatar setup
• app-builder-dev.md - Development guide

License

Copyright © 2021 Agora Lab, Inc.