NexCast

Sub-100ms AI esports commentary as a Chrome extension — Python-free, fully self-hosted, ElevenLabs-quality voice on a single RTX GPU.

---

Architecture

┌──────────────────────────────────────────────────────────────────────┐
│  Chrome Extension (MV3)                                              │
│  ├─ content.ts    — DOM scrape: game title, streamer, last 8 chat   │
│  ├─ background.ts — WebSocket, JWT, stores latest game state,        │
│  │                  triggers next request on audio_done signal       │
│  └─ offscreen.ts  — AudioContext, i16 LE PCM → gapless playback,    │
│                     fires audio_done when queue drains               │
└────────────────────────────┬─────────────────────────────────────────┘
                             │  ws://api.nexcast.club/ws?token={jwt}
                             │  ↓ TEXT  game_event JSON (event-driven)
                             │  ↑ BINARY  i16 LE PCM chunks (24 kHz)
┌────────────────────────────▼─────────────────────────────────────────┐
│  Microservice 1 — API Gateway (Go + Gin)          :8080              │
│  ├─ JWT validation (query param or Bearer header)                    │
│  ├─ POST /auth/login  →  upsert user via state-engine gRPC           │
│  ├─ GET  /me, /me/sessions, PUT /me/preferences  (REST)              │
│  └─ GET  /ws  →  quota check → StreamCommentary gRPC → pipe PCM     │
└──────┬──────────────────────────────────┬────────────────────────────┘
       │ gRPC :9091                        │ gRPC :9092
┌──────▼──────────────────┐   ┌───────────▼───────────────────────────┐
│  Microservice 2          │   │  Microservice 3 — AI Worker           │
│  State & Billing Engine  │   │  (Rust + Tonic + ORT)  :9092         │
│  (Rust + Axum + sqlx)   │   │  ├─ HTTP SSE → llama-server-llm       │
│  :9091                   │   │  │   Llama-3.2-1B → sentences         │
│  ├─ PostgreSQL pool      │   │  ├─ HTTP SSE → llama-server-orpheus   │
│  ├─ users / subscriptions│   │  │   Orpheus-3B → audio tokens        │
│  ├─ quota tracking       │   │  └─ SNAC ONNX decode → i16 LE PCM    │
│  └─ migrations on boot   │   └──────────┬────────────────────────────┘
└─────────────┬────────────┘              │ HTTP :8081 / :8082
              │                ┌──────────▼────────────────────────────┐
       ┌──────▼──────┐         │  Infrastructure (same host, [ai])     │
       │  AWS RDS     │         │  ├─ llama-server-llm   :8081          │
       │  PostgreSQL  │         │  │   Llama-3.2-1B-Instruct-Q8_0      │
       └─────────────┘         │  └─ llama-server-orpheus :8082        │
                               │      Orpheus-3B-Q4_K_M (neural TTS)  │
                               │  (GPU passthrough, internal network)  │
                               └───────────────────────────────────────┘

  Web Portal: https://nexcast.club  (React + Vite, nginx)
  Observability: Prometheus · Grafana · Loki (self-hosted, EKS)

---

Stack

Service	Language / Image	Local Port	Role
api-gateway	Go 1.25 + Gin	8080	WebSocket ingress, JWT, REST
state-engine	Rust 1.88 + Axum	9091	User state, quota, billing (gRPC)
ai-worker	Rust 1.88 + ORT	9092	gRPC server: orchestrates LLM→TTS pipeline
llama-server-llm	llama.cpp (CUDA)	8081	Llama-3.2-1B text generation (internal)
llama-server-orpheus	llama.cpp (CUDA)	8082	Orpheus-3B neural TTS token gen (internal)
web	React + Vite + nginx	3000	nexcast.club landing page
postgres	PostgreSQL 16	5432	Persistent state (→ RDS in prod)
prometheus	Prometheus 2.53	9090	Metrics scrape
grafana	Grafana 11	3001	Dashboards (GitHub OAuth2)
loki	Loki 3.0	3100	Log aggregation

llama-server-llm and llama-server-orpheus are infrastructure (like postgres) — internal Docker network only, no external ports, started automatically with --profile ai.

---

Quick Start

Prerequisites

• Docker + Docker Compose v2 (with BuildKit enabled)
• make, go 1.25, rust 1.88, node 20, jq

Core stack (no GPU required)

cp .env.example .env
make up

Starts: postgres → state-engine (runs migrations) → api-gateway → web

docker compose ps             # all 4 should show "healthy" or "Up"
curl localhost:8080/healthz   # → ok
curl localhost:3000           # → NexCast landing page HTML

Full stack with observability

make up-obs
# Grafana:    http://localhost:3001   (configure GITHUB_OAUTH_CLIENT_ID/SECRET in .env)
# Prometheus: http://localhost:9090

Full stack with AI worker — Ubuntu + NVIDIA GPU (recommended)

Prerequisites:

# Verify GPU is visible
nvidia-smi

# Verify Docker GPU passthrough works
docker run --rm --gpus all nvidia/cuda:12.1-base nvidia-smi

# Install NVIDIA Container Toolkit if needed
curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | \
  sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | \
  sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
sudo apt-get update && sudo apt-get install -y nvidia-container-toolkit
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker

Download models and start:

make models   # ~3.9 GB — Llama-3.2-1B Q8_0 + Orpheus-3B Q4_K_M + SNAC ONNX
make up-ai    # starts full stack: core + llama-server-llm + llama-server-orpheus + ai-worker

Verify all AI services loaded:

make logs-ai
# Wait for llama-server-llm:      "llama server listening at http://0.0.0.0:8081"
# Wait for llama-server-orpheus:  "llama server listening at http://0.0.0.0:8082"
# Wait for ai-worker:             "AI Worker gRPC listening on 0.0.0.0:9092"
# GPU model load takes ~15-30s per llama-server instance

---

Local E2E Testing Guide

This traces the full path: Chrome extension → DOM capture → WebSocket → gRPC → LLM → TTS → audio.

Phase 1 — Start the stack

make up
docker compose ps
# Expected: postgres (healthy), state-engine (Up), api-gateway (Up), web (healthy)

Phase 2 — Get a JWT token

curl -s -X POST http://localhost:8080/auth/login \
  -H 'Content-Type: application/json' \
  -d '{"email":"test@nexcast.club","display_name":"Tester"}' | jq .

Expected response:

{
  "token": "eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9...",
  "user": {
    "user_id": "...",
    "tier": "free",
    "quota_remaining": 50000,
    "voice_preset": "nexcast-v1",
    "commentary_style": "energetic"
  }
}

export JWT="<token from above>"

Phase 3 — Verify state-engine via REST

curl -s -H "Authorization: Bearer $JWT" http://localhost:8080/me | jq .
# → same UserState as login response

curl -s -H "Authorization: Bearer $JWT" http://localhost:8080/me/sessions | jq .
# → { "sessions": [] }  (no sessions yet)

Phase 4 — Test the WebSocket directly (no extension needed)

Install wscat if needed:

npm install -g wscat

Open a WebSocket session:

wscat -c "ws://localhost:8080/ws?token=$JWT"

Once connected, paste a game event (TEXT frame):

{"game":"League of Legends","streamer":"faker","chat":["LETS GO","holy","PogChamp","insane","clip it"],"event_type":"gameplay","session_id":"550e8400-e29b-41d4-a716-446655440000"}

Without AI worker running:

• You will receive: {"type":"quota_exhausted"} — quota was valid but worker unreachable (expected)
• The gateway logs the gRPC connection failure gracefully

With AI worker running (make up-ai):

• You will receive multiple binary frames — raw i16 LE PCM audio at 24 kHz
• Each binary frame is one sentence worth of audio (LLM flushes on ., !, ?)
• First audio arrives within ~1–2 seconds of sending the event

Phase 5 — Load the Chrome Extension

Build the extension:

make ext
# Output: frontend/dist/  (Vite build with MV3 manifest)

Load into Chrome:

1. Navigate to chrome://extensions
2. Toggle Developer Mode (top right)
3. Click Load unpacked → select frontend/dist/
4. Pin the NexCast icon to the toolbar

Sign in:

1. Go to https://www.twitch.tv/ — open any live stream
2. Click the NexCast popup icon
3. Enter test@nexcast.club (or your email) → Sign in
4. Popup shows: tier badge, quota progress bar, voice/style selectors

Phase 6 — Observe the live WebSocket flow

Open Chrome DevTools (F12) → Network tab → filter by WS:

Direction	Frame type	When	Content
↑ Outbound	TEXT	On WS open, then after each audio batch finishes (+ 2s gap)	{"game":"…","streamer":"…","chat":[…],"event_type":"gameplay","session_id":"…"}
↓ Inbound	BINARY	~1–2s after each event	Raw PCM bytes (i16 LE, 24 kHz, mono)

Event-driven loop (no fixed interval):

1. Extension sends game state when WebSocket opens
2. offscreen.ts plays audio chunks gaplessly via nextAt accumulator
3. When the last chunk finishes, offscreen.ts fires audio_done → background.ts
4. background.ts waits 2s (MIN_GAP), sends fresh game state → repeat

What you should hear: Continuous live commentary — each batch starts automatically when the previous one finishes playing.

Phase 7 — Observability stack (optional)

cp .env.example .env
# Edit .env: set GITHUB_OAUTH_CLIENT_ID and GITHUB_OAUTH_CLIENT_SECRET

make up-obs

• Grafana → http://localhost:3001 — login with GitHub OAuth
  • Dashboard "NexCast — Node Metrics": CPU %, Memory, Disk %, Network I/O
• Prometheus → http://localhost:9090/targets — scrape status for all services
• Loki log query (in Grafana → Explore):

  {service="api-gateway"} | json
  {service="state-engine"} | json
  

Phase 8 — Test a schema migration (Day 2 demo)

The state-engine auto-runs sqlx::migrate!() on startup. To demonstrate a live migration:

# 1. Add a new migration file:
echo 'ALTER TABLE subscriptions ADD COLUMN IF NOT EXISTS notes TEXT;' \
  > services/state-engine/migrations/003_add_notes.sql

# 2. Rebuild and restart state-engine only (zero downtime on api-gateway):
docker compose build state-engine
docker compose up -d state-engine

# 3. Verify migration ran:
docker compose logs state-engine | grep migration
# → "Migrations applied"

# 4. Confirm column exists:
docker compose exec postgres psql -U nexcast -d nexcast \
  -c '\d subscriptions'

---

Project Structure

NexCast/
├── frontend/               # Chrome extension (MV3, React + Vite)
│   ├── src/background/    # Service worker: WebSocket client, JWT
│   ├── src/content/       # Twitch DOM scraper (game, streamer, chat)
│   ├── src/offscreen/     # AudioContext: PCM decode + gapless playback
│   └── src/popup/         # Extension UI: login, tier badge, prefs
├── web/                    # nexcast.club landing page (React + Vite)
├── services/
│   ├── api-gateway/        # Go: WebSocket, REST, JWT, gRPC client
│   ├── state-engine/       # Rust: user state, quota, PostgreSQL
│   │   └── migrations/    # SQL migrations (auto-applied on boot)
│   └── ai-worker/          # Rust: gRPC server, HTTP→llama-servers, SNAC ONNX decode (ort)
├── proto/                  # gRPC proto definitions
│   ├── commentary/        # CommentaryService: StreamCommentary
│   └── state/             # StateService: users, quota, sessions
├── infra/
│   ├── prometheus/        # prometheus.yml scrape config
│   ├── grafana/           # Provisioned datasources + node dashboard
│   ├── loki/              # Loki storage config
│   └── promtail/          # Docker socket log shipping
├── scripts/
│   ├── bootstrap-state.sh # Local DB setup
│   ├── download-models.sh # LLM + TTS model download (~3.9 GB)
│   └── gen-proto.sh       # buf generate
├── terraform/             # IaC: EKS + RDS + VPC + ECR + DNS (modular)
├── k8s/                   # Kubernetes manifests: deployments, ingress, observability
├── docker-compose.yml      # Local dev stack
└── Makefile               # make up / up-obs / up-ai / ext / web / test

---

Day 2 Operations

OS / Security Patching (node rotation)

EKS managed node groups are updated by:

1. Terraform aws_eks_node_group AMI version bump → terraform apply
2. EKS drains each node (kubectl cordon + kubectl drain --grace-period=60)
3. New node joins, old node terminates — zero WebSocket drops via Blue/Green pod scheduling

Database Schema Changes

• Migrations live in services/state-engine/migrations/ (numbered SQL files)
• state-engine runs sqlx::migrate!() on every startup
• Blue/Green deployment: new pod applies migration before old pods are terminated
• All migrations are backward-compatible (additive only) to support rollback

---

Infrastructure (Terraform)

terraform/ provisions the full AWS stack via terraform apply:

Resource	Module
VPC (3 AZ, public + private subnets)	modules/vpc
EKS cluster (1× g4dn.xlarge GPU + 2× t3.medium)	modules/eks
RDS PostgreSQL db.t3.micro	modules/rds
IAM roles (EKS node, RDS access)	modules/iam
Route53 nexcast.club + ACM cert (HTTPS)	modules/dns

Remote state: S3 bucket + DynamoDB lock table.

---

Cost Analysis

Metric	Cloud API (ElevenLabs)	Bare-Metal (Orpheus TTS)
Variable cost per 1k chars	~$0.05	$0.00
100 users / month API cost	~$13,500	$0.00
Fixed server cost	~$180/mo	~$380/mo
Revenue (100 × $10/mo)	$1,000	$1,000
Net margin	Massive loss	+$620/mo

Orpheus-3B produces ElevenLabs-comparable voice quality. Both llama-servers and the SNAC decoder run on the same GPU — no additional hardware required. Run terraform destroy between demos to keep the student bill under $5.

---

Troubleshooting — ai-worker

Issues encountered during local development on RTX 5070 Ti (sm_120, CUDA 13.0, Ubuntu 24.04). All fixes are already applied in the codebase; this section documents the "why" for future reference.

---

Runtime: libonnxruntime.so: cannot open shared object file

Symptom: ai-worker starts but SNAC inference crashes — libonnxruntime.so not found.

Fix: ORT's download-binaries feature downloads the SO during cargo build into the Cargo cache (a --mount=type=cache in the Dockerfile). Cache contents are invisible to COPY — extract the SO during the build RUN step before the multi-stage boundary:

find /root/.cache/ort.pyke.io -name "libonnxruntime.so*" -exec cp {} / \; 2>/dev/null || true
# Then in the runtime stage:
COPY --from=builder /libonnxruntime.so* /usr/local/lib/
RUN ldconfig

---

Runtime: ORT download-binaries requires glibc ≥ 2.38

Symptom: __isoc23_strtol@@GLIBC_2.38: symbol not found at startup.

Fix: Base both stages on ubuntu:24.04 (glibc 2.39). Debian 12 (Bookworm) ships 2.36.

---

llama-server: Orpheus produces garbage / silent audio

Symptom: ai-worker receives tokens but SNAC decode yields silence or noise.

Root causes & fixes:

1. Wrong GGUF quantization. IQ3_XS and IQ4_NL formats have Blackwell (sm_120) bugs in llama.cpp — use Q4_K_M or Q8_0 only. make models downloads Q4_K_M.
2. Missing repetition_penalty. Orpheus without repetition_penalty ≥ 1.1 loops tokens endlessly. The request body in tts.rs sets "repetition_penalty": 1.1.
3. Wrong prompt format. Orpheus requires "dan: {text}" wrapped in Llama-3 instruct template, with the start audio token <custom_token_128259> appended by the prompt (not generated).

---

llama-server: CUDA error: no kernel image is available for execution on the device

Symptom: llama-server exits on startup when loading Orpheus or the LLM on RTX 5070 Ti.

Fix: Use image ghcr.io/ggml-org/llama.cpp:server-cuda (CUDA 12.8, supports sm_120). Do NOT use server-cuda13 — it has reported failures on some sm_120 configurations.

---

API gateway: stale gRPC connection after ai-worker restart

Symptom: After rebuilding and restarting ai-worker, the api-gateway keeps failing with connection errors even though the new container is healthy. The old container IP is cached.

Fix: Use the DNS resolver in grpc.NewClient so the gateway re-resolves the service address on each new connection:

// services/api-gateway/internal/clients/worker.go
// Before (passthrough — caches IP at startup):
grpc.NewClient(addr, ...)
// After (DNS — re-resolves on reconnect):
grpc.NewClient("dns:///"+addr, ...)

---

Model files required by make models

File	Size	Source
models/llm/model.gguf	~1.3 GB	bartowski/Llama-3.2-1B-Instruct-GGUF (Q8_0)
models/llm/tokenizer.json	~2 MB	unsloth/Llama-3.2-1B-Instruct
models/tts/orpheus.gguf	~2.5 GB	Mungert/orpheus-3b-0.1-ft-GGUF (q4_k_m)
models/tts/snac24.onnx	~50 MB	laion/SNAC-24khz-decoder-onnx

Use Q4_K_M or Q8_0 for Orpheus — IQ3/IQ4 formats have Blackwell (sm_120) bugs in llama.cpp.