feat: self-hosted NLLB-200 translation service with GPU inference

[beastoin]

Summary

Adds a self-hosted translation service using Meta NLLB-200 (No Language Left Behind) as a drop-in replacement for Google Cloud Translation V3. The service runs on GPU (NVIDIA L4) via CTranslate2 and integrates into the existing realtime listen pipeline with automatic fallback to Google if NLLB is unavailable.

This follows the same self-hosting pattern used for STT (Deepgram → Parakeet) and speaker embeddings (cloud → diarizer).

What's New

NLLB Translation Microservice (nllb_translation/)

• FastAPI service with CTranslate2 + SentencePiece for GPU-accelerated inference
• Supports 22 languages with BCP-47 ↔ NLLB token mapping
• POST /v1/translate — batch translation with optional source language
• Prometheus metrics (/metrics), health (/health), readiness (/ready)
• Model downloaded automatically via HuggingFace init container (~600MB, pinned revision)
• Performance: ~50ms p50, <200ms p99 per sentence on L4 GPU

Provider Selection (utils/translation.py)

• TranslationProvider enum: google, nllb
• TRANSLATION_SERVICE_MODELS env var — comma-separated ordered preference list (mirrors STT_SERVICE_MODELS pattern)
• Provider only changes via explicit TRANSLATION_SERVICE_MODELS — the URL alone never switches provider
• Source language auto-detection via langdetect before NLLB calls, filtered to NLLB's supported language set
• NLLB primary with automatic Google fallback on any error
• Fallback telemetry via record_fallback (component=other, from=nllb, to=google)

Infrastructure

• Helm chart (charts/nllb-translation/) with GPU nodeSelector, tolerations, model volume, HPA, ServiceMonitor
• GitHub Actions workflow (.github/workflows/gcp_nllb_translation.yml) — manual dispatch for dev/prod
• Grafana dashboard for translation metrics (latency, throughput, error rates, cache performance)
• Benchmark suite (scripts/benchmark_nllb_performance.py) with GPU tuning sweep results
• Dedicated GPU node pool: nllb-translation-pool (g2-standard-8, 1x L4, autoscale 0-2) on prod-omi-gke
• Zero-downtime deploys: maxSurge: 1, maxUnavailable: 0 — new pod starts before old one terminates

Observability

• Backend-side Prometheus metrics: omi_translation_requests_total, omi_translation_latency_seconds, omi_translation_chars_total
• NLLB server-side metrics: nllb_requests_total, nllb_inference_latency_seconds, nllb_tokenization_latency_seconds
• Idempotent metric constructors prevent ValueError on module re-import in tests

Configuration

Provider is controlled exclusively by TRANSLATION_SERVICE_MODELS — the URL alone never changes provider. Default is always Google.

# 1. Google only (default — no config needed)
# TRANSLATION_SERVICE_MODELS is unset or empty

# 2. NLLB primary with Google fallback (recommended)
TRANSLATION_SERVICE_MODELS=nllb,google

# 3. NLLB only (no fallback)
TRANSLATION_SERVICE_MODELS=nllb

HOSTED_TRANSLATION_API_URL must be set for nllb to activate — if the URL is missing, nllb is skipped and the next provider in the list is tried.

Files Changed

Area	Files	Description
NLLB service	nllb_translation/{main,Dockerfile,requirements,README}.py	New GPU microservice
Translation layer	utils/translation.py	Provider enum, NLLB dispatch, fallback, metrics, source lang filtering
Coordinator	utils/translation_coordinator.py	source_language threading
Listen pipeline	routers/transcribe.py	TranslationCoordinator construction
Helm	charts/nllb-translation/, charts/backend-listen/	Deployment config
CI	.github/workflows/gcp_nllb_translation.yml	Deploy workflow
Benchmarks	scripts/benchmark_nllb_*.py	Performance testing
Grafana	charts/monitoring/dashboards/omi-services/omi-translation.json	Dashboard
Docs	AGENTS.md	Service map update
Tests	tests/unit/test_translation_nllb.py + others	29 tests (3 skipped)

Test Evidence

Unit Tests (209 passed, 3 skipped)

• Google batch translation helper: 3 tests
• TranslationProvider config dispatch (TRANSLATION_SERVICE_MODELS): 6 scenarios
• NLLB primary dispatch + Google fallback: 3 tests
• Source language auto-detect: 6 tests (zh-cn, short text, LangDetectException, unreliable lang, NLLB unsupported, NLLB supported)
• NLLB batch API: 7 tests (returns, source_language, omission, malformed, truncated, empty)
• Prometheus idempotency: 2 tests
• NLLB language mapping: 3 tests (skipped — NLLB deps not installed)
• Translation optimization/dedup/cost: 177 existing tests (separate files)

L2 Live Testing (NLLB on dev cluster)

Tested TranslationService integration against real NLLB service on dev-omi-gke:

• 10 test cases across 8 language pairs: ES, JA, DE, FR, ZH-CN, KO, AR, HI
• translate_text: single text translation verified for 7 languages
• translate_text_by_sentence: multi-sentence splitting + batch translation verified
• translate_units_batch: 3-unit and 10-unit transcript segment batches verified
• Source language pass-through: FR→EN with explicit source_language verified
• NLLB source language filtering bug found and fixed: langdetect returning languages not in NLLB's 22-language set (e.g. Somali 'so' for English text) caused 400 errors; now filtered before sending to NLLB

E2E Integration

• 54-segment conversation on dev GKE pod (44 Spanish + 10 French), all translations correct
• Full WebSocket /v4/listen pipeline: audio → VAD → STT → TranslationCoordinator → NLLB → conversation lifecycle completed
• Fallback test: simulated NLLB outage → record_fallback fired → Google recovery confirmed
• Helm template rendering verified for both dev and prod values

Deploy Plan

Prod config ships with TRANSLATION_SERVICE_MODELS=nllb,google — NLLB primary with Google fallback. L2 testing confirmed NLLB works correctly on dev cluster.

Sequence

Step	What	Rollback
1. Merge PR	Backend image gets NLLB code + nllb,google config	Revert PR
2. Deploy NLLB service	gh workflow run gcp_nllb_translation.yml -f environment=prod -f branch=main	Delete NLLB deployment
3. Deploy backend-listen	Rolling restart picks up NLLB config. NLLB handles translation with Google fallback.	Set TRANSLATION_SERVICE_MODELS="" and redeploy

Post-deploy verification

1. Pod health: kubectl -n prod-omi-backend get pods -l app.kubernetes.io/name=nllb-translation
2. NLLB health: verify /health and /ready from inside the pod
3. Monitor Grafana dashboard: NLLB latency, error rate, fallback rate to Google
4. If high error rate, set TRANSLATION_SERVICE_MODELS="" (instant rollback to Google only)

Closes #9430

by AI for @beastoin

[beastoin]

@beastoin Required fixes before merge: backend/nllb_translation/Dockerfile:9 plus backend/charts/nllb-translation/*_values.yaml:21 point NLLB_MODEL_DIR at /models/nllb-200-distilled-600M-ct2-int8 but the image copies only app code and the chart mounts no model volume, so the pod will fail during startup; backend/charts/nllb-translation/templates/hpa.yaml:15 with backend/charts/nllb-translation/prod_omi_nllb_translation_values.yaml:75 renders a prod HPA with a bare metrics: key because no target metric is set; backend/charts/nllb-translation/prod_omi_nllb_translation_values.yaml:98 uses maxSurge: 1 / maxUnavailable: 0 for a 1-GPU pod, unlike the existing GPU charts, which can deadlock rollouts when no spare GPU is available; backend/nllb_translation/main.py:138 computes the source language but never applies it to the tokenized input, so the shadow service ignores Google's detected source language; backend/nllb_translation/main.py:165 and the other new route handlers are async def with no awaits and call sync GPU/model work on the event loop; backend/tests/unit/test_translation_shadow.py:27 mutates sys.modules at module scope and fails python3 scripts/check_module_stub_pollution.py; backend/utils/translation.py:581 / :663 add in-function imports against AGENTS. Can you fix these and rerun the focused translation tests plus rendered Helm validation?

---

by AI for @beastoin

[beastoin]

CP9A — Level 1 Live Test (Backend standalone)

Changed-path coverage checklist:

Path ID	Changed path	Happy-path test	Non-happy-path test	L1 result + evidence
P1	utils/translation.py:_translate_google_batch	Unit test: google batch returns tuples	Unit test: None detected lang returns empty	PASS — test_google_batch_returns_tuples, test_google_batch_none_detected_lang
P2	utils/translation.py:_run_shadow_compare	Unit test: logs exact_match_ratio	Unit test: timeout+error don't propagate	PASS — test_shadow_compare_exact_match_ratio, test_shadow_timeout_does_not_propagate, test_shadow_timeout_logs_warning
P3	utils/translation.py:_schedule_shadow_compare	Unit test: all 3 callsites schedule shadow	Unit test: not called without URL, not in google mode	PASS — test_translate_text_returns_google_in_shadow_mode, test_translate_text_by_sentence_schedules_shadow, test_translate_units_batch_schedules_shadow, test_shadow_not_called_in_google_mode, test_shadow_not_called_without_url
P4	nllb_translation/main.py:_translate_batch	Skipped — requires ctranslate2+GPU	Skipped — requires ctranslate2+GPU	SKIP — deps not in CI, 3 NLLB tests properly skip with ImportError
P5	nllb_translation/main.py:_resolve_nllb_code	Skipped — requires ctranslate2	Skipped — requires ctranslate2	SKIP — tested via stubbed import: zh-TW→zho_Hant verified
P6	charts/nllb-translation/*	helm template renders valid manifests	HPA without metric was invalid, now valid	PASS — helm template prod: valid HPA w/ CPU 70%, maxUnavailable:1, PVC mount
P7	AGENTS.md	Docs change, no executable code	N/A	PASS — verified service map entry present

Evidence:

• beast omi dev doctor: 18/18 passed
• beast omi dev boot-check: import clean (6.6s)
• pytest -q tests/unit/test_translation_shadow.py: 12 passed, 3 skipped
• check_module_stub_pollution.py: 544 files, 0 violations
• helm template prod: valid HPA, correct rollout strategy, model volume present
• BCP-47 locale resolution verified: zh-TW→zho_Hant, zh-Hant→zho_Hant, pt-BR→por_Latn

P4/P5 justification: NLLB service requires ctranslate2, sentencepiece, and CUDA GPU. These are not available in CI or on this VPS. The service is a standalone FastAPI app that will be tested during dev cluster deployment (CP9C/test plan). The shadow integration in the backend (P1-P3) is fully tested.

---

by AI for @beastoin

[beastoin]

CP9B — Level 2 Live Test (Backend + App integrated)

Integration scope: This PR adds shadow translation mode which is OFF by default (TRANSLATION_MODE=google). No user-facing behavior changes. The NLLB service requires GPU+model weights and cannot run on this VPS.

Evidence:

• beast omi dev boot-check --full: uvicorn starts, health endpoint responds on :8700 (PID 3590010, killed after verify)
• beast omi dev boot-check: import clean (6.2s)
• Shadow mode is fire-and-forget — errors are swallowed, Google translations always returned unchanged
• Default mode (google) has zero code path changes — all new code gated behind TRANSLATION_MODE == "shadow" check

Integration verification:

• Backend boots with modified TranslationService — new import random, from utils.executors import at top level do not break import chain
• _schedule_shadow_compare is a no-op when TRANSLATION_MODE != "shadow" (verified by unit test)
• App is unaffected — TranslationService API (translate_text, translate_text_by_sentence, translate_units_batch) returns identical results

L2 limitation: The NLLB service itself is a new standalone FastAPI service that requires ctranslate2 + GPU. Full integration testing of the shadow comparison path requires dev GKE cluster deployment with GPU node pool and model weights. This is tracked in the test plan.

---

by AI for @beastoin

[beastoin]

@beastoin .github/workflows/gcp_nllb_translation.yml:61 and .github/workflows/gcp_nllb_translation.yml:73 still tag and deploy with ${GITHUB_SHA::7} after checking out github.event.inputs.branch; when the workflow is dispatched from one ref but the input branch is another, this builds the checked-out branch under the workflow-ref SHA, can reuse an unchanged image tag, and may leave Helm with no pod-template change even though different source was built. Please compute the short SHA from checked-out HEAD after checkout, like gcp_notifications_job.yml, and use that output for both docker build/push and --set image.tag=....

---

by AI for @beastoin

[beastoin]

@beastoin backend/utils/translation.py:792/:796 makes NLLB the primary provider but calls _translate_nllb_batch(contents, dest_language) without a source language; the service only adds the NLLB source token when source_language_code is present (backend/nllb_translation/main.py:197), and the benchmark/perf clients explicitly send source_language_code: "en", so the production primary path is the one path that drops the source hint and returns empty detected-language output. This means auto-detect primary mode can regress multilingual realtime translation outside the English-source smoke tests and can’t update downstream language state from the response; please derive/pass the detected source language into the NLLB primary payload and add a regression test that asserts source_language_code is sent, instead of the current tests/unit/test_translation_shadow.py:672 assertion that locks in the two-argument call. Can you fix that and rerun the focused translation tests plus a non-English-source NLLB smoke test?

---

by AI for @beastoin

[beastoin]

lgtm

[beastoin]

L2 Live Testing Complete — NLLB on Dev Cluster

Bug Found & Fixed

During L2 testing against real NLLB on dev-omi-gke, discovered that langdetect can return languages outside NLLB's 22-language supported set (e.g. detecting English as Somali so), causing NLLB 400 errors and unnecessary fallback to Google.

Fix (utils/translation.py): Added NLLB_SUPPORTED_SOURCE_LANGUAGES set. _detect_source_language() now filters detected language against this set when using the NLLB provider — if unsupported, source hint is omitted and NLLB translates without it.

L2 Test Results (10/10 PASS)

#	Method	Direction	Result
1	translate_text	EN→ES	PASS — "Hola, ¿cómo estás hoy?"
2	translate_text_by_sentence	EN→JA	PASS — 2 sentences translated
3	translate_units_batch	EN→DE	PASS — 3 units translated
4	translate_text	FR→EN	PASS — "Hello world"
5	translate_text	EN→ZH-CN	PASS — "人工智能正在改变世界"
6	translate_text	EN→KO	PASS — "도움으로 고마워요"
7	translate_text	EN→AR	PASS — "مرحباً بك في الطلب"
8	translate_units_batch	EN→ES (10 segs)	PASS — all 10 transcript segments
9	translate_text_by_sentence	EN→FR (5 sentences)	PASS — longer text
10	translate_text	EN→HI	PASS — "कृपया मुझे निकटतम रेस्तरां खोजने में मदद करें"

NLLB latencies: 27–121ms depending on batch size. All 3 entry points (translate_text, translate_text_by_sentence, translate_units_batch) verified against real NLLB service.

Config Change

Prod TRANSLATION_SERVICE_MODELS changed from "" to "nllb,google" — NLLB primary with Google fallback, skipping gradual rollout per manager decision.

Tests

• 209 passed, 3 skipped (NLLB deps not installed)
• 2 new regression tests for source language filtering (test_detect_source_language_nllb_unsupported_returns_empty, test_detect_source_language_nllb_supported_returns_lang)

Commit

61ea3529d — fix: filter detected source languages against NLLB supported set, enable nllb,google in prod

by AI for @beastoin

[beastoin]

Deploy Monitor — T+2h Consolidated Checkpoint (01:23–03:44 UTC Jul 12)

Check	Result
NLLB pod	0 restarts, 13h+ uptime, no OOMKills
Backend-listen	12/12 Running, 0 restarts, revision 65bc6d9489
NLLB errors	zero in 2h window
NLLB→Google fallbacks	zero in 2h window
NLLB request rate	~15 req/min steady
NLLB p50 latency	71ms
NLLB p99 latency	1.27s (cold-path spikes for less common languages)
API health (VPS)	HTTP 200 in 252ms
WS endpoint (VPS)	Alive (403 auth — expected)
New error patterns	None

Traffic context: Off-peak window (01:00–04:00 UTC).

Notes:

• p99 increased from 197ms (T+0) to 1.27s — likely cold-path translations for less common languages, not regression. Will monitor for sustained high p99 at peak.
• Grafana dashboard live with all 22 panels showing real data.
• ServiceMonitor label fix in PR feat: production-grade NLLB translation dashboard, alerting, and ServiceMonitor fix #9521 (permanent).

Status: PASS

Next checkpoints: T+4h (05:23 UTC), T+8h (09:23 UTC), T+24h (Jul 13 01:23 UTC).

by AI for @beastoin

[beastoin]

Deploy Monitor — T+24h Final Verdict: SUCCESS

Monitoring issue: #9622 (closed)

Check	Result
NLLB pod health	0 restarts, 41h uptime, model continuously loaded
NLLB errors	zero
Provider split	99.9% NLLB / 0.1% Google fallback
Google Translate cost	~97% reduction (first full day vs baseline)
All 16 backend-listen pods	Using NLLB, runtime env confirmed
Latency / SLO	Within bounds, no alerts

Verdict: NLLB self-hosted translation operating as designed. Deploy monitoring complete.

Follow-up: ~3,952 requests/day with target_lang="multi" causing unnecessary Google fallbacks — filed separately.

by AI for @beastoin