Building a Fully Local AI Video Translation & Dubbing Studio – ASR, Context-Aware CJK Translation, TTS and Sync

[anhbaokr]

🏷️ Discussion Type

Question

Body

**I'm developing a local AI-powered video studio focused on automatic speech recognition, translation, voice generation, and dubbing synchronization.

The goal is to run the main models locally on the user's computer, with little or no dependence on paid cloud AI APIs.

The current pipeline is roughly:

Video → Local ASR → Context Analysis → Translation → Local TTS → Timing / Lip-Audio Sync → Final Video

One of the biggest challenges I am working on is translation quality for Chinese, Korean, and Japanese video content.

For movies, dramas, anime, historical series, fantasy, cultivation/xianxia, wuxia, modern romance, crime, military, and other genres, a literal translation is often not enough. The system needs to understand:

• Genre-specific terminology and expressions
• Character relationships, titles, honorifics, ranks, and forms of address
• Historical, cultural, martial-arts, cultivation, fantasy, and fictional terminology
• Pronouns, omitted subjects, context, and speaker relationships
• Idioms, slang, fixed expressions, and culturally specific phrases
• Consistency of names, places, skills, factions, techniques, items, and terminology across an entire movie or series
• The appropriate writing style and tone for each genre instead of producing generic machine-translated dialogue

For example, a Chinese cultivation or wuxia drama should not sound like a modern everyday conversation after translation. Likewise, Korean historical dramas, Japanese anime, crime films, military content, and modern dramas may all require different terminology, tone, and translation rules.

My goal is therefore not only:

Speech-to-Text → Translation → Text-to-Speech

but a more context-aware system such as:

ASR → Dialogue & Scene Context → Genre / Terminology Detection → Translation Memory & Glossary → Semantic Quality Checks → TTS → Dubbing Sync

I would really appreciate recommendations from anyone who has worked on similar problems.

I'm particularly interested in open-source models, algorithms, papers, datasets, translation-memory techniques, terminology management, contextual translation methods, local LLM approaches, ASR correction, TTS, and dubbing synchronization that could help build this type of system.

If you know any GitHub projects or have experience with Chinese / Korean / Japanese media translation and local AI dubbing, I would be very interested in your suggestions.**

Guidelines

• I have read and understood this category's guidelines before making this post.

[karthikj30]

Hey @anhbaokr
Please Mark this as answered if this helps you

The core move: two passes, not streaming

Pass 1 — extract, before translating a line. Run the full transcript through a local LLM to build:

• Character roster — speakers, name variants, and the relationship graph (who is 师父/师兄, who uses 반말 vs 존댓말 to whom)
• Glossary — names, places, sects, techniques, ranks, with a locked target rendering
• Register map — politeness level per speaker-pair, and where it shifts (a K-drama dropping 존댓말 is plot, not style; naive MT erases it)

Pass 2 — translate each cue with speaker label + surrounding cues + the glossary slice injected as hard constraints.

Persist the glossary across episodes. Episode 1 seeds episode 2. For a 60-episode xianxia, a human locking 金丹 → "Golden Core" once beats any amount of model cleverness.

On genre detection: don't make it a pipeline stage. Buckets are too coarse. Ship style profiles (prompt fragment + glossary pack) the user can pick or override. Cultivation terminology is a glossary problem, and fan-translation conventions are your ground truth.

Sync — the constraint that kills amateur dubs

Chinese is dense; English runs 1.5–2× the syllables. Fix it in translation, not audio:

1. Compute a syllable budget per cue from ASR timing
2. Pass it into the prompt as a constraint
3. Retranslate with compression instruction if it overruns
4. Use TTS duration control to hit the target
5. Time-stretch last, ±10% max

Also allow cue merge/split — rigid 1:1 cue mapping is a hard quality ceiling.

Repos closest to your goal

[VideoLingo](https://github.com/Huanshere/VideoLingo) — read this first. Uses a "Translate-Reflect-Adaptation" three-step process for contextual coherence, enforces Netflix-standard single-line subtitles, supports GPT-SoVITS and Fish-TTS. That reflection loop is the QA pass you'd otherwise design yourself. But it can't dub multiple characters separately because WhisperX's speaker distinction isn't reliable enough, and Japanese output is rated noticeably weaker than Chinese. No speaker separation = no register mapping = the exact gap you'd fill.

[pyVideoTrans](https://github.com/jianchang512/pyvideotrans) (~16k ⭐) — the best base to fork. Speaker diarization to distinguish roles, per-speaker dubbing voices, F5-TTS/CosyVoice/GPT-SoVITS cloning, and pause-and-proofread at every stage (recognition, translation, dubbing). GPL-v3, CLI + WebUI + desktop, fully offline or API, integrates Qwen3-ASR and Faster-Whisper. That per-stage proofreading hook is where your glossary editor plugs in.

[KrillinAI](https://github.com/krillinai/KrillinAI) — Go, LLM-driven, staged CLI invocation with a JSON manifest instead of log parsing. Worth it for the resumable/scriptable design.

[Linly-Dubbing](https://github.com/Kedreamix/Linly-Dubbing) (~3k ⭐) — Demucs + UVR5 vocal separation, one-shot cloning from 3–10s, lip-sync via Linly-Talker. Chinese-first audio engineering.

[bluez-dubbing](https://github.com/Globluez/bluez-dubbing) — modular, REST API + CLI + web UI, WhisperX word-level timing with VAD alignment. Clean architecture reference.

[Chenyme-AAVT](https://github.com/chenyme/Chenyme-AAVT) (~3k ⭐) — minimal baseline to measure against.

Components you'll depend on

• [GPT-SoVITS](https://github.com/RVC-Boss/GPT-SoVITS) — best CJK few-shot cloning; default for character voices
• [FunASR](https://github.com/modelscope/FunASR) (Paraformer) — beats Whisper on native Chinese, punctuation + VAD built in
• [WhisperX](https://github.com/m-bain/whisperX) — word-level timestamps; non-negotiable for timing
• [pyannote-audio](https://github.com/pyannote/pyannote-audio) — diarization. VideoLingo's weak point, so your differentiator
• [UVR5](https://github.com/Anjok07/ultimatevocalremovergui) / [Demucs](https://github.com/facebookresearch/demucs) — vocal isolation, gives you the background bed to mix over
• [IndexTTS-2](https://github.com/index-tts/index-tts) — precise duration control built for dubbing, independent timbre/emotion control. The single most useful TTS feature for your sync problem
• [CosyVoice](https://github.com/FunAudioLLM/CosyVoice) — zh/ja/ko cross-lingual zero-shot cloning, ~150ms streaming
• [MuseTalk](https://github.com/TMElyralab/MuseTalk) — if you do lip-sync. MIT with commercially usable weights, 256×256 at 30+ FPS on one V100 — unlike [Wav2Lip](https://github.com/Rudrabha/Wav2Lip), whose open release is non-commercial only.

What I'd build first

Don't build the pipeline. Build glossary + roster extraction + the two-pass translator as a standalone .srt → .srt tool. You can then evaluate against human fansubs with no ASR/TTS variance in the way. ASR and TTS are integration work; contextual CJK media translation is the unsolved part, and it's where this project wins or doesn't.

[antoniovasssss]

Hi @anhbaokr

1. Core Architecture: Two-Pass Contextual Translation (Rather than translating line-by-line during ASR output, process the full transcript in two distinct phases using a local LLM (such as Qwen2.5 or DeepSeek-R1):

• Pass 1 (Extraction): Parse the transcript to generate a Character Roster (mapping speaker relationships, honorifics, and register like 반말 vs. 존댓말 or 师父/师兄), a Glossary/Terminology Matrix (locking names, sects, and techniques across episodes), and a Register Map.
• Pass 2 (Translation): Translate dialogue cues while injecting the roster and locked glossary as hard system prompt constraints, passing surrounding context for coherent tone.

2. Key Local Models & Tools

• ASR & Diarization: Use https://github.com/modelscope/FunASR (Paraformer) for native Chinese accuracy and punctuation, WhisperX for word-level timestamps, and pyannote-audio for speaker diarization.
• TTS & Voice Cloning: Use GPT-SoVITS for CJK character voice cloning, CosyVoice for cross-lingual zero-shot TTS, and IndexTTS-2 for built-in duration control.
• Audio Engineering: Isolate voice tracks using Demucs or UVR5 to preserve background audio (BGM/SFX).
• Lip-Sync (Optional): Use MuseTalk for real-time 30+ FPS lip synchronization.

3. Reference Open-Source Projects

• VideoLingo: Features a 3-step "Translate-Reflect-Adaptation" pipeline enforcing single-line subtitle standards.
• pyVideoTrans: Offers multi-speaker dubbing, offline workflow, and human-in-the-loop proofreading hooks at each stage.
• Linly-Dubbing: Integrates vocal separation, voice cloning, and lip-syncing into an end-to-end pipeline.

4. Dubbing Synchronization Strategy

• Calculate a target syllable budget per cue from ASR timestamps before translation. Feed the character length limit directly into your local LLM prompt as a constraint, re-translating overrunning lines before applying mild TTS duration adjustments or time-stretching ($\pm 10%$ max).

[Pusri27]

Hey @anhbaokr,

Honestly, the architectural breakdown from @karthikj30 is pretty spot on, because handling character relationship mapping and glossary extraction before translating dialogue cues is basically the only reliable way to keep honorifics and xianxia terminology intact

Just to toss in a couple of extra practical points from the local pipeline engineering side:

• Local LLM Selection and Context Window: For parsing full episode transcripts in Pass 1, running something like Qwen2.5 (14B/32B) or DeepSeek R1 Distill Qwen locally via Ollama, vLLM, or llama.cpp gives you huge context windows and phenomenal native CJK handling compared to generic Llama models
• Structured JSON Outputs (Grammars / Function Calling): Force your local LLM to output character rosters and glossaries strictly via JSON Schema or BNF Grammars, which completely avoids regex headaches and makes your glossary persistence between episodes completely deterministic
• Smart Audio Ducking and Merging: When mixing the cloned TTS audio back with the UVR5 or Demucs separated background tracks, use dynamic volume ducking via FFmpeg or Python (pydub / sox) keyed specifically to your speech timestamps so BGM does not fight with the new dialogue

Like mentioned earlier, definitely start small by building out the standalone subtitle to subtitle extraction and translation loop first before wiring up the full audio and TTS pipeline

[Lxcardoza993]

You're right that literal translation destroys CJK content — but the gap between "literal" and watchable is mostly context injection, not model quality. The architecture that works in practice:

Two-pass translation at the scene/season level, not line-by-line.

• Pass 1 — context extraction. Feed the full transcript (with speaker IDs + timestamps) to a local LLM and have it output a structured "context packet": character roster with relationships/age/status; honorific map (Japanese さん/君/様 → who addresses whom how); genre glossary (xianxia: 灵气/金丹/渡劫; wuxia: 门派/轻功; historical titles); per-scene tone. Ask it to output a closed set of term choices ("仙界 as one canonical term") so pass 2 never free-styles.
• Pass 2 — translation. Line by line, with the packet injected as system context plus the previous 1–2 translated lines for flow. This fixes most "honorifics and xianxia terms come out wrong" cases, because the model stops guessing.

ASR: for CJK the practical local defaults are Whisper large-v3 (fine for mixed zh/ja/ko) or, for Mandarin-heavy content, the FunASR/Paraformer family — noticeably better on zh and it gives word-level timestamps. Keep speaker diarization as its own step (pyannote/whisperX); don't trust ASR speaker guesses.

TTS + sync: one consistent voice ID per character — GPT-SoVITS (zh, few-shot voice cloning), VOICEVOX/AivisSpeech (ja, consistent but less expressive), community VITS checkpoints for Korean. For timing, reuse the ASR timeline: translate each subtitle cue, generate TTS per cue, align by start timestamp. Accept ±80–120 ms drift per cue; true lip sync on close-ups is an order of magnitude harder and rarely worth it.

The error you'll actually fight: genre-term consistency across a 40-episode series. The glossary-in-pass-1 approach keeps 灵气 called the same thing in episode 40 as episode 1 — that's the thing dubbed releases get wrong most visibly.

Hardware-wise this is very achievable locally: llama.cpp/LM Studio + a 12B-class qwen model handles pass 1 comfortably; the TTS stack runs on the same box. Happy to go deeper on any of the stages — ASR→translation handoff being the one that breaks pipelines most often.

If this maps out the pipeline you need, marking it as accepted helps the next person building the same stack. 🎬