Automated Video Translator and Dubbing

This project supports two main flows from one shared pipeline:

1. dub.py for the Idealabs assignment: YouTube URL -> download -> diarized transcription -> English translation -> XTTS voice clones -> ffmpeg remix -> dubbed video on disk.
2. app.py for the full product: Streamlit UI with YouTube URL, upload, local file input, multi-language dubbing, optional subtitle burn-in, summary, and Q&A.

Features

• YouTube download with yt-dlp
• Transcription with OpenAI gpt-4o-transcribe-diarize (speaker labels + timestamps)
• Long videos are split into ≤20-minute chunks for the OpenAI duration limit, then timestamps are merged
• User-selected source and destination languages (Streamlit sidebar / CLI flags)
• Auto-detect fallback when source is auto: OpenAI omits the language hint and Google Translate uses source="auto"
• English → Hindi translation with IndicTrans2 (ai4bharat/indictrans2-en-indic-dist-200M); Google Translate for other pairs / fallback
• Translation with Google Translate via deep_translator (non-Indic pairs)
• Multi-speaker Coqui XTTS-v2 voice cloning (one clone per detected speaker)
• Cue-level timing fit and merged dubbed audio
• Fast ffmpeg remix for assignment output
• Optional MoviePy subtitle burn-in in Streamlit
• OpenAI transcript summary and Q&A
• Optional edge-tts single-voice path via --no-clone

Setup

1. Install Python dependencies

Use Python 3.11 (Coqui TTS does not support Python 3.12+).

uv venv .venv
.venv\Scripts\activate
uv pip install -r requirements.txt

XTTS downloads model weights on first use. A CUDA GPU is strongly recommended.

IndicTrans2 also downloads its Hugging Face model on first English → Hindi translation. Other language pairs keep using Google Translate.

By loading XTTS you agree to Coqui's non-commercial CPML terms (https://coqui.ai/cpml) unless you have a commercial license. The app sets COQUI_TOS_AGREED=1 for non-interactive runs.

2. Install system tools

• ffmpeg is required for audio extraction, reference clips, duration fit, and remix.
• ImageMagick is required only if you use subtitle burn-in through MoviePy/TextClip.

3. Create environment file

copy example.env .env

Important values:

• OPENAI_API_KEY required for diarized transcription, Summary, and Q&A
• openai_whisper_model defaults to gpt-4o-transcribe-diarize

Note: some OpenAI projects do not allow whisper-1. Prefer the diarize model.

4. Create output directory

mkdir output

Run the assignment script

python dub.py "https://www.youtube.com/watch?v=..."

Or run without arguments and paste the URL at the prompt:

python dub.py

Useful options:

python dub.py "https://www.youtube.com/watch?v=..." --source-lang hi --dest-lang en
python dub.py "https://www.youtube.com/watch?v=..." --source-lang auto --dest-lang en
python dub.py "https://www.youtube.com/watch?v=..." --no-clone --voice en-US-GuyNeural

• --source-lang defaults to auto (OpenAI/Google auto-detect). Pass a code like hi or es when you know the spoken language.
• --dest-lang defaults to en (translation target).

The script prints progress to the terminal and saves the dubbed video under output/<video_name>/.

Multi-speaker path:

1. Diarize speech into speaker-labeled cues (speakers.json)
2. Cut a 2–10s reference clip per speaker under refs/
3. Synthesize each English cue with that speaker’s XTTS clone
4. Merge and ffmpeg-replace audio

Run the full Streamlit app

streamlit run app.py

Submit your OpenAI API key in the sidebar before processing video.

The app supports:

• YouTube URL input
• File upload
• Local video path
• Source language selection (or Auto-detect fallback)
• Destination language selection for translation
• Assignment mode for English dubbing defaults
• Multi-speaker voice clone (XTTS) checkbox
• Summary and Q&A after a video has been processed

Output files

Each processed video gets its own folder under output/.

Typical artifacts:

• source audio as .mp3
• speakers.json (speaker labels per cue)
• refs/speaker_*.wav (clone references)
• original / translated / corrected SRT files
• transcript text as text.txt
• merged dubbed audio as merged_with_silence.wav
• final dubbed video as <video_name>_translated.mp4

Render notes

• Render is best for shorter demo clips, not long 30-minute or 2-hour assignment submissions.
• Transcription uses the OpenAI diarize API (no local Whisper required).
• XTTS cloning needs enough RAM/GPU on the host running the app.
• Keep the Streamlit app as the deploy target and use the CLI locally for long assignment runs.