Supernan AI Hindi Dubbing Pipeline

Zero-Cost | Open-Source | Python 3.12 Compatible

Submission: Supernan AI Automation Intern Challenge Cost: ₹0 | Runtime: Google Colab Free T4 GPU

---

Pipeline Architecture

Input Video (Kannada/English)
    │
    ▼
[1] FFmpeg ──────────── Extract segment (0:15–0:30) + convert to 25fps/480p
    │
    ▼
[2] Whisper Medium ──── Transcribe with word-level timestamps
    │
    ▼
[3] GoogleTranslator ── Length-constrained Hindi translation
    │                   (max words = duration × 2.5 words/sec)
    ▼
[4] edge-tts Neural ─── Per-segment Hindi TTS (hi-IN-SwaraNeural, rate=-10%)
    + FFmpeg atempo ──── Dynamic speed adjustment [0.85x–1.15x] per segment
    │
    ▼
[5] Wav2Lip GAN ──────── Lip sync at 25fps/480p
    │
    ▼
[6] GFPGAN v1.4 ──────── Face restoration + 2x upscale (480p → 960p)
    │
    ▼
[7] FFmpeg H264 ──────── Final 1920×1080, CRF15, 44.1kHz AAC stereo
    │
    ▼
Output: FINAL_dubbed.mp4

---

Setup Instructions

Requirements

• Google Colab Free Tier (T4 GPU) or any NVIDIA GPU 8GB+ VRAM
• Python 3.12+

Install Dependencies

pip install openai-whisper deep-translator pydub soundfile \
            edge-tts nest-asyncio gfpgan facexlib basicsr

# Fix basicsr torchvision compatibility
sed -i 's/from torchvision.transforms.functional_tensor import rgb_to_grayscale/from torchvision.transforms.functional import rgb_to_grayscale/' \
    $(pip show basicsr | grep Location | awk '{print $2}')/basicsr/data/degradations.py

# Wav2Lip
git clone https://github.com/Rudrabha/Wav2Lip.git
sed -i 's/opencv-python==4.1.0.25/opencv-python>=4.3.0/' Wav2Lip/requirements.txt
sed -i 's/torch==1.1.0/torch>=2.0.0/' Wav2Lip/requirements.txt
sed -i 's/torchvision==0.3.0/torchvision>=0.15.0/' Wav2Lip/requirements.txt
pip install -r Wav2Lip/requirements.txt
sed -i 's/librosa.filters.mel(hp.sample_rate, hp.n_fft/librosa.filters.mel(sr=hp.sample_rate, n_fft=hp.n_fft/' Wav2Lip/audio.py

# Download checkpoints
wget https://huggingface.co/numz/wav2lip_studio/resolve/main/Wav2lip/wav2lip_gan.pth \
     -O Wav2Lip/checkpoints/wav2lip_gan.pth
wget https://github.com/TencentARC/GFPGAN/releases/download/v1.3.0/GFPGANv1.4.pth

Run

# Standard run (with lip sync)
python dub_video.py --input "Hygiene - Kannada.mp4" --output output/dubbed.mp4

# Skip lip sync (clean audio replacement)
python dub_video.py --input video.mp4 --no-lipsync

# Custom segment
python dub_video.py --input video.mp4 --start 0 --end 60

# Higher accuracy transcription
python dub_video.py --input video.mp4 --whisper large-v2

---

All Dependencies

Package	Version	Purpose
openai-whisper	latest	English transcription
deep-translator	latest	English → Hindi
edge-tts	latest	Hindi neural TTS
nest-asyncio	latest	Async fix for Colab
pydub	latest	Audio assembly
soundfile	latest	WAV read/write
gfpgan	latest	Face restoration
basicsr	1.4.2	GFPGAN dependency
facexlib	latest	Face detection
torch	≥2.0.0	GPU inference
ffmpeg	system	Video/audio processing

---

Estimated Cost Per Minute of Video

Step	GPU Time (per min video)	Cost @ RTX4090 $0.44/hr
Whisper Medium	~15s	$0.002
edge-tts (CPU)	~5s	$0.001
Wav2Lip GAN	~90s	$0.011
GFPGAN per frame	~120s	$0.015
FFmpeg encode	~10s	$0.001
Total	~4 min GPU	~$0.030/min

Full 5-min video: ~$0.15 | 500 hours: ~$900

---

Scale Question: 500 Hours Overnight

Processing 500 hours (30,000 minutes) on one Colab = 83 days. Production architecture:

Architecture

500hr videos
    │
    ▼
AWS S3 (storage)
    │
    ▼
Lambda: split into 1-min chunks → 30,000 jobs
    │
    ▼
Redis queue (30,000 jobs)
    │
    ├── Worker 1 → dub_video.py
    ├── Worker 2 → dub_video.py
    ├── Worker 3 → dub_video.py  (200 parallel workers)
    └── ...
    │
    ▼
S3 (30,000 dubbed chunks)
    │
    ▼
Lambda: stitch chunks → 500hr final output

Cost Breakdown

30,000 chunks × 4 min GPU = 120,000 GPU mins = 2,000 GPU hours
RTX 4090 on RunPod: $0.44/hr
Compute:   2,000 × $0.44  = $880
S3 storage (500hr @ 1GB/hr): $11.50
Transfer + Lambda:  ~$15
─────────────────────────────
Total: ~$907 for 500 hours
Cost per hour: ~$1.81
Time with 200 parallel workers: ~10 hours overnight ✅

---

Known Limitations

Limitation	Root Cause	Production Fix
Lip sync imperfect on head rotation	Wav2Lip trained on frontal LRS2 faces only	MuseTalk (ByteDance 2024) — handles 3D head movement. Requires Python 3.10 + mmpose, incompatible with Colab 3.12. Would run in Docker on Python 3.10.
No voice cloning	XTTS requires Python <3.12	Use XTTS on Python 3.10 Docker or RunPod
Translation sometimes literal	Google Translate lacks domain context	Claude API with prompt: "Translate for a childcare training video. Use simple conversational Hindi. Keep syllable count close to English."
GFPGAN slow on large videos	Per-frame O(n)	Batch frames, use fp16 half-precision

---

Resourceful Architecture (₹0 Budget)

Challenge	Solution
No GPU budget	Google Colab Free T4 (15GB VRAM)
Wav2Lip blurs face at 480p	GFPGAN v1.4 per-frame restoration + 2x upscale
Hindi audio too long for slot	Per-segment atempo capped at 1.15x max
Wav2Lip trained at 25fps, video at 60fps	Convert to 25fps before inference
XTTS broken on Python 3.12	Microsoft Neural edge-tts — free, natural, any Python version
Best voice reference selection	RMS energy window scan — picks clearest 10s of speech
MuseTalk/VideoReTalking need Python 3.10	Documented in limitations — would use Docker in production

---

What I'd Improve With More Time

1. MuseTalk on Docker (Python 3.10) — ByteDance 2024 model handles rotating faces, natural head movement. This is the single biggest quality improvement available.

2. XTTS voice cloning — Clone the original speaker's voice for authentic dubbing instead of a generic neural voice.

3. LLM-guided translation — Replace Google Translate with Claude API using a domain prompt that preserves syllable count and uses natural conversational Hindi for nanny training content.

4. SyncNet confidence scoring — Compute lip-sync confidence on output, auto-retry with different parameters if below threshold.

5. Montreal Forced Aligner — Phoneme-level alignment between Hindi audio and video frames for sub-frame accuracy.