🎬 Authentic-Dubber

Towards Authentic Movie Dubbing with Retrieve-Augmented
Director-Actor Interaction Learning

Rui Liu¹, Yuan Zhao¹, Zhenqi Jia¹

¹ S2LAB, College of Computer Science, Inner Mongolia University, Hohhot, China

(Accepted by AAAI 2026)

🔖 Abstract

The automatic movie dubbing model generates vivid speech from given scripts, replicating a speaker's timbre from a brief timbre prompt while ensuring lip-sync with the silent video. Existing approaches simulate a simplified workflow where actors dub directly without preparation, overlooking the critical director-actor interaction. In contrast, authentic workflows involve a dynamic collaboration: directors actively engage with actors, guiding them to internalize the context cues, specifically emotion, before performance. To address this issue, we propose a new Retrieve-Augmented Director-Actor Interaction Learning scheme to achieve authentic movie dubbing, termed Authentic-Dubber, which contains three novel mechanisms: (1) We construct a multimodal Reference Footage library to simulate the learning footage provided by directors. Note that we integrate Large Language Models (LLMs) to achieve deep comprehension of emotional representations across multimodal signals. (2) To emulate how actors efficiently and comprehensively internalize director-provided footage during dubbing, we propose an Emotion-Similarity-based Retrieval-Augmentation strategy. This strategy retrieves the most relevant multimodal information that aligns with the target silent video. (3) We develop a Progressive Graph-based speech generation approach that incrementally incorporates the retrieved multimodal emotional knowledge, thereby simulating the actor's final dubbing process. The above mechanisms enable the Authentic-Dubber to faithfully replicate the authentic dubbing workflow, achieving comprehensive improvements in emotional expressiveness. Both subjective and objective evaluations on the V2C Animation benchmark dataset validate the effectiveness.

🔍 Overview

💥 Updates

• 2025-12-03: We released the training and inference scripts.
• 2025-12-03: The Speech Demo Page is now live.
• 2025-11-18: 🎉 Our paper has been accepted by AAAI 2026!

🏁 Installation

We recommend using Anaconda to manage the environment.

# Clone the repository
git clone https://github.com/AI-S2-Lab/Authentic-Dubber.git
cd Authentic-Dubber

# Create environment (Example)
conda create -n authentic_dubber python=3.9
conda activate authentic_dubber

# Install dependencies
pip install -r requirements.txt