Lip Sync and Synthesis

Lip Sync & Synthesis

Section: Pipeline · Text-to-Speech · Home · Mixing-and-Export

Lip sync aligns dubbed audio timing to mouth movements; lip synthesis can regenerate facial/mouth video. Both require manifest-gated commercial-safe models when used in shipping lanes.

Architecture and Workflow

The system follows a strict execution order within the application pipeline. Lip Sync (M22) occurs at the audio level to improve timing without modifying video frames, while Lip Synthesis (M23) repairs original footage by modifying facial regions to match the improved audio.

Pipeline Integration

The Lip Sync and Synthesis stages are integrated into the Trackdub.Application layer, utilizing orchestration workflows that manage stage planning, execution snapshots, and artifact preservation.

flowchart TD
    TTS[TTS Generation] --> M22[M22: Lip Sync Audio Level]
    M22 --> M23[M23: Video Lip Synthesis]
    M23 --> Preview[Preview Mix / Export]
    
    subgraph M22_Logic [M22: Audio Lip Sync]
        A1[Phoneme Alignment]
        A2[Timing Planning]
        A3[Audio Stretching]
    end
    
    subgraph M23_Logic [M23: Video Synthesis]
        S1[Face Detection]
        S2[Landmark Tracking]
        S3[Mouth Repair]
    end
    
    M22 --- M22_Logic
    M23 --- M23_Logic

Loading

Lip Sync (M22)

The primary goal of M22 is to improve dubbed timing at the phoneme/viseme level. It utilizes forced alignment to map translated text to audio features, allowing for conservative audio stretching that matches the original speaker's cadence.

Key Components

• IForcedAligner: Interface for models that emit phoneme/token logits.
• Wav2Vec2CtcForcedAligner: A commercial-safe ONNX implementation based on Facebook's wav2vec2 models.
• LipSyncWorkflow: Orchestrates the alignment process, including phoneme inventory mapping and timing planning.

Logic Flow

1. Phoneme Alignment: The system extracts phoneme/token logits from the dubbed audio.
2. Timing Planning: It creates a PhonemeTimingPlan comparing dubbed phonemes to source speaker cadence.
3. Conservative Stretching: The system applies IPhonemeStretchService to adjust audio, prioritizing vowels and nuclei while clamping stretch ratios to avoid artifacts.

Lip Synthesis (M23)

M23 focuses on video-level repairs. Unlike full portrait animation, this stage modifies only the mouth/facial regions of the original footage to align with the audio processed in M22.

LatentSync Implementation

The active commercial model for video lip synthesis is LatentSync 1.6. It is an ONNX/ORT-backed engine that performs lip synthesis by conditioning video frame mouth regions on audio features.

sequenceDiagram
    participant App as LipSynthesisWorkflow
    participant Engine as LatentSyncEngine
    participant ONNX as ONNX Runtime
    
    App->>Engine: SynthesizeAsync(Video, Audio)
    Engine->>Engine: Detect Face & Landmarks
    Engine->>Engine: Prepare Audio Features
    Engine->>ONNX: Run Inference (LatentSync 1.6)
    ONNX-->>Engine: Generated Mouth Region
    Engine->>Engine: Composite & Reinsert
    Engine-->>App: Patched Video Artifact

Loading

Synthesis Quality Gates

To ensure "commercial-safe" output, the synthesis engine implements several skip/fail reasons:

Status	Description
Synthesized	Successful generation of patched video.
SkippedNoFace	No face detected in the segment.
SkippedNonFrontal	Face pose angle is too extreme for synthesis.
SkippedOccluded	Mouth or face is partially covered.
SkippedLowConfidence	Model confidence below threshold.

Model Governance and Safety

Trackdub enforces a "no fake readiness" policy. A Lip Sync or Synthesis provider is not considered ready until its manifest is verified, the model files are present, and the hardware environment is confirmed.

• Commercial Lane: Only commercial_use_verified: true models like LatentSync 1.6 are permitted for shipping.
• Experimental Lane: Providers using Python/CUDA or unproven ONNX exports are labeled experimental and require explicit runtime checks.
• Artifact Preservation: Original media and prior successful artifacts are never overwritten; synthesis creates new "patched" video artifacts.

Summary

The Lip Sync and Synthesis modules provide a sophisticated visual refinement layer to the Trackdub pipeline. By separating timing alignment (M22) from frame synthesis (M23), the project maintains original video authority while achieving realistic dubbed performance. All operations are manifest-gated and strictly local-first, ensuring privacy and commercial safety.