Mixing and Export

Mixing & Export

Section: Pipeline · Lip-Sync-and-Synthesis · Home · Desktop-UI-Architecture

Mixing & Export reconciles source audio (ambiance, music, or normalized audio) with TTS takes and optional lip-sync video. A MixPlan describes layering, ducking, and panning for preview or final export.

Mixing Architecture

Mixing is centered around the MixPlan, a domain record that encapsulates all required metadata for combining audio stems and speech artifacts.

The MixPlan Entity

The MixPlan contains a collection of MixSpeechClip objects and MixDuckRegion objects. It tracks gain settings for the source audio and dubbed speech, as well as lead/tail seconds for audio ducking.

classDiagram
    class MixPlan {
        +Guid ProjectId
        +ArtifactKind SourceAudioKind
        +double SourceGainDb
        +double DubbedSpeechGainDb
        +double DuckingGainDb
        +List~MixSpeechClip~ SpeechClips
        +List~MixDuckRegion~ DuckingRegions
        +bool RestoreOriginalPan
        +bool ApplyTimbrePolish
    }
    class MixSpeechClip {
        +int SegmentIndex
        +double StartSeconds
        +double EndSeconds
        +string TakeRelativePath
        +bool IsSilentGap
    }
    class MixDuckRegion {
        +int SegmentIndex
        +double GainDb
        +double StartSeconds
        +double EndSeconds
    }
    MixPlan *-- MixSpeechClip
    MixPlan *-- MixDuckRegion

Loading

MixPlan Building Logic

The MixPlanBuilder service in the Application layer is responsible for creating a MixPlan from a MixPlanBuildRequest. It selects the most appropriate source audio artifact (prioritizing ambiance stems over normalized audio) and maps TTS takes to their corresponding segments.

Key logic in MixPlanBuilder includes:

• Artifact Selection: Choosing between ArtifactKind.Ambiance and ArtifactKind.NormalizedAudio.
• Ducking Resolution: Automatically calculating ducking gain based on the source artifact type (defaulting to 0dB for ambiance and -13dB for original mixes).
• Lip-Sync Fallback: Prioritizing ArtifactKind.LipSyncTake if available on disk, otherwise falling back to standard ArtifactKind.TtsTake.
• Stale Take Detection: Identifying if a TTS take is stale based on whether the underlying translation text has changed.

Audio Ducking & Gains

Ducking ensures that background audio is lowered while dubbed speech is playing. The system calculates these regions based on the duration of speech clips and user-defined lead/tail buffers.

Setting	Default Value	Description
DuckingLeadSeconds	0.05s	Buffer before speech starts to begin ducking.
DuckingTailSeconds	0.18s	Buffer after speech ends before restoring volume.
DuckingGainDb (Ambiance)	0dB	Default ducking applied to clean ambiance/stems.
DuckingGainDb (Original)	-13dB	Default ducking applied when mixing over original audio.
Gain Range	-96dB to +24dB	The valid boundary for all gain settings.

Preview Mixing

The PreviewMixViewModel manages the user interface and state for generating real-time mix previews. It allows users to define a specific range within the media for previewing.

flowchart TD
    UI[Preview UI] -->|Range/Gain Input| VM[PreviewMixViewModel]
    VM -->|CanGenerateCheck| Request[PreviewMixStageRequest]
    Request -->|Execute| Builder[MixPlanBuilder]
    Builder -->|Generate| Plan[MixPlan]
    Plan -->|Result| VM
    VM -->|Update| Status[Preview Status & Warnings]

Loading

The preview system validates if a mix can be generated based on the presence of playable source media, a valid transcript revision, and a valid time range where EndSeconds > StartSeconds.

Export & Muxing

The Export system handles the creation of the final media file. This involves "muxing" (multiplexing) where the reconciled audio mix is combined with the original or repaired video stream.

Export Workflow

The export process relies on MixPlan but extends it to full-file requirements, including:

• Muxing: Using FFmpeg via the FfmpegVideoRecomposer to combine streams.
• Output Verification: Ensuring the final file meets technical requirements.
• Lip-Sync Compositing: If lip-synced segments are available, they are composited into the video during the export phase. Warnings are surfaced if synthesis takes are not ready for compositing.

Error Handling & Warnings

The system identifies several conditions that may compromise the quality of the mix or export. These are represented by MixPlanWarningCode.

Warning Code	Description
MissingTake	No synthesized speech exists for a segment.
StaleTake	The translation text has changed since the TTS was generated.
InvalidTake	The take status is not "Completed".
MissingTakeArtifact	The database record for a take exists but the audio file is missing.
LipSyncArtifactMissing	A lip-sync record exists but the file is missing; falls back to TTS audio.

Summary

The Mixing & Export system provides a robust framework for reconciling dubbed artifacts with source media. By utilizing a data-driven MixPlan, Trackdub ensures that both real-time previews and final exports follow consistent rules for gain, ducking, and panning. The system prioritizes artifact integrity through stale-detection and disk-verification, ensuring that the final output accurately reflects the current state of the project's transcripts and translations.