Text to Speech

Text-to-Speech (TTS)

Section: Pipeline · Translation-Engine · Home · Lip-Sync-and-Synthesis

TTS synthesizes dubbed audio from translated segments using local ONNX models (e.g. Kokoro, Qwen). It follows translation and precedes mix/export, producing immutable audio artifacts for preview and export.

Architecture and Components

The TTS system is structured with a clear separation between application orchestration and inference implementation. The TtsWorkflow manages the high-level logic, while specific engines like KokoroTtsEngine and Qwen3TtsEngine handle the heavy lifting of neural synthesis.

High-Level TTS Flow

The following diagram illustrates the flow from a translated segment to a generated TTS artifact.

flowchart TD
    A[Translated Segment] --> B{TTS Workflow}
    B --> C[Select Engine]
    C --> D[KokoroTtsEngine]
    C --> E[Qwen3TtsEngine]
    D --> F[Neural Synthesis]
    E --> F
    F --> G[Audio Artifact]
    G --> H[Timing Reconciliation]

Loading

The workflow selects an engine based on the available model manifest and hardware capabilities.

Key Components

Component	Description
TtsWorkflow	Orchestrates the synthesis process, including segment selection and artifact persistence.
KokoroTtsEngine	Implements neural TTS using the Kokoro model via ONNX Runtime.
Qwen3TtsEngine	Implements neural TTS using the Qwen model architecture.
InferenceSessionPool	Manages ONNX Runtime sessions to optimize memory and performance.
TtsSegmentState	Tracks the status (Completed, Stale, Failed) and metadata of a synthesized segment.

Synthesis Engines

Trackdub employs specific ONNX wrappers for different neural architectures. These engines are responsible for transforming text tokens into PCM audio samples.

Kokoro TTS Engine

The KokoroTtsEngine utilizes ONNX Runtime to execute the Kokoro model. It manages model sessions and handles the conversion of input text into audio waves.

Qwen TTS Engine

Similar to Kokoro, the Qwen3TtsEngine provides a concrete implementation for the Qwen architecture. It integrates with the InferenceSessionPool to maintain warm sessions for faster inference across multiple segments.

Data Models and Status Tracking

The state of synthesis for each segment is carefully tracked to support resumable execution and user feedback in the UI.

TTS Segment State Fields

Field	Type	Description
Status	TtsTakeStatus	Current state: NotRun, Completed, Stale, or Failed.
IsStale	bool	Indicates if the source text has changed since the last synthesis.
ArtifactRelativePath	string	Filesystem path to the generated audio artifact.
DurationSeconds	double	The length of the synthesized audio.
StretchRatioApplied	double	The ratio used to fit the audio into the original segment timing.

Synthesis Lifecycle Sequence

The following sequence diagram shows the interaction between the UI, Application, and Inference layers during synthesis.

sequenceDiagram
    participant UI as "Avalonia UI"
    participant App as "TtsWorkflow"
    participant Inf as "TtsEngine (ONNX)"
    participant Pool as "SessionPool"

    UI->>App: Trigger Run (Global or Segment)
    App->>App: Capture Execution Snapshot
    App->>Inf: SynthesizeAsync(Text, Voice)
    Inf->>Pool: AcquireSession(ModelId)
    Pool-->>Inf: Session Ready
    Inf->>Inf: Perform Neural Inference
    Inf-->>App: Audio Samples
    App->>App: Persist Artifact (WAV)
    App-->>UI: Update Segment State

Loading

Synthesis must be user-triggered; the app does not auto-start synthesis when media loads to preserve resources.

Timing Reconciliation

Synthesized audio rarely matches the exact duration of the original speech. Trackdub includes logic to stretch or compress speech to fit the target duration.

• Stretch Ratios: Ratios are clamped to avoid unnatural sounding artifacts.
• Unsafe Ratios: If a stretch ratio is considered unsafe, the system may skip the stage or preserve the original take.
• Phoneme Alignment: For higher quality, the system supports phoneme-aligned lip sync (M22) to improve timing at the viseme level.

Implementation Details

Artifact Preservation

Trackdub adheres to the "no fake readiness" rule. A stage is only marked as Completed if the audio samples were successfully generated and persisted to disk. If a stage fails, prior usable artifacts are preserved to allow for manual recovery.

Commercial Safety

The synthesis stage is governed by the model manifest. If "Commercial Mode" is active, the runtime planner blocks any TTS engine that relies on non-commercial or unverified model weights.

Summary

Text-to-Speech Synthesis in Trackdub provides the critical bridge between translated text and audible dubbing. By leveraging local ONNX engines like Kokoro and Qwen, the system ensures privacy and performance. The architecture supports complex timing reconciliation and maintains strict artifact governance to provide a reliable, workstation-grade experience. Technical truth is prioritized, ensuring that the UI accurately reflects the state of every synthesized segment.