Diarization and Speaker Management

Diarization & Speaker Management

Section: Pipeline · Transcription-and-VAD · Home · Translation-Engine

Diarization identifies speakers, partitions turns, and keeps identities attached through translation/TTS. Local ONNX models (e.g. SortFormer) run on-device after VAD.

Architecture and Pipeline Integration

The Diarization system is implemented as a formal pipeline stage, SpeakerDiarizationStage, which implements the ITranscriptGenerationStage interface. It interacts with the SpeakerAssignmentService to process audio artifacts and generate DiarizationResult data, which includes a list of identified speakers and their respective temporal turns.

Data Flow

The following diagram illustrates the flow of data through the Diarization stage within the broader transcript generation context.

flowchart TD
    Start[Start Diarization Stage] --> CheckEnabled{Is Diarization Enabled?}
    CheckEnabled -- No --> DefaultPlan[Build Default Region Plan]
    CheckEnabled -- Yes --> GetAudio[Retrieve Diarization Audio Artifact]
    
    GetAudio --> EngineRun[Execute Diarization Engine]
    EngineRun --> ResultCheck{Result Generated?}
    
    ResultCheck -- Yes --> Persist[Persist Speaker Turns & Artifacts]
    Persist --> BuildPlan[Build Transcript Region Plan]
    ResultCheck -- No --> BuildPlan
    
    DefaultPlan --> End[Return Updated Context]
    BuildPlan --> End

Loading

This diagram shows how the stage decides between running a full AI inference session or falling back to a default region plan based on user preferences.

Key Components

Component	Responsibility
SpeakerDiarizationStage	Orchestrates the execution of diarization logic within the transcription pipeline.
SpeakerAssignmentService	High-level service managing speaker creation, diarization reruns, and assignment persistence.
DiarizationResult	Data contract containing the identified ProjectSpeaker list and SpeakerTurn records.
TranscriptRegionPlan	A temporal mapping used to align diarized turns with the overall transcript timeline.
DiarizationStageHandler	Manages model-specific logic, including model downloading and SHA-256 hash verification.

Diarization Engine and Models

Trackdub utilizes ONNX-based engines for diarization. The primary engine referenced is SortFormer, which is responsible for partitioning speech into speaker-specific segments. The engine requires a validated model cache and specific execution providers (such as Windows ML or DirectML) to run efficiently on local hardware.

SortFormer Diarization

The SortFormer engine processes "Diarization Audio Artifacts" (often normalized 16kHz mono audio). It generates speaker turns that include StageRunId metadata to maintain traceability within the project database.

Model Governance and Safety

Every model used for diarization must have a manifest entry. The DiarizationStageHandler ensures that model files match expected SHA-256 hashes before execution. If a model is missing, the system reports a RequiredDiarizationModelStatus indicating whether the model is available or can be auto-downloaded.

Speaker and Voice Management

Once diarization is complete, speakers are registered as ProjectSpeaker entities. Management includes renaming speakers, assigning specific voice profiles for TTS, and managing reference clips for voice cloning.

Workflow Sequence

sequenceDiagram
    participant UI as User Interface
    participant SS as SpeakerAssignmentService
    participant RW as TranscriptArtifactWriter
    participant AS as ArtifactStore

    UI->>SS: Request Diarization Rerun
    SS->>SS: Run Diarization Engine
    SS->>RW: Write Diarization Artifact
    RW->>AS: Store JSON Result (pipeline/diarization-result-*)
    SS->>UI: Return Refreshed Project State
    
    UI->>SS: Rename Speaker ("Speaker 1" -> "Host")
    SS->>UI: Update Display Name
    
    UI->>SS: Assign Voice (Speaker A -> "af_heart")
    SS->>UI: Persist Voice Assignment

Loading

The sequence shows the interaction between user requests and the underlying services to update speaker metadata and voice mappings.

Speaker Metadata

Feature	Description
Speaker ID	A unique GUID assigned to each distinct speaker found in the audio.
Display Name	A user-editable string (e.g., "Host", "Guest") for better identification.
Voice Assignment	Mapping a speaker to a specific TTS voice variant (e.g., "af_heart").
Reference Clips	Short audio segments extracted from the source used for speaker analysis or voice cloning.

Artifact Preservation and Persistence

Diarization results are persisted as project artifacts with the ArtifactKind.DiarizationResult. To avoid overwriting stable data during partial runs or reruns, results are written to run-scoped paths (e.g., pipeline/diarization-result-{guid}.json). The TranscriptArtifactWriter is responsible for these atomic updates, ensuring that provenance metadata (e.g., "generated-diarization") is correctly recorded.

Execution Statuses

The stage tracks various execution outcomes to inform the UI and downstream processes:

• Completed: Diarization ran successfully and produced speaker turns.
• Skipped: Diarization was disabled by the user or no speech regions were found.
• Failed: The engine encountered an error or a required model was unavailable.
• Canceled: The user aborted the process during execution.

Conclusion

Diarization & Speaker Management is a foundational component of the Trackdub dubbing workstation. By providing an automated way to partition audio and manage speaker identities using local ONNX models, the system enables high-quality, multi-speaker dubbing workflows while maintaining strict data privacy. The integration with artifact persistence and voice assignment ensures that speaker identities remain consistent throughout the project lifecycle, from initial transcription to final mix and export.