Transcription and VAD

Transcription & VAD

Section: Pipeline · Media-Ingest-and-Preparation · Home · Diarization-and-Speaker-Management

The Transcription & Voice Activity Detection (VAD) systems isolate speech and convert it to timed text. They typically run after audio normalization and before translation/TTS, using local ONNX models (e.g. Silero VAD, Whisper ASR) with honest readiness and artifact preservation.

Pipeline Orchestration

Transcription and VAD are implemented as discrete stages within the Trackdub pipeline. Each stage is designed to be resumable, immutable once started, and capable of preserving artifacts upon failure or skipping.

Logic Flow

The transcription workflow generally follows a sequence where VAD identifies speech regions, followed by Diarization (if enabled), and finally ASR to generate the text.

graph TD
    Start[Start Pipeline] --> Norm[Normalized Audio Artifact]
    Norm --> VAD_Stage[VAD Stage Handler]
    VAD_Stage --> Speech_Regions[Speech Region Metadata]
    Speech_Regions --> Diar_Stage[Diarization Stage]
    Diar_Stage --> ASR_Stage[ASR Generation Stage]
    ASR_Stage --> Transcript[Transcript Revision Artifact]
    
    subgraph "Transcription & VAD Subsystem"
    VAD_Stage
    Speech_Regions
    ASR_Stage
    end

Loading

The pipeline ensures that original media and prior successful artifacts are never overwritten in-place, recording exact provenance for every generated segment.

Voice Activity Detection (VAD)

VAD is handled primarily by the VadStageHandler, which utilizes the SileroVadSpeechRegionDetector. This stage identifies where speech starts and ends, allowing the ASR stage to focus only on relevant audio segments.

Silero VAD Implementation

The detector processes audio in chunks (typically 512 samples at 16kHz) and uses a stateful ONNX model to track speech probability. It manages transitions between "speech" and "silence" states using configurable thresholds.

Component	Description
SileroVadSpeechRegionDetector	The core engine wrapping the Silero ONNX model.
VadStageHandler	Orchestrates the VAD process within the application pipeline.
SpeechRegion	Data structure representing start/end timestamps of detected speech.

sequenceDiagram
    participant App as VadStageHandler
    participant Detector as SileroVadSpeechRegionDetector
    participant Model as Silero ONNX Model
    
    App->>Detector: DetectAsync(audioBuffer)
    activate Detector
    loop Every Audio Chunk
        Detector->>Model: Run Inference
        Model-->>Detector: Speech Probability
        Detector->>Detector: Update State (Speech/Silence)
    end
    Detector-->>App: List of SpeechRegions
    deactivate Detector

Loading

Automatic Speech Recognition (ASR)

The ASR system converts detected speech into text. This is handled by the AsrGenerationStage and executed by the WhisperOnnxAudioTranscriptionEngine.

Whisper ONNX Engine

The engine uses the OpenAI Whisper model architecture optimized for ONNX Runtime. It supports multiple model tiers (Tiny, Base, Small, Medium, Large) and can perform both transcription and timestamp alignment.

Key ASR Features

• Local Inference: Runs entirely on-device using onnxruntime or windows-ml.
• Model Provisioning: The AsrGenerationStage ensures that relevant Whisper models are downloaded and verified before execution.
• Timestamp Accuracy: Generates per-word or per-segment timestamps for precise alignment.

classDiagram
    class AsrGenerationStage {
        +HandleAsync(AsrRequest)
        -EnsureModelsAvailable()
    }
    class IAudioTranscriptionEngine {
        <<interface>>
        +TranscribeAsync(Audio)
    }
    class WhisperOnnxAudioTranscriptionEngine {
        -InferenceSessionPool
        -WhisperTokenizer
        +TranscribeAsync(Audio)
    }
    AsrGenerationStage ..> IAudioTranscriptionEngine
    IAudioTranscriptionEngine <|.. WhisperOnnxAudioTranscriptionEngine

Loading

Data Models and Artifacts

Transcription results are persisted as TranscriptRevision artifacts in the project SQLite database. Each revision contains a collection of TranscriptSegment objects.

Transcript Segment Structure

Field	Type	Description
Id	Guid	Unique identifier for the segment.
SegmentIndex	int	Sequence order within the transcript.
StartSeconds	double	Start time relative to audio start.
EndSeconds	double	End time relative to audio start.
Text	string	The transcribed speech content.
SpeakerId	Guid?	Optional link to a detected speaker.

Implementation Details

Model Manifests

Every model used in Transcription & VAD must be governed by a manifest. This manifest tracks the model_id, license, sha256 checksum, and commercial_allowed status. The system strictly blocks non-commercial models when the application is in commercial-safe mode.

// Example of ASR model provisioning in AsrGenerationStage
private async Task EnsureModelsAvailable(string modelAlias, CancellationToken ct)
{
    var model = await _modelInventory.GetByAliasAsync(modelAlias, ct);
    if (!model.IsAvailable)
    {
        await _modelDownloader.DownloadAsync(model.Id, ct);
    }
}

Summary

The Transcription & VAD system provides a robust, locally executed pipeline for converting audio into structured text. By combining state-of-the-art models like Silero and Whisper with a strict artifact-management architecture, Trackdub ensures high accuracy while maintaining user privacy and data integrity. This subsystem serves as the foundation for all subsequent translation and dubbing workflows.