Media Playback System

Media Playback System

Section: Desktop App · Timeline-and-Waveform-Editor · Home · Cloud-API-and-Job-Management

Playback is layered: playback backends in Media.Playback, UI coordination in Avalonia. Primary engine is libmpv; LibVLC is the fallback. Supports synchronized original video + dubbed take preview.

Core Architecture

The system is built on a set of interfaces that abstract the underlying media engine. This allows the application to remain agnostic of whether it is using libmpv or LibVLC.

Component Overview

Component	Description
IMediaPlaybackBackend	The base interface for controlling a media player (Load, Play, Pause, Seek).
ICompositedPlaybackBackend	An extension of the backend that supports external audio composition for dub previews.
AvaloniaDubPlaybackCoordinator	Coordinates between the UI, the media backend, and the project state (TTS takes, segments).
LibMpvCompositedPlaybackBackend	The primary concrete implementation using the libmpv library.

Playback Flow Diagram

This diagram illustrates how the coordination layer interacts with the backend to manage media.

flowchart TD
    UI[Avalonia UI] --> Coord[DubPlaybackCoordinator]
    Coord --> Backend[ICompositedPlaybackBackend]
    Backend --> LibMpv[libmpv Native]
    Coord --> State[PlaybackState]
    State -.-> UI

Loading

Synchronization and Dub Preview

A critical feature of the system is the ability to preview dubbed audio in place of or alongside the original video's audio. This is managed by the AvaloniaDubPlaybackCoordinator, which tracks which AI-generated "takes" are currently active for specific transcript segments.

Synchronized Audio Switching

When playback reaches a segment that has an associated dubbed take, the coordinator can instruct the backend to switch audio sources or mix them.

• Segment Tracking: The system uses TranscriptSegment indices to identify playback positions.
• TTS Integration: Generated audio artifacts (TTS takes) are mapped to these segments.
• Real-time Updates: The PlaybackState record is updated via observables to reflect current position, duration, and status (Playing, Paused, Buffering).

sequenceDiagram
    participant UI as UI ViewModel
    participant Coord as DubPlaybackCoordinator
    participant Backend as libmpv Backend
    UI->>Coord: SetActiveDubTake(segmentId, takeId)
    Coord->>Backend: LoadAudioOverlay(path)
    Coord->>Backend: Play()
    loop Every Frame
        Backend-->>Coord: Position Update
        Coord-->>UI: Update Timeline
    end

Loading

libmpv Implementation

The LibMpvCompositedPlaybackBackend serves as the primary implementation. It handles native interop with libmpv.dll (on Windows) or shared objects on Linux/macOS.

Key Implementation Details

• Native Layout: Playback natives are expected in native/{rid}/.
• Async Operations: Uses ValueTask for non-blocking state transitions.
• Event Handling: Forwards native events (property changes, log messages) into .NET observables.

Playback State Management

Playback state is represented by an immutable record to ensure thread safety and consistent UI updates through ReactiveUI.

Data Model: PlaybackState

Field	Type	Description
Status	PlaybackStatus	Current state: Idle, Buffering, Playing, Paused.
Position	TimeSpan	Current playback timestamp.
Duration	TimeSpan	Total duration of the loaded media.
Volume	double	Current volume level (0.0 to 1.0).

UI Integration: Timeline and Waveforms

The AvaloniaProjectDisplayBuilder builds the visual representation of the playback data for the Avalonia UI. It creates TimelineTrackViewModel and TimelineClipViewModel objects that correspond to the media duration and segment boundaries.

• Waveform Alignment: Segments are aligned with waveform data using WaveformSegmentBoundary.
• Visual Status: Segments display their current dubbing status (e.g., "Dub not generated", "Completed") which informs the user's playback expectations.

Summary

The Media Playback System provides a robust, synchronized foundation for reviewing AI-assisted dubbing. By abstracting the native playback engines behind a composited backend interface, it enables seamless switching between source and dubbed audio while maintaining high-performance video synchronization across Windows, macOS, and Linux.