Skip to content

Architecture From Bits to Two Eyes

Symphoenix edited this page Aug 6, 2026 · 1 revision

Architecture: from bits to two eyes

SyLC is a pipeline, not a codec with a window attached.
The difficult part is preserving meaning — view identity, timestamps, colour and ownership — while every stage runs on a different clock.

The application is written in Python for orchestration and interface work, with the playback hot path moved into native code. The central native module combines demuxing, decoder integration and Direct3D 11 rendering behind a pybind11 boundary; edge264.dll supplies the extended H.264/MVC engine; libavcodec handles native HEVC paths; libmpv owns audio and compatibility playback.


Top-level map

                         ┌──────────────────────────────┐
MKV / MP4 / MOV ───────►│ native or libavformat demux │
SSIF / M2TS / BDMV ────►│ Blu-ray resolver + demux    │
MV-HEVC ────────────────►│ native avcodec source       │
                         └──────────────┬───────────────┘
                                        │ access units / view packets
                        ┌───────────────┴────────────────┐
                        │                                │
                        ▼                                ▼
              edge264 H.264/MVC                  avcodec HEVC
              base + dependent views             one or two views
                        │                                │
                        └───────────────┬────────────────┘
                                        │ timestamped YUV eye planes
                                        ▼
                              native D3D11 renderer
                         colour · scale · subtitles · stereo
                                        │
                     ┌──────────────────┼──────────────────┐
                     ▼                  ▼                  ▼
                embedded view    detached / projectors   Cast/export taps

audio streams ───────────────► libmpv ───────────────► master clock

The diagram is linear for readability. The implementation is deliberately concurrent because optical I/O, CPU decode, GPU rendering and audio presentation have no interest in taking turns politely.


Source routing

Matroska and Blu-ray MVC

The C++ demuxer built on libmatroska/libebml locates MVC tracks, separates base and dependent NAL units, and holds compressed storage until the decoder's completion callbacks have finished with it. A zero-copy ring avoids needless compressed-data churn.

For normal MVC playback the same native module resolves edge264.dll, validates its C ABI, feeds access units and releases the Python GIL for the whole native operation. The older Python/ctypes bridge remains a safety fallback, not the preferred per-NAL toll booth.

H.264 outside Matroska

A libavformat-backed demuxer extends native edge264 decoding to MP4, AVI, MOV, FLV, WebM and raw H.264 when the bundled FFmpeg libraries support the container. If the native path rejects a source, libmpv can take over rather than turning format purity into a blank screen.

HEVC and MV-HEVC

The in-process HEVC source uses the bundled avcodec runtime. It supports 8-bit and 10-bit planes, software decoding and D3D11VA copy-back with automatic software fallback. For MV-HEVC it requests both view IDs and pairs them by PTS before exposing the ordinary stereo-frame interface.


The MVC decoder's extra obligations

Extending H.264 to MVC required more than teaching the parser one new NAL type.

Obligation Why it exists
Inter-view prediction The dependent eye may reference pictures from the base eye
Per-view decoded-picture buffers Each eye has its own reference and reordering state
SPS ↔ subset-SPS fallback Real discs do not always repeat parameter sets where a tidy sample would
PPS inheritance Dependent-view slices may rely on parameter-set relationships established elsewhere
Picture-order pairing Base and dependent completions can emerge offset around B-frame GOPs
Bounded recovery Buffer pressure must advance or conceal safely instead of deadlocking both eyes

The output contract is a matched frame carrying base and dependent planes. A single decoded eye may be useful for thumbnails or fallback, but it is not promoted as a valid stereo pair.


Decode, present, repeat — on separate clocks

A naive player decodes a frame and immediately displays it. That works until a key frame costs 100 ms in a 24 fps film with a 41.7 ms frame interval.

SyLC separates responsibilities:

Worker Responsibility
Demux/decode path Keep compressed input and native workers productive, publish completed frames
Presentation path Select the frame matching the media clock and absorb short decode spikes
Audio/libmpv Provide audible output and the master clock
UI thread Handle controls and state changes without becoming the video scheduler
D3D11 renderer Upload planes, convert colour, assemble stereo and present

Back-pressure bounds memory and stops decode from running arbitrarily far ahead. Releasing the GIL around native demux/decode work lets Python remain the conductor without insisting on playing every instrument.

The native decoder selects a worker count from the available CPU budget. SYLC_EDGE264_THREADS exists for controlled qualification, not as a universal “more is faster” knob. Reference-picture dependencies are unmoved by enthusiasm.


One renderer, many surfaces

The native renderer is the single image-forming path. Each visible surface owns the D3D11 resources appropriate to its device and window, while shared state keeps stereo parameters, subtitle content and optional depth geometry coherent.

The same decoded pair can feed:

  • the embedded control/preview view;
  • the detached frame-packed window;
  • two Dual Projector eye windows;
  • Glasses full-SBS output;
  • the encoder surface used by SyLC Cast;
  • the MV-HEVC exporter through its own isolated decode job.

“Same pair” does not mean “one texture object everywhere.” D3D11 resources are device-local. It means every consumer sees the same versioned frame/geometry decision instead of independently recomputing what the film meant.


The optional depth branch

For 2D→3D, renderer surfaces share one process-wide inference service:

  1. one surface temporarily holds the input-leader lease;
  2. a downscaled frame is read back through a non-blocking staging ring;
  3. the GPU runs Depth Anything V3 while CPU stages refine the previous result;
  4. optical flow transports temporal state in video time;
  5. the service publishes an immutable, sequence-numbered geometry map;
  6. each surface uploads only a newer map;
  7. a slow or failed inference never blocks the film.

This is intentionally orthogonal to source decode. Once synthesized views are bound where authored left/right planes would normally live, subtitles, projectors and Cast continue through their existing routes.

The full depth story deserves — and has — its own page: Real-time 2D→3D AI pipeline.


Failure boundaries

SyLC's most important boundaries are not module boundaries; they are promises about failure.

  • A failed optional depth engine disarms synthesis and leaves normal 2D playback running.
  • A failed native H.264 route can fall back to libmpv where that still produces an honest picture.
  • A missing MVC dependent view may fall back to labelled 2D, never counterfeit stereo.
  • D3D11 or device reset paths detach optional surfaces rather than corrupting the shared frame state.
  • Model and TensorRT artifacts are verified before trust.
  • Export uses independent decoder instances and cannot commandeer the live playback session.

Graceful degradation is not the same as silent degradation. The interface and logs should say which route is active.


Where to read the implementation

Area Main source
Application orchestration and routing SyLC_3D_Player.py
MVC thread/fallback decoder mvc_decoder.py
Native demuxer, decoder wrapper and renderer mvc_realtime_demuxer/
Extended H.264/MVC decoder edge264/
Native HEVC and MV-HEVC source lavf_hevc_source.py
Blu-ray resolution and ISO lifecycle bluray_disc.py
PGS extraction and display subtitle_manager.py, pgs_subtitle_parser.py
Shared 2D→3D service synth3d_matting_service.py and native depth engine sources
Quest sender cast_sender/
Spatial-video export mvhevc_exporter.py, vexu_injector.py

Build prerequisites and commands live in Building from source. Correctness gates live in Engineering standards and validation.

Clone this wiki locally