Repository navigation
Architecture From Bits to Two Eyes
SyLC is a pipeline, not a codec with a window attached.
The difficult part is preserving meaning — view identity, timestamps, colour and ownership — while every stage runs on a different clock.
The application is written in Python for orchestration and interface work, with the playback hot path moved into native code. The central native module combines demuxing, decoder integration and Direct3D 11 rendering behind a pybind11 boundary; edge264.dll supplies the extended H.264/MVC engine; libavcodec handles native HEVC paths; libmpv owns audio and compatibility playback.
┌──────────────────────────────┐
MKV / MP4 / MOV ───────►│ native or libavformat demux │
SSIF / M2TS / BDMV ────►│ Blu-ray resolver + demux │
MV-HEVC ────────────────►│ native avcodec source │
└──────────────┬───────────────┘
│ access units / view packets
┌───────────────┴────────────────┐
│ │
▼ ▼
edge264 H.264/MVC avcodec HEVC
base + dependent views one or two views
│ │
└───────────────┬────────────────┘
│ timestamped YUV eye planes
▼
native D3D11 renderer
colour · scale · subtitles · stereo
│
┌──────────────────┼──────────────────┐
▼ ▼ ▼
embedded view detached / projectors Cast/export taps
audio streams ───────────────► libmpv ───────────────► master clock
The diagram is linear for readability. The implementation is deliberately concurrent because optical I/O, CPU decode, GPU rendering and audio presentation have no interest in taking turns politely.
The C++ demuxer built on libmatroska/libebml locates MVC tracks, separates base and dependent NAL units, and holds compressed storage until the decoder's completion callbacks have finished with it. A zero-copy ring avoids needless compressed-data churn.
For normal MVC playback the same native module resolves edge264.dll, validates its C ABI, feeds access units and releases the Python GIL for the whole native operation. The older Python/ctypes bridge remains a safety fallback, not the preferred per-NAL toll booth.
A libavformat-backed demuxer extends native edge264 decoding to MP4, AVI, MOV, FLV, WebM and raw H.264 when the bundled FFmpeg libraries support the container. If the native path rejects a source, libmpv can take over rather than turning format purity into a blank screen.
The in-process HEVC source uses the bundled avcodec runtime. It supports 8-bit and 10-bit planes, software decoding and D3D11VA copy-back with automatic software fallback. For MV-HEVC it requests both view IDs and pairs them by PTS before exposing the ordinary stereo-frame interface.
Extending H.264 to MVC required more than teaching the parser one new NAL type.
| Obligation | Why it exists |
|---|---|
| Inter-view prediction | The dependent eye may reference pictures from the base eye |
| Per-view decoded-picture buffers | Each eye has its own reference and reordering state |
| SPS ↔ subset-SPS fallback | Real discs do not always repeat parameter sets where a tidy sample would |
| PPS inheritance | Dependent-view slices may rely on parameter-set relationships established elsewhere |
| Picture-order pairing | Base and dependent completions can emerge offset around B-frame GOPs |
| Bounded recovery | Buffer pressure must advance or conceal safely instead of deadlocking both eyes |
The output contract is a matched frame carrying base and dependent planes. A single decoded eye may be useful for thumbnails or fallback, but it is not promoted as a valid stereo pair.
A naive player decodes a frame and immediately displays it. That works until a key frame costs 100 ms in a 24 fps film with a 41.7 ms frame interval.
SyLC separates responsibilities:
| Worker | Responsibility |
|---|---|
| Demux/decode path | Keep compressed input and native workers productive, publish completed frames |
| Presentation path | Select the frame matching the media clock and absorb short decode spikes |
| Audio/libmpv | Provide audible output and the master clock |
| UI thread | Handle controls and state changes without becoming the video scheduler |
| D3D11 renderer | Upload planes, convert colour, assemble stereo and present |
Back-pressure bounds memory and stops decode from running arbitrarily far ahead. Releasing the GIL around native demux/decode work lets Python remain the conductor without insisting on playing every instrument.
The native decoder selects a worker count from the available CPU budget. SYLC_EDGE264_THREADS exists for controlled qualification, not as a universal “more is faster” knob. Reference-picture dependencies are unmoved by enthusiasm.
The native renderer is the single image-forming path. Each visible surface owns the D3D11 resources appropriate to its device and window, while shared state keeps stereo parameters, subtitle content and optional depth geometry coherent.
The same decoded pair can feed:
- the embedded control/preview view;
- the detached frame-packed window;
- two Dual Projector eye windows;
- Glasses full-SBS output;
- the encoder surface used by SyLC Cast;
- the MV-HEVC exporter through its own isolated decode job.
“Same pair” does not mean “one texture object everywhere.” D3D11 resources are device-local. It means every consumer sees the same versioned frame/geometry decision instead of independently recomputing what the film meant.
For 2D→3D, renderer surfaces share one process-wide inference service:
- one surface temporarily holds the input-leader lease;
- a downscaled frame is read back through a non-blocking staging ring;
- the GPU runs Depth Anything V3 while CPU stages refine the previous result;
- optical flow transports temporal state in video time;
- the service publishes an immutable, sequence-numbered geometry map;
- each surface uploads only a newer map;
- a slow or failed inference never blocks the film.
This is intentionally orthogonal to source decode. Once synthesized views are bound where authored left/right planes would normally live, subtitles, projectors and Cast continue through their existing routes.
The full depth story deserves — and has — its own page: Real-time 2D→3D AI pipeline.
SyLC's most important boundaries are not module boundaries; they are promises about failure.
- A failed optional depth engine disarms synthesis and leaves normal 2D playback running.
- A failed native H.264 route can fall back to libmpv where that still produces an honest picture.
- A missing MVC dependent view may fall back to labelled 2D, never counterfeit stereo.
- D3D11 or device reset paths detach optional surfaces rather than corrupting the shared frame state.
- Model and TensorRT artifacts are verified before trust.
- Export uses independent decoder instances and cannot commandeer the live playback session.
Graceful degradation is not the same as silent degradation. The interface and logs should say which route is active.
| Area | Main source |
|---|---|
| Application orchestration and routing | SyLC_3D_Player.py |
| MVC thread/fallback decoder | mvc_decoder.py |
| Native demuxer, decoder wrapper and renderer | mvc_realtime_demuxer/ |
| Extended H.264/MVC decoder | edge264/ |
| Native HEVC and MV-HEVC source | lavf_hevc_source.py |
| Blu-ray resolution and ISO lifecycle | bluray_disc.py |
| PGS extraction and display |
subtitle_manager.py, pgs_subtitle_parser.py
|
| Shared 2D→3D service |
synth3d_matting_service.py and native depth engine sources |
| Quest sender | cast_sender/ |
| Spatial-video export |
mvhevc_exporter.py, vexu_injector.py
|
Build prerequisites and commands live in Building from source. Correctness gates live in Engineering standards and validation.
Start
Watch & preserve
Under the hood