feat: demux progressive (non-fragmented) MP4#4
Merged
Conversation
HLS with fMP4/CMAF segments loaded a single video frame and played no audio. The fMP4 demuxer emitted a fragment track-major — every video sample of a moof, then every audio sample — and with delivery pacing on (always, for HLS) the paced video burst spans the whole fragment duration in wall time, so the fragment audio behind it arrives a full fragment late. The decoder trims that audio as stale and its audio-gated presentation clock never starts: one frame, silence. Consume a moof mdat as a decode-time-ordered merge across its trun runs instead, interleaving the tracks at frame granularity the way the MPEG-TS demuxer's output naturally is. Decode order within each run is preserved; single-track fragments (split-stream fMP4 legs) are delivered byte-for-byte as before. Verified against ffmpeg-generated CMAF (H.264+AAC, 2s fragments): worst delivery-order PTS gap between tracks drops from ~2.1s (the fragment duration) to ~190ms (B-frame reorder depth), same access-unit counts and timestamps; a video-only fMP4 stream demuxes identically before and after. Rebuilt basis_media_native.dll (win-x64) and libbasis_media_native.so (arm64-v8a).
basis_avcc_to_annexb is bounds-checked, so an output larger than the buffer fails the conversion rather than overrunning — but with a 1-3 byte NAL length prefix, an access unit made of many small NALs could outgrow the fixed 64-byte headroom (each NAL trades its length prefix for a 4-byte start code) and be dropped. Size the buffer from the worst-case growth instead: a NAL is at least nal_len_size + 1 bytes, so growth is bounded by (4 - nal_len_size) * (size / (nal_len_size + 1)). For the common 4-byte prefix this stays exactly size + 64. Rebuilt basis_media_native.dll (win-x64) and libbasis_media_native.so (arm64-v8a).
A classic progressive MP4 (ftyp moov mdat with sample tables, no moof fragments; the most common recorded-file shape) played nothing: the demuxer was fragmented-MP4-only, so it announced the tracks from the moov and then found no trun runs to slice the mdat with. No error surfaced; the symptom was a silent black screen. Parse the classic sample tables (stts/ctts/stsc/stco/co64/stsz/stss) and walk the mdat as a forward stream, merging both tracks in file-offset order (the muxer's chunk interleave). The box loop now reads headers first and streams a progressive mdat sample-by-sample instead of buffering whole boxes, so file size is no longer bounded by the 256 MB box cap; fragment mdats keep the buffered path. Faststart layout (moov before mdat) is required by a one-way byte source; a trailing-moov file now reports a clear remux-with-faststart error instead of silence. Verified offline against the fragmented-only demuxer: the progressive recording that surfaced the bug goes from zero output to all 28 video + 42 audio samples; a generated 12 s faststart file demuxes fully with a worst A/V delivery-order gap of 233 ms (inside the paced clock's 250 ms audio floor); fMP4 output (CMAF HLS fixture and a fragmented video-only stream) is byte-identical before and after; a non-faststart file produces the remux error. Rebuilt basis_media_native.dll (win-x64) and libbasis_media_native.so (arm64-v8a).
A malformed (or crafted) stbl repeating stts/ctts/stsc/stss/stsz/stco/co64 overwrote the already-parsed table pointer, leaking the earlier allocation. First box of each kind now wins; the stsz guard keys on sample_count so the constant-size form (which allocates nothing) is covered too. Behaviour on well-formed files is unchanged - harness output is byte-identical on the progressive and CMAF fixtures. Rebuilt both binaries.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
A classic progressive MP4 —
ftyp moov mdatwith sample tables and nomooffragments, i.e. the most common recorded-file shape on the internet — played nothing. The demuxer was fragmented-MP4-only: it announced the tracks from themoov, then found no fragment runs to slice themdatwith, and emitted zero samples with no error. Symptom: silent black screen.This adds classic sample-table demuxing:
moov'sstts/ctts/stsc/stco/co64/stsz/stsstables are parsed per track (an fMP4 init segment carries these boxes with zero entries, which leaves them inert — the fragmented path is untouched).mdatis walked as a forward stream, merging both tracks in file-offset order — the muxer's own chunk interleave. The box loop now reads headers first and streams a progressivemdatsample-by-sample instead of buffering whole boxes, so file size is no longer bounded by the 256 MB box cap; fragmentmdats keep the buffered path.Verification
Offline A/B against the fragmented-only demuxer (same closed-loop harness as #1):
One serving note that surfaced during testing: a finite file served without
Accept-Rangesclassifies as live and plays unpaced (fast-forward) — that's the delivery auto-detect working as documented, not this demuxer; real CDNs send the header.Both binaries rebuilt and committed (win-x64 DLL, arm64-v8a
.so).Notes
sttstotals = exact duration; per-sample offsets = a seek index) — deliberately not wired here, since exposing them needs new engine ABI; this PR is playback only.stz2(compact sample sizes) isn't handled — rare in practice; such a file keeps the old behaviour.