Add MP4 DASH/CMAF subtitle support (wvtt, stpp, tx3g in fragmented MP4) - #13178
Merged
Conversation
Extracting subtitles from DASH/CMAF media files mostly failed:
- Fragmented WebVTT cue times were scaled by the movie (mvhd) timescale
instead of the track's mdhd timescale, landing e.g. 5/3 too late with
GPAC's mvhd=600/track=1000 layout.
- TTML (stpp) tracks were invisible everywhere: the "subt" handler was
not recognized, and nothing parsed stpp samples.
- Muxed fMP4 lost subtitles entirely: Moof kept only one traf and Traf
only one trun, and tfhd track ids were never used to slice the right
mdat byte ranges.
- IsmtDfxp text-sniffed whole mdats as one XML document, silently
dropping every mdat that holds more than one TTML sample document.
- The last DASH segment often carries no sample duration in trun/tfhd;
the moov/mvex/trex defaults were not parsed, truncating the last cue.
- seconv rejected MP4 files under 10 KB, which excludes typical
subtitle-only DASH files (a few KB).
Rework the fragmented path in MP4Parser: keep all trafs/truns, resolve
each fragment's track via tfhd track id, slice exact sample ranges via
data offsets and sizes (so muxed segments work), time samples from tfdt
plus durations in the track timescale (with per-track continuation when
tfdt is absent), fill missing sample durations/sizes from mvex/trex, and
decode wvtt cue boxes, stpp TTML documents (deduplicating cues repeated
across segment documents, shifting zero-based Smooth Streaming times)
and tx3g samples. Lone .m4s segments without their init are sniffed by
content. Progressive TTML tracks ("subt" handler, stpp samples) are now
regular subtitle tracks. CmafParser also produced seconds instead of
milliseconds and preferred the wrong timescale.
Verified against GPAC-packaged DASH live/onDemand profiles and MP4Box
-frag muxed files: all wvtt/stpp/tx3g variants now extract every cue
with timings identical to the progressive reference.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This was referenced Aug 4, 2026
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Subtitles in MP4 DASH/CMAF files (fragmented MP4) mostly failed to load. This PR makes all common DASH subtitle shapes work — WebVTT (
wvtt), TTML/IMSC1 (stpp) and MPEG-4 timed text (tx3g) — whether the file is a subtitle-only DASH representation (init +.m4ssegments, concatenated or onDemand single-file), a muxed fragmented MP4 with video/audio, a lone.m4ssegment without its init, or a progressive MP4.What was broken
mvhd) timescale instead of the track'smdhdtimescalesubthandler +stppsamples unsupportedMoofkept onetraf,Trafonetrun,tfhdtrack ids unusedIsmtDfxpparsed each mdat as a single XML documentmoov/mvex/trexdefaults not parsedtrun/tfhdomit sample durationsCmafParserdivided ticks with integer math and preferredmvhd.cmftcue times truncated to whole seconds, and in seconds instead of msWhat this PR does
Mp4Parser: fragmented samples are now sliced precisely per track — alltraf/trunboxes are kept, each fragment resolves its track viatfhdtrack id, sample byte ranges come from data offsets + sizes, and timing comes fromtfdt+ sample durations in the track timescale (with per-track continuation whentfdtis absent, as inMP4Box -fragoutput). Missing durations/sizes fall back tomvex/trexdefaults.wvttcue boxes (vttc/payl/sttg, incl. position info),stppTTML documents (deduplicating cues repeated across segment documents; zero-based Smooth-Streaming-style docs are shifted by the sample start), andtx3glength-prefixed text. Lone.m4ssegments without an init are sniffed by content.subthandler recognized;stppsample parsing added toStbl, so TTML tracks appear as normal subtitle tracks (incl. the track picker).IsmtDfxp: delegates to the box-aware parser when it finds TTML cues; otherwise splits concatenated TTML documents before XML parsing.CmafParser: correct timescale preference and millisecond math.Testing
subt/stpp,trexdefaults, and TTML document splitting (incl.ttm:metadata red herrings and namespaced roots)..m4s), onDemand profile,-fragmuxed files, and progressive muxes, for all three codecs: every variant now extracts all 6 test cues with timings identical to the progressive reference (previously: wrong times, 3/6 cues, or nothing).🤖 Generated with Claude Code