Skip to content

Add MP4 DASH/CMAF subtitle support (wvtt, stpp, tx3g in fragmented MP4) - #13178

Merged
niksedk merged 1 commit into
mainfrom
fix/mp4-dash-subtitle-support
Aug 4, 2026
Merged

Add MP4 DASH/CMAF subtitle support (wvtt, stpp, tx3g in fragmented MP4)#13178
niksedk merged 1 commit into
mainfrom
fix/mp4-dash-subtitle-support

Conversation

@niksedk

@niksedk niksedk commented Aug 4, 2026

Copy link
Copy Markdown
Member

Summary

Subtitles in MP4 DASH/CMAF files (fragmented MP4) mostly failed to load. This PR makes all common DASH subtitle shapes work — WebVTT (wvtt), TTML/IMSC1 (stpp) and MPEG-4 timed text (tx3g) — whether the file is a subtitle-only DASH representation (init + .m4s segments, concatenated or onDemand single-file), a muxed fragmented MP4 with video/audio, a lone .m4s segment without its init, or a progressive MP4.

What was broken

Problem Effect
Fragmented WebVTT used the movie (mvhd) timescale instead of the track's mdhd timescale All DASH-WebVTT cue times skewed (5/3 too late with GPAC's 600/1000 layout)
subt handler + stpp samples unsupported TTML-in-MP4 tracks completely invisible, progressive and fragmented
Moof kept one traf, Traf one trun, tfhd track ids unused Muxed fMP4 yielded no subtitles at all
IsmtDfxp parsed each mdat as a single XML document mdats holding several TTML sample docs were silently dropped (partial extraction)
moov/mvex/trex defaults not parsed Last DASH segment's cue truncated/dropped when trun/tfhd omit sample durations
seconv 10 KB minimum for MP4 Subtitle-only DASH files (a few KB) never reached the MP4 parser
CmafParser divided ticks with integer math and preferred mvhd .cmft cue times truncated to whole seconds, and in seconds instead of ms

What this PR does

  • Mp4Parser: fragmented samples are now sliced precisely per track — all traf/trun boxes are kept, each fragment resolves its track via tfhd track id, sample byte ranges come from data offsets + sizes, and timing comes from tfdt + sample durations in the track timescale (with per-track continuation when tfdt is absent, as in MP4Box -frag output). Missing durations/sizes fall back to mvex/trex defaults.
  • Sample decoders: wvtt cue boxes (vttc/payl/sttg, incl. position info), stpp TTML documents (deduplicating cues repeated across segment documents; zero-based Smooth-Streaming-style docs are shifted by the sample start), and tx3g length-prefixed text. Lone .m4s segments without an init are sniffed by content.
  • Progressive TTML: subt handler recognized; stpp sample parsing added to Stbl, so TTML tracks appear as normal subtitle tracks (incl. the track picker).
  • IsmtDfxp: delegates to the box-aware parser when it finds TTML cues; otherwise splits concatenated TTML documents before XML parsing.
  • seconv: small MP4 files (> 100 bytes) are tried against the MP4 parser, falling back to the text loader as before.
  • CmafParser: correct timescale preference and millisecond math.

Testing

  • 10 new unit tests build minimal init/media segments programmatically: track-vs-movie timescale, lone segment, muxed multi-traf, TTML document times + dedup + zero-based shift, progressive subt/stpp, trex defaults, and TTML document splitting (incl. ttm: metadata red herrings and namespaced roots).
  • Verified end-to-end against GPAC (MP4Box) DASH packages — live profile (init + 8 .m4s), onDemand profile, -frag muxed files, and progressive muxes, for all three codecs: every variant now extracts all 6 test cues with timings identical to the progressive reference (previously: wrong times, 3/6 cues, or nothing).
  • All existing suites pass: libse 831, seconv 290, libuilogic 140, UI 1270.

🤖 Generated with Claude Code

Extracting subtitles from DASH/CMAF media files mostly failed:

- Fragmented WebVTT cue times were scaled by the movie (mvhd) timescale
  instead of the track's mdhd timescale, landing e.g. 5/3 too late with
  GPAC's mvhd=600/track=1000 layout.
- TTML (stpp) tracks were invisible everywhere: the "subt" handler was
  not recognized, and nothing parsed stpp samples.
- Muxed fMP4 lost subtitles entirely: Moof kept only one traf and Traf
  only one trun, and tfhd track ids were never used to slice the right
  mdat byte ranges.
- IsmtDfxp text-sniffed whole mdats as one XML document, silently
  dropping every mdat that holds more than one TTML sample document.
- The last DASH segment often carries no sample duration in trun/tfhd;
  the moov/mvex/trex defaults were not parsed, truncating the last cue.
- seconv rejected MP4 files under 10 KB, which excludes typical
  subtitle-only DASH files (a few KB).

Rework the fragmented path in MP4Parser: keep all trafs/truns, resolve
each fragment's track via tfhd track id, slice exact sample ranges via
data offsets and sizes (so muxed segments work), time samples from tfdt
plus durations in the track timescale (with per-track continuation when
tfdt is absent), fill missing sample durations/sizes from mvex/trex, and
decode wvtt cue boxes, stpp TTML documents (deduplicating cues repeated
across segment documents, shifting zero-based Smooth Streaming times)
and tx3g samples. Lone .m4s segments without their init are sniffed by
content. Progressive TTML tracks ("subt" handler, stpp samples) are now
regular subtitle tracks. CmafParser also produced seconds instead of
milliseconds and preferred the wrong timescale.

Verified against GPAC-packaged DASH live/onDemand profiles and MP4Box
-frag muxed files: all wvtt/stpp/tx3g variants now extract every cue
with timings identical to the progressive reference.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@niksedk
niksedk merged commit 2c7af43 into main Aug 4, 2026
1 check passed
@niksedk
niksedk deleted the fix/mp4-dash-subtitle-support branch August 4, 2026 05:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant