Transmux TS to fMP4 #887

joeyparrish · 2017-06-15T22:31:45Z

On platforms that support Transport Stream via MediaSource, we are able to play TS content. Currently, that is just Edge and Chromecast. (Safari is not working due to #743.)

This is consistently surprising to developers, and not everyone can transmux their content to fMP4 ahead of time. Developers expect that working content will work everywhere.

To meet this expectation, we should start offering transmuxing of TS content in the browser. We should optimize this, if needed, to reduce impact on CPU and battery life. If the process is too expensive, we should warn developers and provide guidance on transmuxing content en masse on the server, to reduce the burden on clients.

We should examine the transmuxing implementation of hls.js (which is also Apache-licensed) before we begin work. There may be a way to collaborate with hls.js to reuse their transmuxer rather than build a new one.

forbesjo · 2017-06-16T14:19:50Z

https://github.com/videojs/mux.js is another library you could look at (I may be biased).

joeyparrish · 2017-06-20T21:50:51Z

@forbesjo, thank you for the suggestion. We will look into it!

joeyparrish · 2017-07-12T22:43:34Z

We have decided to use https://github.com/videojs/mux.js as the basis of our transmuxing feature.

Here's a rough outline:

Today

We parse the manifest and create our internal data structures describing the content
We query the platform for the MIME types and codecs we find in the content
If the container isn't supported, the platform rejects the MIME type, and we reject the content

With transmuxing

We parse the manifest and create our internal data structures describing the content
We query the platform for the MIME types and codecs we find in the content
If the container isn't supported, we check if mux.js has been loaded
If mux.js is available and can demux the container, we install transmuxing filters, relabel the content as MP4, and try again
Transmuxing filters will use mux.js to transmux each segment after download

If the application developer is not interested in transmuxing, no changes are required. If the application developer wants TS transmuxing, they merely have to upgrade Shaka Player to v2.3+ and load mux.js in parallel with Shaka.

@forbesjo, how does mux.js manage API changes? If we ask our users to load mux.js, and we have tested with v4.1.5, will there be any breaking API changes in v4.2? Should we advise them only to use specific versions? What's the best way to stay informed of important changes to mux.js?

forbesjo · 2017-07-13T16:08:09Z

mux.js follows semver so there will only be breaking changes in major versions. I would only advise users to use specific versions if they prefer certainty or reproducible builds over automatic updates.

Right now updates aren't communicated beyond watching the repo but we plan on announcing updates in the #announcements channel in http://slack.videojs.com/ as well as posting change log notes to Github releases.

joeyparrish · 2017-07-17T16:26:38Z

@forbesjo, thanks!

forbesjo · 2017-07-19T15:48:37Z

Users can also be directed to the video.js #playback channel for questions/information that doesn't fit in a Github issue.

tchakabam · 2017-07-19T20:55:02Z

mux.js is probably a really good choice! great to hear that, looking forward to TS-based HLS support in Shaka!

Do you see the need to run the transmuxing inside a worker script? This is how it was done in Hls.js and also in videojs-contrib-hls.

So in my experience transmuxing, parsing the TS, and building/writing the fMp4 is rather effortless task (it's not like decoding anything or so, "just" read/writing container data). We have been doing this for audio streams, but of course with video it will generally be a higher data throughput.

But it's probably better to not do this on the UI thread still :)

I have been building a library for transmuxing, that currently supports output of MP4, but the usecase it was tested with was only MP3 (for HLS with MP3 elementary streams playback on Firefox, that only accepts MP3 when contained as fMP4, Chrome does support Mp3 "elementary" btw). It is a general pipelining framework with parsers, demuxers, muxers, etc.. It's a bit GStreamer inspired :)
It also supports running stuff in workers. The mp4-packager was based on a Mozilla experimental project under Apache license.

See: https://github.com/tchakabam/multimedia.js

There is also another great library for reading & building MP4 data: https://github.com/madebyhiro/codem-isoboxer

and here is a node-based Transport-Stream parser which even seems to be DVB-T compliant: https://github.com/mkrufky/node-dvbtee

tchakabam · 2017-07-19T20:56:13Z

@joeyparrish If you are still looking for Hls.js know-how, I have been contributing a bit to it :) So you can shoot me your questions if you have any.

Because Safari's support for TS via MSE is broken, reject all TS content in Safari. Transmuxing (#887) will compensate. https://bugs.webkit.org/show_bug.cgi?id=165342 Closes #743 Change-Id: Iff8a6a7c159e48a4f388d99396915752e649c6ac

joeyparrish · 2017-10-07T18:51:16Z

Just a quick status update: we are making good progress on transmuxing with mux.js, and we hope to publish something soon.

Because Safari's support for TS via MSE is broken, reject all TS content in Safari. Transmuxing (#887) will compensate. https://bugs.webkit.org/show_bug.cgi?id=165342 Closes #743 Change-Id: Iff8a6a7c159e48a4f388d99396915752e649c6ac

Supporting Transport Stream by transmuxing it to MP4, with mux.js library. Issue #887 Change-Id: I70361bb364721fc5e72cba7d0fabc311ce0ba8b5

avelad · 2017-10-24T08:26:56Z

I just tested the functionality in the master branch, but it's not working with all TS playlists.

Tested in: "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/62.0.3202.62 Safari/537.36"

Samples:

joeyparrish · 2017-10-24T15:47:17Z

It seems that https://content.jwplatform.com/manifests/vM7nH0Kl.m3u8 contains raw AAC (no container), which we don't support in our parser yet. I filed #1083 to improve this.

The other stream (http://vod.izzitv.mx/Content/HLS/VOD/IRIS_DLVY5849991423000097/ATEME_HLS_P_DLVY5849991423000097.m3u8) fails with an HTTP 403 error, even with mixed content restriccions disabled.

avelad · 2017-10-24T20:57:15Z

Sorry, I forget the geoblocking... You need use a query string parameter in master and media playlist (?vu=12345) http://vod.izzitv.mx/Content/HLS/VOD/IRIS_DLVY5849991423000097/ATEME_HLS_P_DLVY5849991423000097.m3u8?vu=12345

Master playlist (http://vod.izzitv.mx/Content/HLS/VOD/IRIS_DLVY5849991423000097/ATEME_HLS_P_DLVY5849991423000097.m3u8?vu=12345)

#EXTM3U
#EXT-X-VERSION:4
#EXT-X-MEDIA:TYPE=AUDIO,GROUP-ID="Audio",NAME="Spanish",DEFAULT=NO,AUTOSELECT=YES,LANGUAGE="spa",URI="ATEME_HLS_P_DLVY5849991423000097_audio_spa_96kbps.m3u8"
#EXT-X-STREAM-INF:PROGRAM-ID=1,BANDWIDTH=337000,CODECS="avc1.42E020,mp4a.40.5",AUDIO="Audio"
ATEME_HLS_P_DLVY5849991423000097_video_200kbps.m3u8
#EXT-X-STREAM-INF:PROGRAM-ID=1,BANDWIDTH=97000,CODECS="mp4a.40.5",AUDIO="Audio"
ATEME_HLS_P_DLVY5849991423000097_audio_spa_96kbps.m3u8
#EXT-X-STREAM-INF:PROGRAM-ID=1,BANDWIDTH=759000,CODECS="avc1.42E020,mp4a.40.5",AUDIO="Audio"
ATEME_HLS_P_DLVY5849991423000097_video_600kbps.m3u8
#EXT-X-STREAM-INF:PROGRAM-ID=1,BANDWIDTH=1394000,CODECS="avc1.42E020,mp4a.40.5",AUDIO="Audio"
ATEME_HLS_P_DLVY5849991423000097_video_1.2Mbps.m3u8
#EXT-X-STREAM-INF:PROGRAM-ID=1,BANDWIDTH=2028000,CODECS="avc1.640020,mp4a.40.5",AUDIO="Audio"
ATEME_HLS_P_DLVY5849991423000097_video_1.8Mbps.m3u8
#EXT-X-STREAM-INF:PROGRAM-ID=1,BANDWIDTH=97000,CODECS="mp4a.40.5",AUDIO="Audio"
ATEME_HLS_P_DLVY5849991423000097_audio_spa_96kbps.m3u8
#EXT-X-STREAM-INF:PROGRAM-ID=1,BANDWIDTH=3495000,CODECS="avc1.640020,mp4a.40.5",AUDIO="Audio"
ATEME_HLS_P_DLVY5849991423000097_video_3.2Mbps.m3u8
#EXT-X-STREAM-INF:PROGRAM-ID=1,BANDWIDTH=4658000,CODECS="avc1.640029,mp4a.40.5",AUDIO="Audio"
ATEME_HLS_P_DLVY5849991423000097_video_4.3Mbps.m3u8
#EXT-X-STREAM-INF:PROGRAM-ID=1,BANDWIDTH=97000,CODECS="mp4a.40.5",AUDIO="Audio"
ATEME_HLS_P_DLVY5849991423000097_audio_spa_96kbps.m3u8

Media playlist (http://vod.izzitv.mx/Content/HLS/VOD/IRIS_DLVY5849991423000097/ATEME_HLS_P_DLVY5849991423000097_video_4.3Mbps.m3u8?vu=12345)

#EXTM3U
#EXT-X-VERSION:3
#EXT-X-TARGETDURATION:11
#EXTINF:10,
ATEME_HLS_P_DLVY5849991423000097_video_4.3Mbps.ts?length=5534908&offset=0
#EXTINF:10,
ATEME_HLS_P_DLVY5849991423000097_video_4.3Mbps.ts?length=5651844&offset=5534908
#EXTINF:10,
ATEME_HLS_P_DLVY5849991423000097_video_4.3Mbps.ts?length=5566492&offset=11186752
#EXTINF:10,
ATEME_HLS_P_DLVY5849991423000097_video_4.3Mbps.ts?length=5430004&offset=16753244
#EXTINF:10,
ATEME_HLS_P_DLVY5849991423000097_video_4.3Mbps.ts?length=5582660&offset=22183248
#EXTINF:10,
ATEME_HLS_P_DLVY5849991423000097_video_4.3Mbps.ts?length=5605220&offset=27765908
#EXTINF:10,
ATEME_HLS_P_DLVY5849991423000097_video_4.3Mbps.ts?length=5391276&offset=33371128
#EXTINF:10,
ATEME_HLS_P_DLVY5849991423000097_video_4.3Mbps.ts?length=5563484&offset=38762404
#EXTINF:10,
ATEME_HLS_P_DLVY5849991423000097_video_4.3Mbps.ts?length=5524944&offset=44325888
#EXTINF:10,
ATEME_HLS_P_DLVY5849991423000097_video_4.3Mbps.ts?length=5608040&offset=49850832
#EXTINF:10,
ATEME_HLS_P_DLVY5849991423000097_video_4.3Mbps.ts?length=5707116&offset=55458872
#EXTINF:10,
ATEME_HLS_P_DLVY5849991423000097_video_4.3Mbps.ts?length=5241064&offset=61165988
#EXTINF:10,
ATEME_HLS_P_DLVY5849991423000097_video_4.3Mbps.ts?length=5567056&offset=66407052
#EXTINF:10,
ATEME_HLS_P_DLVY5849991423000097_video_4.3Mbps.ts?length=5566492&offset=71974108
#EXTINF:3,
ATEME_HLS_P_DLVY5849991423000097_video_4.3Mbps.ts?length=1385748&offset=77540600
#EXT-X-ENDLIST

I think that the problem is related to mime type or content extension with query params.

joeyparrish · 2017-10-24T20:59:54Z

Yes, that's probably the issue. We aren't doing proper URI parsing in the HLS parser when we use extensions to guess at MIME types. Shouldn't be hard to fix, but it's a shame HLS can't just provide MIME types in the first place.

joeyparrish · 2017-10-24T21:02:04Z

Filed #1085 to track the extension parsing issue.

NikitaYeryomin · 2017-12-09T01:24:42Z

Hello! Could you tell if you already support TS-based HLS? The issue is closed but your FAQ still suggests to subscribe to it: https://github.com/google/shaka-player/blob/master/docs/tutorials/faq.md

joeyparrish · 2017-12-11T18:33:00Z

Yes, we do, in the master branch and in the upcoming v2.3 release. If the browser does not support TS, your application will need to load mux.js for transmuxing. If mux.js is in the environment, Shaka Player will detect that and use it to transmux TS content automatically. No other integration work is required.

And you are right that our FAQ is out-of-date. Thank you for bringing that to our attention. I have filed #1183 to track updating the docs.

joeyparrish added type: enhancement New feature or request component: HLS The issue involves Apple's HLS manifest format labels Jun 15, 2017

joeyparrish added this to the v2.3.0 milestone Jun 15, 2017

This was referenced Jun 16, 2017

Get TS segments playing in Safari #743

Closed

MANIFEST.UNPLAYABLE_PERIOD error on HLS manifest #885

Closed

michellezhuogg self-assigned this Jun 27, 2017

tanquetav mentioned this issue Jul 19, 2017

Enable shaka in hls source videos clappr/dash-shaka-playback#31

Open

joeyparrish mentioned this issue Jul 22, 2017

No decoders for requested formats: video/mp2t #939

Closed

shaka-bot pushed a commit that referenced this issue Oct 20, 2017

Transmux from Transport Stream to MP4

e516a64

Supporting Transport Stream by transmuxing it to MP4, with mux.js library. Issue #887 Change-Id: I70361bb364721fc5e72cba7d0fabc311ce0ba8b5

joeyparrish closed this as completed Oct 23, 2017

joeyparrish mentioned this issue Oct 24, 2017

Support raw AAC in HLS #1083

Closed

joeyparrish mentioned this issue Oct 24, 2017

Fix URI parsing in HLS parser #1085

Closed

shaka-bot added the archived label Mar 22, 2018

shaka-project locked and limited conversation to collaborators Mar 22, 2018

shaka-bot added the status: archived Archived and locked; will not be updated label Apr 15, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Transmux TS to fMP4 #887

Transmux TS to fMP4 #887

joeyparrish commented Jun 15, 2017

forbesjo commented Jun 16, 2017

joeyparrish commented Jun 20, 2017

joeyparrish commented Jul 12, 2017

forbesjo commented Jul 13, 2017

joeyparrish commented Jul 17, 2017

forbesjo commented Jul 19, 2017

tchakabam commented Jul 19, 2017 •

edited

Loading

tchakabam commented Jul 19, 2017 •

edited

Loading

joeyparrish commented Oct 7, 2017

avelad commented Oct 24, 2017 •

edited

Loading

joeyparrish commented Oct 24, 2017

avelad commented Oct 24, 2017

joeyparrish commented Oct 24, 2017

joeyparrish commented Oct 24, 2017

NikitaYeryomin commented Dec 9, 2017 •

edited

Loading

joeyparrish commented Dec 11, 2017

Transmux TS to fMP4 #887

Transmux TS to fMP4 #887

Comments

joeyparrish commented Jun 15, 2017

forbesjo commented Jun 16, 2017

joeyparrish commented Jun 20, 2017

joeyparrish commented Jul 12, 2017

Today

With transmuxing

forbesjo commented Jul 13, 2017

joeyparrish commented Jul 17, 2017

forbesjo commented Jul 19, 2017

tchakabam commented Jul 19, 2017 • edited Loading

tchakabam commented Jul 19, 2017 • edited Loading

joeyparrish commented Oct 7, 2017

avelad commented Oct 24, 2017 • edited Loading

joeyparrish commented Oct 24, 2017

avelad commented Oct 24, 2017

joeyparrish commented Oct 24, 2017

joeyparrish commented Oct 24, 2017

NikitaYeryomin commented Dec 9, 2017 • edited Loading

joeyparrish commented Dec 11, 2017

tchakabam commented Jul 19, 2017 •

edited

Loading

tchakabam commented Jul 19, 2017 •

edited

Loading

avelad commented Oct 24, 2017 •

edited

Loading

NikitaYeryomin commented Dec 9, 2017 •

edited

Loading