Infer CODECS and other missing information in HLS

[jamescahall]

• What version of Shaka Player are you using?

2.1.1

• Can you reproduce the issue with our latest release version?

Yes

• Can you reproduce the issue with the latest code from master?

Yes

• Are you using the demo app or your own custom app?

Either

• If custom app, can you reproduce the issue using our demo app?

Yes

• What browser and OS are you using?

Any

• What did you do?

Latest code says it supports unprotected HLS. Trying to load standard m3u8.

• What did you expect to happen?

Playback HLS file playable by standard HTML5 video tag.

• What actually happened?

Throws 4023 error asking for CODECS in video track attributes. When using standard H264 codecs, this should not be required and is not a requirement to include in HLS manifests. Should default to trying to use H264 decoders.

[jamescahall]

adding manually to each flavor:
,CODECS="avc1.42E01E"

Doesn't appear to solve the issue.
Example flavor m3u8 it links to has:
#EXTM3U
#EXT-X-TARGETDURATION:10
#EXT-X-ALLOW-CACHE:YES
#EXT-X-PLAYLIST-TYPE:VOD
#EXT-X-VERSION:3
#EXT-X-MEDIA-SEQUENCE:1
#EXTINF:2.000,
LINK_TO_TS_FILE HERE

[joeyparrish]

Codec information is required in HLS because we've always been able to expect manifests to provide that information. It seems reasonable to make some basic assumptions about HLS, though, if the information is often omitted.

You said "trying to load standard m3u8". Can you provide a link? It will be much easier to diagnose if we can see the manifest and reproduce the error.

I also noticed that you wrote LINK_TO_TS_FILE_HERE in your sample manifest. Keep in mind that for MediaSource-based playback, the browser has to support both the codecs and the container format. Last time I checked, only Safari, Edge, and Chromecast support TS.

[jamescahall]

I can't provide a link here for security reasons. I can send you one if you have a private method to do so. As for your comment on TS, it is supported by those you listed but also by Chrome and nearly every TV that runs an HTML5 based browser (including Opera and Firefox OS). It is the most common format used for HLS delivery so if HLS is supported, TS should be as well.

As for the CODECs information, the HLS specification lists it as a SHOULD include not a MUST include. See here:

https://tools.ietf.org/html/draft-pantos-http-live-streaming-22

Every EXT-X-STREAM-INF tag SHOULD include a CODECS attribute

I've been working on HLS streams for 6+ years now and frequently they do not contain the CODEC value for video streams using the standard avc1.42E01E encoding.

[joeyparrish]

You can find my email address in the CONTRIBUTORS file to send me a link privately.

Contrary to what you are saying about TS, it is not supported in Chrome. You can see the results of a MediaSource probe by visiting https://shaka-player-demo.appspot.com/support.html in the browser of your choice. In Chrome, for example, you'll see this line:

 "video/mp2t; codecs=\"avc1.42E01E\"": false,

When queried for MPEG 2 TS support, Chrome reports that it is not supported.

While MP4 is supported by every major browser (Chrome, Firefox, Edge, IE11, Safari, Opera, Chromecast), you will find that TS support is much more limited (Edge, IE11, Safari, Chromecast). I do not have any data on TV platforms.

[jamescahall]

TS in a HLS m3u8 manifest plays perfectly in Chrome and I test with it on a daily basis and have been doing so for many years. Why MediaSource says otherwise I don't know. I'll send an example to your email.

…

On May 16, 2017, at 6:32 AM, Joey Parrish ***@***.***> wrote: You can find my email address in the CONTRIBUTORS file to send me a link privately. Contrary to what you are saying about TS, it is not supported in Chrome. You can see the results of a MediaSource probe by visiting https://shaka-player-demo.appspot.com/support.html in the browser of your choice. In Chrome, for example, you'll see this line: "video/mp2t; codecs=\"avc1.42E01E\"": false, When queried for MPEG 2 TS support, Chrome reports that it is not supported. While MP4 is supported by every major browser (Chrome, Firefox, Edge, IE11, Safari, Opera, Chromecast), you will find that TS support is much more limited (Edge, IE11, Safari, Chromecast). I do not have any data on TV platforms. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or mute the thread.

[jamescahall]

Hi Joey, I retract my statement about Chrome. I had forgotten I have had a fallback format for Chrome all this time. Sent you an example m3u8 though by email that is a very common format and what is typically used by iOS / Android / Smart TVs. Our primary use case is Smart TVs so right now we have to toggle between ShakaPlayer and the standard HTML5 video player to use non-protected HLS in this format. Thanks James

[jamescahall]

In regards to TVs, HLS with TS is supported by 99%+ of Smart TVs and I have tested this extensively since 2012 in actual implementations on: Samsung o Sharp o Sony o Hisense o Western Digital o Foxxum o Netrange o TiVo o Vizio o Freetime o Vestel o Metz o Medion o Aldisued o Seiki o Westinghouse o Element o Panasonic o Mstar Chips o Philips o Prodea o Blaupunkt o Haier o Medion o Orion o TCL o Sanyo o Expressluck o JTC o Eternity o Arris Cable Boxes o Firefox OS

[joeyparrish]

HLS with TS should work in Shaka Player in browsers (or TVs) that support TS via MediaSource. The code to recognize TS in HLS is already in Shaka Player. We don't transmux it to mp4 on platforms that don't support it, so you should not expect TS to be supported on Chrome, Firefox, or Opera.

I will look at the manifest you sent me and we can add sensible defaults for codecs.

[jamescahall]

It doesn’t appear to work as you describe. I’ve tested on Safari which supports TS and even when I manually add CODECS to get past that error, the m3u8 file will not play with ShakaPlayer. The player doesn’t throw any errors in that case but I just get a black screen. Trying to manually force play has no affect.

[joeyparrish]

Does it work in Microsoft Edge?

[jamescahall]

Sorry but I am unable to confirm that as I do not have a windows computer to test on.

[joeyparrish]

You may be running into this: #743

[joeyparrish]

I have looked at the HLS manifest you sent, and there are several issues preventing playback. Here's a sanitized version of what's in the manifest, for the sake of documenting what we're doing:

#EXTM3U
#EXT-X-STREAM-INF:PROGRAM-ID=1,BANDWIDTH=479232,RESOLUTION=480x256
http://foo.bar/1/index.m3u8
#EXT-X-STREAM-INF:PROGRAM-ID=1,BANDWIDTH=154624,RESOLUTION=480x256
http://foo.bar/2/index.m3u8
#EXT-X-STREAM-INF:PROGRAM-ID=1,BANDWIDTH=248832,RESOLUTION=480x256
http://foo.bar/3/index.m3u8
#EXT-X-STREAM-INF:PROGRAM-ID=1,BANDWIDTH=674816,RESOLUTION=480x256
http://foo.bar/4/index.m3u8
#EXT-X-STREAM-INF:PROGRAM-ID=1,BANDWIDTH=1011712,RESOLUTION=640x352
http://foo.bar/5/index.m3u8
#EXT-X-STREAM-INF:PROGRAM-ID=1,BANDWIDTH=1317888,RESOLUTION=848x480
http://foo.bar/6/index.m3u8
#EXT-X-STREAM-INF:PROGRAM-ID=1,BANDWIDTH=56320,CODECS="mp4a.40.2"
http://foo.bar/7/index.m3u8

These are the main issues with this in today's Shaka Player:

1. No CODECS attribute in most EXT-X-STREAM-INF tags
2. Video+Audio variants use multiplexed streams (which we do not currently assume)
3. Audio-only variant is not multiplexed (so assuming multiplexed everywhere would be a mistake)

The manifest appears to come from Kaltura, so I will also reach out to them and ask if they can add more information to these playlists in a future version of their product. From the HLS spec:

Every EXT-X-STREAM-INF tag SHOULD include a CODECS attribute.

In the mean time, there are a few things we can do to better infer the content characteristics:

1. If there is an AUDIO attribute pointing to audio streams, assume the stream contains video only.
2. If there is a VIDEO attribute pointing to video streams, assume the stream contains audio only.
3. If there is a RESOLUTION attribute, assume the streams contains at least video. If there is no AUDIO attribute, it may also contain audio muxed in.

The spec also states:

If an EXT-X-STREAM-INF tag or EXT-X-I-FRAME-STREAM-INF tag contains the CODECS attribute, the attribute value MUST include every media format [RFC6381] present in any Media Segment in any of the Renditions specified by the Variant Stream.

So we will assume that a single codec string means the content is not multiplexed.

[jamescahall]

Ok thanks Joey. Makes sense and I think your plan would make these playable.

[joeyparrish]

@itaykinnrot, can you point me in the right direction to file a feature request on Kaltura? See my comment above for context.

[joeyparrish]

This fix was just released in v2.1.2.

[jamescahall]

Confirmed the error is gone. Unfortunately typical HLS is till not running in Safari due to the TS bug.

[joeyparrish]

Thank you for confirming. We can discuss Safari TS issues further on #743.