Support multiplexed streams in HLS start time probe

[thedracle]

What version of Shaka Player are you using*
Latest from master.

Can you reproduce the issue with our latest release version:
Latest release does not support muxjs transmuxing.

Can you reproduce the issue with the latest code from master:
Yes

Are you using the demo app or your own custom app:
Both.

If custom app, can you reproduce the issue using our demo app:
Yes

What browser and OS are you using:
OSX/Chrome

What are the manifest and license server URIs:
(you can send the URIs to shaka-player-issues@google.com instead, but please use GitHub and the template for the rest)

This probably isn't the sample I analyzed below, but it illustrates the same issue.
https://shakarepro.s3-us-west-1.amazonaws.com/repro/mbr/manifest.m3u8

What did you do?

Attempted to play the above HLS stream.

What did you expect to happen?
For it to play.

What actually happened?

I am seeing this issue where the timestamps output by muxjs are different than my TS segments when the presentationTimeOffset isn't 0.

Basically my first MPEGTS segment starts at 133500:

        {
            "media_type": "video",
            "stream_index": 0,
            "key_frame": 1,
            "pkt_pts": 133500,
            "pkt_pts_time": "1.483333",
            "pkt_dts": 133500,
            "pkt_dts_time": "1.483333",
            "best_effort_timestamp": 133500,
            "best_effort_timestamp_time": "1.483333",
            "pkt_duration": 3750,
            "pkt_duration_time": "0.041667",
            "pkt_pos": "564",
            "pkt_size": "10638",
            "width": 320,
            "height": 240,
            "pix_fmt": "yuv420p",
            "sample_aspect_ratio": "1:1",
            "pict_type": "I",
            "coded_picture_number": 0,
            "display_picture_number": 0,
            "interlaced_frame": 0,
            "top_field_first": 0,
            "repeat_pict": 0
        }

In the logs I see:

First segment 750k.ts starts at 1.4833333333333334

Which matches the first PTS (Although there is an earlier PTS for the first audio packet in the stream which appears later).

The presentationTimeOffset is set to 1.4833333333333334, but the code to adjust the PTS via the transmuxer sets it to adjust by 0:

lib/media/transmuxer.js:

this.muxTransmuxer_.setBaseMediaDecodeTime(startTime * timescale);

^ Tracing in the debugger 'startTime' is 0.

As a result the transmuxed mp4 seems to adjust all of the timestamps:

        {
            "media_type": "video",
            "stream_index": 0,
            "key_frame": 1,
            "pkt_pts": 7500,
            "pkt_pts_time": "0.083333",
            "pkt_dts": 7500,
            "pkt_dts_time": "0.083333",
            "best_effort_timestamp": 7500,
            "best_effort_timestamp_time": "0.083333",
            "pkt_pos": "2877",
            "pkt_size": "28480",
            "width": 640,
            "height": 480,
            "pix_fmt": "yuv420p",
            "sample_aspect_ratio": "1:1",
            "pict_type": "I",
            "coded_picture_number": 0,
            "display_picture_number": 0,
            "interlaced_frame": 0,
            "top_field_first": 0,
            "repeat_pict": 0
        },

And MediaSourceExtensions croaks with the following error:

video frame with PTS 141667us has negative DTS -66667us after applying timestampOffset, handling any discontinuity, and filtering against append window

Also, I wonder if getStartTimeFromTsSegment_ in lib/hls/hls_parser.js needs to be adjusted to search for a minimum timestamp closer to the beginning of the TS stream, rather than just reading the PTS out of the first TS packet it finds that has one.

[vaage]

@michellezhuogg could you take a look at this issue as you are more familiar with the trans-muxing code?

[TheModMaker]

The problem is this is using multiplexed content (a stream with both audio and video). Also, the audio and video streams don't start at the same time. We use the first packet to get the time, but the other stream (which appears later) starts before that packet. Some browsers don't support multiplexed content and require the streams to be separate; we usually only support separate streams, but don't explicitly forbid multiplexed streams.

It may be possible for this to happen for non-multiplexed streams as well. A media segment doesn't have to start with a key frame. If it starts with a B-frame, then the PTS times of the frames will be out of order. We should parse the first few frames until we hit a key frame to avoid this.

But that won't help your stream, since the problem is that there are multiple streams within the TS packet stream. If we see that it is a multiplexed stream (which I think we will always know), then we can probe further to find the second stream, but it may require a second download probe to get the timestamps.

[thedracle]

@TheModMaker

I think it is maybe a combination of the multiple streams and the fact the timestamp from the first PES packet is used, and the issue I detailed above.

This is just an experiment and not really a suggestion, that parses through more than one PES packet to try to find the earlier PTS:
https://github.com/thedracle/shaka-player/pull/2/files

It succeeds on the clip I provided above (Probably a more robust strategy would be required in reality), but it still fails with the same MediaSourceExtension errors due to the setBaseMediaDecodeTime not being set correctly.

[TheModMaker]

I think that issue is a problem with Chrome. Chrome confuses PTS/DTS times, so if we use the PTS times as the offset, the DTS may become negative. This is technically a bug in Chrome that they are actively working on, but we should still support this case. Unfortunately the fix is to use DTS times, which may cause a gap at the beginning of the stream, but gap jumping should be able to compensate for that.

[TheModMaker]

Actually I think it is a problem with how we are adjusting the times. The segment we get from muxjs has been adjusted to start at 0, but we also use presentationTimeOffset to adjust the segment further, which causes it to be negative. To be able to support both non-zero VOD and live content, we need to patch muxjs so we don't have to use setBaseMediaDecodeTime. You can follow along in videojs/mux.js#168.

[joeyparrish]

When mux.js 4.3.3 comes out, we will have an option to keep the original TS timestamps.

However, there is also an issue with your content:

#EXTM3U
#EXT-X-VERSION:4
#EXT-X-TARGETDURATION:5
#EXT-X-MEDIA-SEQUENCE:0
#EXT-X-PLAYLIST-TYPE:VOD
#EXTINF:4.098689,
#EXT-X-BYTERANGE:216012@0
750k.ts
#EXTINF:4.000000,
#EXT-X-BYTERANGE:254552@216012
750k.ts
#EXTINF:4.000000,
#EXT-X-BYTERANGE:236880@470564
750k.ts
#EXTINF:0.000000,
#EXT-X-BYTERANGE:18424@707444
750k.ts
#EXT-X-ENDLIST

The last segment has a duration of 0, which is not valid. Even with the fix to mux.js, your content will not play because of that.

[boredom2]

Just as an Update. mux.js has been updated yesterday to v4.4.0.

[joeyparrish]

Yes, and we will be updating to that shortly.

[joeyparrish]

The updated mux.js does not seem to fix the negative DTS issue. However, the rejection of segments with positive PTS with negative DTS is definitely a Chrome bug (http://crbug.com/398141). It can be fixed by the Chrome experimental flag --enable-features=MseBufferByPts. I will ask the Chrome team if the negative DTS allowance can be brought out from behind that flag.

[joeyparrish]

The label "bad content" is for the 0-length segment at the end.
The label "enhancement" is for our update to mux.js 4.4 with the timestamp fix.
The label "external" is for the component related to Chrome's DTS bug.

[joeyparrish]

The mux.js update is complete. @thedracle, can you fix the 0-length segment in your content?

[thedracle]

@joeyparrish,

Yes, I'll generate a new clip, it's something to do with how FFMPEG is generating those segments from the clip I provided, and test again.

[thedracle]

Here is a HLS clip with similar encoding, that doesn't have the zero duration issue:
https://shakarepro.s3-us-west-1.amazonaws.com/cut5/mbr/manifest.m3u8

[joeyparrish]

Excellent! With the latest Shaka Player and the Chrome flag I mentioned before, the new clip is working.

I will continue to follow-up with Chrome team on this issue.

[joeyparrish]

mux.js v4.4.0 update cherry-picked to Shaka Player v2.3.1.