Smooth period transition #2580

aescarcha · 2018-05-22T13:11:46Z

This PR introduces buffer reuse for compatible codecs and preload of next periods to avoid jumps and freezes when switching periods.

It also introduces some changes to the SourceBufferSink to handle the appends and removes with a queue.

It's the same as #2575 but with clearer commit history.
Closes #1708

robertbryer · 2018-05-22T13:47:32Z

Good stuff!

Trying to seek to the next period doesn't work when you have already preloaded into the period. I get:

"This SourceBuffer has been removed from the parent media source."

So I think a normal buffer reset is being called after the preload when you seek like this.

davemevans · 2018-05-22T13:51:35Z

src/streaming/Stream.js

@@ -644,6 +702,80 @@ function Stream(config) {
        checkIfInitializationCompleted();
    }

+    function isCompatibleWithStream(stream) {
+        return compareCodecs(stream, Constants.VIDEO) && compareCodecs(stream, Constants.AUDIO);


This will return false for audio-only (and video-only streams), which is incorrect.

This requires a few changes like checking if the previous / next period has audio and video to avoid saving the non-needed buffer or recreating it, will add it last.

I added commit d7f4b8f to allow preload of audio-only or video-only periods, but won't work when switching from a video/audio period to a video-only one

Following that commit, this works great for multiperiod audio-only services 🍾

davemevans · 2018-05-22T14:11:50Z

Streams signalling InbandEvent throw the following error on Period transition, and playback stops:

Uncaught TypeError: Cannot read property 'addInbandEvents' of undefined
    at Object.onMediaFragmentLoaded (streaming\controllers\BufferController.js:227)
    at core\EventBus.js:88
    at Array.forEach (<anonymous>)
    at Object.trigger (core\EventBus.js:88)
    at Object.onFragmentLoadingCompleted (streaming\controllers\FragmentController.js:138)
    at core\EventBus.js:88
    at Array.forEach (<anonymous>)
    at Object.trigger (core\EventBus.js:88)
    at Object.onLoadingCompleted (streaming\models\FragmentModel.js:310)
    at core\EventBus.js:88

Example manifest:

<?xml version="1.0" encoding="UTF-8"?>
<MPD type="static" xmlns="urn:mpeg:dash:schema:mpd:2011" xmlns:dvb="urn:dvb:dash-extensions:2014-1" profiles="urn:dvb:dash:profile:dvb-dash:2014,urn:dvb:dash:profile:dvb-dash:isoff-ext-live:2014" minBufferTime="PT2.049S" maxSegmentDuration="PT3.84S" mediaPresentationDuration="PT38.40S">
	<BaseURL serviceLocation="A" dvb:priority="1" dvb:weight="1">http://rdmedia.bbc.co.uk/dash/ondemand/testcard/1/</BaseURL>
	<Period duration="PT19.2S" start="PT0S">
		<AdaptationSet startWithSAP="2" segmentAlignment="true" id="1" sar="1:1" mimeType="video/mp4" >
			<InbandEventStream schemeIdUri="tag:rdmedia.bbc.co.uk,2014:events/ballposition" value="1"/>
			<Role schemeIdUri="urn:mpeg:dash:role:2011" value="main"/>
			<BaseURL>avc3-events/</BaseURL>
			<SegmentTemplate startNumber="1" timescale="1000" duration="3840" media="$RepresentationID$/$Number%06d$.m4s" initialization="$RepresentationID$/IS.mp4" />
			<Representation id="1920x1080i25" codecs="avc3.640028" height="1080" width="1920" frameRate="25" scanType="interlaced" bandwidth="8060152" />
		</AdaptationSet>
		<AdaptationSet startWithSAP="2" segmentAlignment="true" id="2" codecs="mp4a.40.5" audioSamplingRate="48000" lang="eng" mimeType="audio/mp4" >
			<AudioChannelConfiguration schemeIdUri="urn:mpeg:dash:23003:3:audio_channel_configuration:2011" value="2"/>
			<Role schemeIdUri="urn:mpeg:dash:role:2011" value="main"/>
			<BaseURL>audio/</BaseURL>
			<SegmentTemplate startNumber="1" timescale="1000" duration="3840" media="$RepresentationID$/$Number%06d$.m4s" initialization="$RepresentationID$/IS.mp4" />
			<Representation id="96kbps" bandwidth="96000" />
		</AdaptationSet>
	</Period>
	<Period start="PT19.2S" duration="PT19.2S">
		<AdaptationSet startWithSAP="2" segmentAlignment="true" id="1" sar="1:1" mimeType="video/mp4" >
			<InbandEventStream schemeIdUri="tag:rdmedia.bbc.co.uk,2014:events/ballposition" value="1"/>
			<Role schemeIdUri="urn:mpeg:dash:role:2011" value="main"/>
			<BaseURL>avc3-events/</BaseURL>
			<SegmentTemplate startNumber="6" presentationTimeOffset="19200" timescale="1000" duration="3840" media="$RepresentationID$/$Number%06d$.m4s" initialization="$RepresentationID$/IS.mp4" />
			<Representation id="1920x1080i25" codecs="avc3.640028" height="1080" width="1920" frameRate="25" scanType="interlaced" bandwidth="8060152" />
		</AdaptationSet>
		<AdaptationSet startWithSAP="2" segmentAlignment="true" id="2" codecs="mp4a.40.5" audioSamplingRate="48000" lang="eng" mimeType="audio/mp4" >
			<AudioChannelConfiguration schemeIdUri="urn:mpeg:dash:23003:3:audio_channel_configuration:2011" value="2"/>
			<Role schemeIdUri="urn:mpeg:dash:role:2011" value="main"/>
			<BaseURL>audio/</BaseURL>
			<SegmentTemplate startNumber="6" presentationTimeOffset="19200" timescale="1000" duration="3840" media="$RepresentationID$/$Number%06d$.m4s" initialization="$RepresentationID$/IS.mp4" />
			<Representation id="96kbps" bandwidth="96000" />
		</AdaptationSet>
    </Period>
</MPD>

davemevans · 2018-05-22T14:23:27Z

Overall, this is great stuff. The performance is significantly better than without this patch, but I see some strange behaviour (differing between browsers) happening and the result is not completely seamless. I believe the content is correct and should be seamlessly playable.

In Chrome (66, Windows 7), the video always pauses on the frame displaying timecode 00:00:18:18. The audio sounds like it is seamless but it's hard to tell without a continuous tone.

In Firefox (61, Windows 7), the video and audio glitch on the frame displaying timecode 00:00:19:04 (which is the final frame of the first Period).

Example MPD:

Same as in #2580 (comment), but with InbandEventStreams removed.

robertbryer · 2018-05-22T15:28:51Z

src/streaming/SourceBufferSink.js

+            return buffer.buffered;
+        } catch (e) {
+            log('getAllBufferRanges exception: ' + e.message);
+            return [];


I'm expecting this function to either return something null-like, or an object implementing the timeranges interface: .start(n) .end(n) .length. Array here works, because the empty array returns length 0 and then nobody calls start or end in that case, but it seems initially confusing reading this to be returning an array. Maybe add an explainer?

Null it is then

aescarcha · 2018-05-23T12:16:11Z

I added 2 commits to fix @robertbryer issues.

@bbcrddave I can reproduce the InbandEvent issue, but if I remove the <InbandEventStream tag from the mpd it works flawlessly in Chrome 66, and have a small audio glitch in Firefox 60 (both in MacOS).

EDIT: Fixed the InbandEvent issue, I also tested on Windows 10 and saw some frames are dropped while transitioning on Firefox if using old hardware ( i5-540M, RAM 4GB ) but Chrome and Edge worked like a charm. I tested it in a newer laptop and every browser worked fine.

epiclabsDASH · 2018-05-23T20:40:59Z

Thanks for all the great feedback! Given the importance of the changes of this PR and the need of testing it exhaustively, I am going to keep this out of dash.js v2.7.0 (code freeze today). Let's continue testing this and I will do the merge once we feel comfortable with this PR and after releasing dash.js v2.7.0.

davemevans · 2018-05-24T08:13:37Z

I updated Chrome to the latest patch level and it now appears to play smoothly through the transition 👍
I still get an audible glitch and momentary paused frame in Firefox, in all versions (Beta, Nightly etc).

davemevans · 2018-05-24T08:21:01Z

Something weird is happening which, in the worst case, is occasionally causing playback to stop and, in the best case, causing segments to be redownloaded. Given the manifest below, at the period transition, a portion of the buffered but as-yet-unplayed range is removed. Sometimes the segments are redownloaded and appended again, sometimes playback stalls. The buffered range removed is always the same.

Log highlights:

[119488] Requesting seek to time: 19.2
[119501] Seeking to: 19.2
[119503] Removing video buffer from: 0 to 18.7
[119513] Removing video buffer from: 27.38 to 35.06

Sample manifest:

<?xml version="1.0" encoding="UTF-8"?>
<MPD type="static" xmlns="urn:mpeg:dash:schema:mpd:2011" xmlns:dvb="urn:dvb:dash-extensions:2014-1" profiles="urn:dvb:dash:profile:dvb-dash:2014,urn:dvb:dash:profile:dvb-dash:isoff-ext-live:2014" minBufferTime="PT2.049S" maxSegmentDuration="PT3.84S" mediaPresentationDuration="PT38.40S">
	<BaseURL serviceLocation="A" dvb:priority="1" dvb:weight="1">http://rdmedia.bbc.co.uk/dash/ondemand/testcard/1/</BaseURL>
	<Period duration="PT19.2S" start="PT0S">
		<AdaptationSet startWithSAP="2" segmentAlignment="true" id="1" sar="1:1" mimeType="video/mp4" >
			<InbandEventStream schemeIdUri="tag:rdmedia.bbc.co.uk,2014:events/ballposition" value="1"/>
			<Role schemeIdUri="urn:mpeg:dash:role:2011" value="main"/>
			<BaseURL>avc3-events/</BaseURL>
			<SegmentTemplate startNumber="1" timescale="1000" duration="3840" media="$RepresentationID$/$Number%06d$.m4s" initialization="$RepresentationID$/IS.mp4" />
			<Representation id="1920x1080i25" codecs="avc3.640028" height="1080" width="1920" frameRate="25" scanType="interlaced" bandwidth="8060152" />
		</AdaptationSet>
	</Period>
	<Period start="PT19.2S" duration="PT19.2S">
		<AdaptationSet startWithSAP="2" segmentAlignment="true" id="1" sar="1:1" mimeType="video/mp4" >
			<InbandEventStream schemeIdUri="tag:rdmedia.bbc.co.uk,2014:events/ballposition" value="1"/>
			<Role schemeIdUri="urn:mpeg:dash:role:2011" value="main"/>
			<BaseURL>avc3-events/</BaseURL>
			<SegmentTemplate startNumber="6" presentationTimeOffset="19200" timescale="1000" duration="3840" media="$RepresentationID$/$Number%06d$.m4s" initialization="$RepresentationID$/IS.mp4" />
			<Representation id="1920x1080i25" codecs="avc3.640028" height="1080" width="1920" frameRate="25" scanType="interlaced" bandwidth="8060152" />
		</AdaptationSet>
    </Period>
</MPD>

aescarcha · 2018-05-24T09:02:49Z

@bbcrddave fixed in ea21de8 , seek was called if track was only audio or only video, it's weird that it only caused troubled with some streams...

…exception

… while preloading the next one

…, using logger

This was referenced May 22, 2018

No tear down between periods #2575

Closed

Safari/Mac live stream fail fix #2579

Closed

davemevans reviewed May 22, 2018

View reviewed changes

robertbryer reviewed May 22, 2018

View reviewed changes

epiclabsDASH added the Feature request from DASH IF IOP label May 23, 2018

epiclabsDASH added this to the 2.8 milestone May 25, 2018

aescarcha force-pushed the smoothPeriodTransition branch from 6325e95 to c18e23b Compare June 8, 2018 12:25

aescarcha added 11 commits June 20, 2018 09:27

Checking if next period is compatible with the current one

65c3cb2

Reusing buffer when the next period is compatible with the old one

21dd3b7

Prefetch of new stream periods

849d0dd

Fixes for safari multiperiod

72d6d9a

Array to define codec compatibility

f55a5d0

Fix seeking to next period while preloading

54d301a

SourceBufferSink getAllBufferRanges will return null if it throws an …

fce4bda

…exception

Period preload will work with inbandEvents

fe727aa

Audio-only and video-only streams will use multiperiod preload

d793e62

Deactivating preloaded period if seeking back into the current period…

2b3bb85

… while preloading the next one

Won't seek while transitioning period if track is only video or audio…

37f8637

…, using logger

aescarcha force-pushed the smoothPeriodTransition branch from c18e23b to 37f8637 Compare June 20, 2018 07:41

epiclabsDASH merged commit 6565361 into Dash-Industry-Forum:development Jun 20, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Smooth period transition #2580

Smooth period transition #2580

aescarcha commented May 22, 2018

robertbryer commented May 22, 2018

davemevans May 22, 2018

aescarcha May 23, 2018

aescarcha May 23, 2018

davemevans May 24, 2018

davemevans commented May 22, 2018

davemevans commented May 22, 2018

robertbryer May 22, 2018

aescarcha May 23, 2018

aescarcha commented May 23, 2018 •

edited

epiclabsDASH commented May 23, 2018

davemevans commented May 24, 2018

davemevans commented May 24, 2018

aescarcha commented May 24, 2018

Smooth period transition #2580

Smooth period transition #2580

Conversation

aescarcha commented May 22, 2018

robertbryer commented May 22, 2018

davemevans May 22, 2018

Choose a reason for hiding this comment

aescarcha May 23, 2018

Choose a reason for hiding this comment

aescarcha May 23, 2018

Choose a reason for hiding this comment

davemevans May 24, 2018

Choose a reason for hiding this comment

davemevans commented May 22, 2018

davemevans commented May 22, 2018

robertbryer May 22, 2018

Choose a reason for hiding this comment

aescarcha May 23, 2018

Choose a reason for hiding this comment

aescarcha commented May 23, 2018 • edited

epiclabsDASH commented May 23, 2018

davemevans commented May 24, 2018

davemevans commented May 24, 2018

aescarcha commented May 24, 2018

aescarcha commented May 23, 2018 •

edited