Shaka 3 rebuffers indefinitely on VDMS dash live stream after 3-5 minutes (works fine in shaka 2.x)

[TheJohnBowers]

Have you read the FAQ and checked for duplicate open issues?
Yes

What version of Shaka Player are you using?
3.0.8

Can you reproduce the issue with our latest release version?
Yes

Can you reproduce the issue with the latest code from master?
Yes - But I cannot reproduce it in shaka 2.4/2.5

Are you using the demo app or your own custom app?
Demo App

If custom app, can you reproduce the issue using our demo app?
NA

What browser and OS are you using?
Chrome on macos

For embedded devices (smart TVs, etc.), what model and firmware version are you using?
NA

What are the manifest and license server URIs?

Manifest:
https://content-ausw7.uplynk.com/channel/b965d55496414804891dde9c96928ffb.mpd?rays=abcd&delay=30
License:
https://content.uplynk.com/wv

What did you do?

Just play back any Verizon Media Dash Live stream for for 5 minutes or so and you will get infinite rebuffering

What did you expect to happen?
Playback without infinite rebuffering

What actually happened?

I recorded a little demo video that guides you through what is happening:

https://content.uplynk.com/player5/PjpqgfaCDjuJxkQHgrBvXsa.html

Basically after ~ 3 minutes shaka starts redownloading all slices after a certain slice after ever manifest update it gets to a point where it is downloading more than a dozen slices after each manifest refresh and then it starts infinitely buffering.

[TheModMaker]

Probably a duplicate of #3139, but different enough that I won't mark this as a duplicate.

[avelad]

@ismena, I have seen that you have assigned this issue, is there any progress with this problem, do you think it will be fixed for the next release?

[ismena]

Huh, apparently I am indeed assigned to this issue o.O
I haven't had a chance to look yet, but okay, I'll bite the bullet and shoot to fix it for 3.1

[ismena]

Alright, my fix for #3139 doesn't fix this :(
@theodab is stepping in to help me out and make sure this is solved for 3.1

[avelad]

@theodab @ismena do you have any update?

[avelad]

@ismena @theodab Do you have an update on this issue? I am very interested.

[joeyparrish]

The team was pretty badly divided among other non-open-source projects at Google for a while, but we're mostly back together and focused on Shaka Player again. This morning we spent some time shuffling issues and priorities around, with the goal to get v3.1 done by the end of the month. This morning we decided that this is one of three top-priority issues, and this one has been assigned to me.

I'm beginning work on diagnosis now.

[joeyparrish]

With v3.0.10, I can reproduce the issue as you describe in your video.

With the latest code from the master branch, I get different behavior. Instead of re-downloading segments, at some point, it stops fetching segments entirely. It doesn't show a buffering spinner.

In that branch, in this state, seeking to the live edge will cause it to fetch a few segments, but then it stops again. The frame on the screen never seems to update.

I see these interesting Chrome media logs on master:

timestamp	property	value
00:00:50.949	info	"DecryptingDemuxerStream: no key for key ID 36A883B024EB4CC9BAF50CC5E30B571E; will resume decrypting after new usable key is available"
00:00:51.380	info	"DecryptingDemuxerStream: no key for key ID E9F4E449B45149DA9FE1739A1FDA3F8D; will resume decrypting after new usable key is available"
00:00:51.460	kVideoPlaybackRoughness	18.369
00:00:51.460	kVideoPlaybackFreezing	0.03351
00:00:51.460	kFramerate	30
00:00:51.460	pipeline_buffering_state	{"for_suspended_start":false,"reason":"DEMUXER_UNDERFLOW","state":"BUFFERING_HAVE_NOTHING"}

It looks to me like with the master branch, we fetch ahead to meet the buffering goal, but we can't play that buffer out because of a missing key.

So it appears that the re-downloading behavior is fixed in master, revealing another issue which is related to key rotation.

[joeyparrish]

Normally, the way external key rotation should work is that we see new PSSH boxes in the content at some point, and we use those to trigger a new key session and fetch an updated license. These new PSSH boxes make their way to us through the encrypted event from the video element. This event fires any time the platform encounters a PSSH box in MP4 content.

However, I see your MPD has a PSSH embedded in it. When we see a PSSH embedded in the manifest, we don't listen for encrypted events. Instead, the PSSH in the manifest overrides anything that we might find in the content.

For external key rotation, you don't want this. So you can use the manifest.dash.ignoreDrmInfo config to ignore what's in the manifest and only use the encrypted event.

When I tried this, however, I found that there aren't any encrypted events at all from your content. I've seen this before, in content where the PSSH in the manifest is the only PSSH. To me, this wouldn't make sense in a stream with key rotation, though.

[TheJohnBowers]

So this sort of key rotation works with shaka 2.4/2.5 No issues. I definitely disagree that this is bad content. If the PSSH can be exclusively included in a manifest and not in the encoded content, I don't see where there is justification to say that key rotation should not work in this scenario.

[joeyparrish]

Sorry, maybe it was rash to call it bad content. What I should say instead is "I don't understand how this content works". :-)

[joeyparrish]

I see where I made a mistake looking at the MPD: there is a PSSH in the manifest per period. So the rotation should be handled by the manifest updates, and PSSH boxes are not needed in the content at all.

I'll double-check that we're reacting to those properly and fetching licenses for them.

[joeyparrish]

The key rotation issue seems to be caused by acfa1a8, which is not in any release version.

In that change, a call to filter the manifest on update was removed from the DASH parser. This filtering had a non-obvious side-effect of updating the init data in the DRM engine and triggering new sessions for key rotation.

With that call restored (though that may or may not be the best solution), I'm able to play for quite a long time so far. I think that fixed it in the master branch.

One more issue I see is a failed assertion several minutes in: "segment_index.js:656 Assertion failed: Cannot have evicted segments in non-first Periods". I'm seeing this in both v3.0.10 and master, and it doesn't coincide with the playback failure in v3.0.10, so it could be unrelated.

So, here's what's left to sort out:

• What change in master fixes the re-downloading issue? It should be cherry-picked to v3.0.x.
• What is the best fix for the broken key rotation in master? This needs to be fixed ASAP.
• What is causing the failed assertion? Is the assertion wrong, or have we messed up the state of MetaSegmentIndex somewhere? At what change did this assertion start failing?

[TheJohnBowers]

If you would like a similar test stream that does not rotate keys every 3 minutes to compare and contrast:

https://content.uplynk.com/channel/410beb25b7db4e4fb2e8c42724c07f67.mpd
https://content.uplynk.com/wv

[joeyparrish]

• What change in master fixes the re-downloading issue? It should be cherry-picked to v3.0.x.

I performed a git bisect to track down the fix in master that is still needed in v3.0.x. At each step, I had to manually add back the filter() call I mentioned that was removed in acfa1a8. This process revealed that the redownloading issue was fixed in 7e50866. We will cherry-pick this to v3.0.x, so this should fix the issue in v3.0.11.

[joeyparrish]

• What is causing the failed assertion? Is the assertion wrong, or have we messed up the state of MetaSegmentIndex somewhere? At what change did this assertion start failing?

git bisect reveals that this assertion has been failing on your content since 78357ed, which is also when the assertion was first introduced. I'll dig deeper to find out how we can fix it.

[joeyparrish]

I found the root cause of the issue with the failed assertion, and the result seems to be harmless.

The assertion is meant to double-check our assumptions, namely that we evict content from earlier to later. If we've evicted anything from a period, it means that everything in earlier periods should have been evicted, too.

Except that in practice, the eviction process is driven by updates and merging. When a period vanishes from the manifest in an update (as it should), nothing in the parser triggers the final evict() call on that period's SegmentIndex. The result is that we hold onto one or a very small number of references per index, but playback continues anyway because they will never be used.

[joeyparrish]

Here's the final summary:

I have a fix for the failed assertion in review. Again, the assertion failure is harmless to playback.

I also have a fix in review for the key rotation failure in the master branch. This will ensure we don't lose key rotation when v3.1 comes out. This issue does not affect v3.0.

The fix for the redownloading issue is already in the master, and I'm working on a round of cherry-picks from master to v3.0.x which will appear in v3.0.11.

Thanks!

[joeyparrish]

The fix for the master branch key rotation issue has been merged. Everything should be working in the nightly build after tonight.

The fix for the redownloading issue has been cherry-picked, but not yet pushed to v3.0.x until I can verify that all (29) cherry-picks for this branch so far are correct and passing tests. I have a few test failures to investigate before I can push those changes to that branch, but the fix will be out in v3.0.11 shortly afterward.