Prefetch audio variants of a particular language for seamless audio switch.

[gkatsev]

Have you read the FAQ and checked for duplicate open issues?
Yes, there isn't something specific, though it builds on top of #4784.

Is your feature request related to a problem? Please describe.

I want to switch audio streams with a small a gap in playback as possible.

Describe the solution you'd like

Extend segment prefetch to also prefetch alternate audio streams.

Describe alternatives you've considered

Additional context

We had a requirement for seamless audio switch, and to enable that we extended the segment prefetch feature to be able to prefetch the segments of the alternate audio streams. Then, when we want to switch streams, we use the segment prefetch that's mapped to the specific audio stream.

To do this, we added the following options, one of which can also help reduce prefetched segments:

• streaming.prefetchAudioLanguages: A list of languages that we match audio streams against for knowing whether we should prefetch that stream. i.e ['en', 'es', 'jp'] to English, Spanish, and Japanese audio languages. Or ['en'] if you had multiple different English language audio tracks.
• disableVideoPrefetch: don't prefetch video segments.

More specific technical details: modify segment prefetch to not delete segments on get. Instead, it has an evict method which is given a time, and it removes segments before this time. This allows keeping the current segment prefetched so that if we switch to another audio stream and then back, we already have the segment available. Then in the streaming engine, we create a map between all the audio streams we care about and their associated segment prefetches and when the variant switches we grab the relevant segment prefetch from the map. In conjunction with setting rebufferingGoal to 0 or a very small number, the switches between audio streams can happen quickly enough that they aren't noticeable.

This work is already complete, and we'd like to contribute it upstream. We tried to make the options be generic, but there's definitely room to refine these if necessary.

[avelad]

More specific technical details: modify segment prefetch to not delete segments on get. Instead, it has an evict method which is given a time, and it removes segments before this time.

I agree to change the prefetch management. Please do a single PR with this change.

To do this, we added the following options, one of which can also help reduce prefetched segments:

* `streaming.prefetchAudioLanguages`: A list of languages that we match audio streams against for knowing whether we should prefetch that stream. i.e `['en', 'es', 'jp']` to English, Spanish, and Japanese audio languages. Or `['en']` if you had multiple different English language audio tracks.

* `disableVideoPrefetch`: don't prefetch video segments.

We currently use prefetch for Low Latency and will use it in Preload API.

So there must be the following options at least:
disableVideoPrefetch (default false)
disableAudioPrefetch (default false)
disableTextPrefetch (default false)
prefetchAudioLanguages (default[])

If disableAudioPrefetch is false, we ignore prefetchAudioLanguages. If disableAudioPrefetch is true, then we use prefetchAudioLanguages and only prefetch the specified audios.

If you agree with this I will be happy to review the implementation of the two PRs

[gkatsev]

Sounds good. I'll probably get started with what we have already and then update it with the new options. And I'll be happy to break up it up into multiple PRs if that's wanted.

[OrenMe]

this sounds like a really cool feature, I wanna make sure I understood it. So incentive is to have ABR like switch experience but for audio where user can switch languages, it will switch as quickly as possible but user will not experience a pause while audio changes?
And tradeoff is the amount of buffer ahead for audio? to not keep too much cause we are now saving more audio tracks?
How does this scale on low memory devices? Smart TVs for example?
So I need to pass streaming.prefetchAudioLanguages array of languages, but didn't understand why I will need to set disableVideoPrefetch in this case? and how does it relate to setting rebufferingGoal to 0? won't it come on expense of robustness in case of unstable network?

[gkatsev]

it will switch as quickly as possible but user will not experience a pause while audio changes?

yup, that's exactly right. In reality, there's the slightest of gaps, but it's pretty unnoticeable. Most importantly is that the player doesn't need to wait to download more segments and can switch instantly. This is also why a low rebufferingGoal is needed. It represents the amount of buffer needed before playback starts, and given we've already prefetched at least one segment worth of content, we don't need shaka to wait before playback starts.

How does this scale on low memory devices? Smart TVs for example?

We're still doing more testing. However, we haven't seen many issues assuming rebufferingGoal isn't set to 0 or a very very low number. This is also why we thought that disableVideoPrefetch might be helpful. So, we're not keeping large segments in js memory beyond them getting appended to the source buffer.

didn't understand why I will need to set disableVideoPrefetch in this case

The thought here is to minimize memory usage so that we don't keep the video segments downloaded if we're potentially keeping a bunch of audio segments downloaded. However, we're not sure if it's strictly necessary but can't imagine that it hurts to have this option.