Restrict ABR choices based on video element size

[niklaskorz]

Current situation:
Shaka player offers a restriction configuration, the interesting fields in this case being maxWidth and maxHeight.

Behavior of current situation:
The list of available video streams, for ABR and for the user, will be filtered by the restrictions set in the configuration.

Problem with current situation:

1. If no stream matches the configured restrictions, the user will be presented a black screen with audio.
2. Streams that do not pass the filter will no longer be selectable by the user, although this is required if the user knows that the viewport size will change and therefore wants to pre-buffer the desired resolution.

Solutions:

1. A more user-friendly behavior would be to make maxWidth and maxHeight more flexible, choosing the first video stream that at least fits the viewport (stream.width >= maxWidth || stream.height >= maxHeight) as the maximum size.
2. Only apply the restrictions on the stream set passed to the ABR manager. Do not filter the track list that's exposed and selectable by shaka-player.

Possible implementation:
https://github.com/alugha/shaka-player/tree/feature-abr-resolution

Issues with my possible implementation:
I am experiencing a bug where a specific video is not playable if the ABR manager chooses, based on the viewport, the second best resolution in Google Chrome. In this case, and only for this video, the buffering starts at 0.04 instead of 0.00, therefore making it impossible to start playback from the beginning. Seeking to 0.1 or changing the video stream works.
In comparison, dash.js has no issues playing this video with setLimitBitrateToPortal set to true.

[niklaskorz]

Any comments on this?

[joeyparrish]

Sorry for the delay.

The bug where you seek to 0.1 is likely caused by Chrome's mixup of DTS/PTS. If the higher res streams use more b-frames, it's possible that this bug in Chrome would affect those streams, but not the lower-res or lower-profile ones. A solution to #555 could be an effective workaround for this Chrome bug.

Problem with current situation:

1. If no stream matches the configured restrictions, the user will be presented a black screen with audio.

Restricting all streams should be an error. There is no reason to restrict all streams, and no reason to expect playback if all streams are restricted.

Before we go too far into redesigning Restrictions, let me give you some history. The reason we created the Restrictions object was to allow apps to give us hints about things we can't know.

For example, say your content has a policy that forbids HD playback on a certain device. Your license server may be able to communicate to your application that it does not meet the requirements for HD. If there's no other indication that the HD streams are not playable (key statuses, etc), then your application needs to tell the library to forbid those HD streams.

If they are selected, playback will definitely fail. Not only should they not be selected by AbrManager, the user should not be able to select them, either.

That is what Restrictions was originally designed for: app-communicated, DRM-based restrictions. The first extension of Restrictions beyond that was by @sanbornhnewyyz in #32 to add maxBandwidth. @sanbornhnewyyz, what is the high-level purpose of this parameter for you? Are you trying to limit data usage for mobile users?

@niklaskorz, here are my suggestions. These options don't break the concept of serious, DRM-based restrictions:

1. Introduce a second Restrictions object that only applies to AbrManager. The original Restrictions object restricts the availability of tracks for critical reasons. The second Restrictions object (maybe nested under abr in config) will only restrict which tracks AbrManager can select. The user can override these second restrictions. This option is more flexible, and might be a better fit for things like limiting mobile data use. It would be a little more complex, though, since it requires the app to update the config every time the video changes size. At least the app should know exactly when the video changes size, because it is in control of going fullscreen.
2. Add a config flag for AbrManager that is similar to dash.js's setLimitBitrateToPortal. If set, AbrManager will use the size of the video element as an additional limitation. This flag won't affect any other part of the system, such as the track list. This option is simpler for the app developer, but less flexible. It also requires the library to poll the video element's size, which is ugly.
3. A mix of the two. A second Restrictions object, which can be auto-updated with resolution restrictions based on the video element if a certain flag is set.

@niklaskorz, @sanbornhnewyyz, thoughts on these options? @sanbornhnewyyz, am I guessing correctly that you want to limit mobile data usage?

[niklaskorz]

Thanks for your response, @joeyparrish :)
I fully agree that the current restriction set should only fulfill its original purpose.
As for your suggestions, I would opt for the first one, as an additional config object is more flexible (e.g., it would also allow user settings on the application's side) than enforcing the player's size. The third one would also be a possibility, but not really necessary as the program usually knows about any size changes happening to its video element, thus being able to apply them to the Shaka configuration if needed.

[ondrejpar]

Hi @joeyparrish ,
As a side effect, the last commit removed minAudioBandwidth, maxAudioBandwidth and the corresponding Video restrictions - they were replaced with minBandwidth and maxBandwidth.
This is the situation we were in with Shaka 1.6.x. The "Shaka v2 Upgrade Guide" still states that the restrictions were split.
Is this how the ABR is supposed to work from now on? Actually, I quite liked the possibility to restrict audio and video streams separately.
If this is expected behavior, the upgrade.md should probably be changed too.

[joeyparrish]

We will update the guide for v2.1. In v2.1, we will support HLS, where we don't know the individual bandwidth requirements of video and audio. Instead, we will only be able to use the bandwidth of the variant (video+audio combination).

The addition of a second restrictions object will only be available in v2.1 and will not be backported to v2.0.x. I apologize for the inconvenience.

[dobrusev]

Hi @joeyparrish I think adding restrictions to the AbrManager is good idea.

I was looking into a way how to change BANDWIDTH_UPGRADE_TARGET_ in SimpleAbrManager to value I think fits best to my streams specifics. I have a few design restrictions and implementing custom abrManager is not an option for me.

Just a few questions:

• Is there a way how SimpleAbrManager to be parametrized to be able to set BANDWIDTH_UPGRADE_TARGET_?
• Is there a way to add BANDWIDTH_UPGRADE_TARGET_, BANDWIDTH_DOWNGRADE_TARGET_ and SWITCH_INTERVAL_MS to abr restrictions?
• Is there a way how to use maxBandwidth to influence chooseVideoStream_ algorithm?

Thank you in advance!

[joeyparrish]

@dobrusev,

I've filed a separate issue (#744) to make those things configurable.

As for maxBandwidth, that would restrict which video streams are available for AbrManager to choose.

[dobrusev]

@joeyparrish thank you very much!

[leandromoreira]

@joeyparrish I liked your suggestion number 2.

Add a config flag for AbrManager that is similar to dash.js's setLimitBitrateToPortal. If set, AbrManager will use the size of the video element as an additional limitation. This flag won't affect any other part of the system, such as the track list. This option is simpler for the app developer but less flexible. It also requires the library to poll the video element's size, which is ugly.

I was browsing through the docs looking for something similar but I didn't find it, the only thing I found was to restrict statically the max|min Height|Width|Pixels, anyway, is anyone working on this? Or are you still interested in this feature?

Btw, HLS.js also has this feature under the config capLevelToPlayerSize.

For sure HLS.js also needs to query the size from time to time.