Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Restrict ABR choices based on video element size #565

Closed
niklaskorz opened this issue Oct 22, 2016 · 9 comments
Closed

Restrict ABR choices based on video element size #565

niklaskorz opened this issue Oct 22, 2016 · 9 comments
Assignees
Labels
status: archived Archived and locked; will not be updated type: enhancement New feature or request
Milestone

Comments

@niklaskorz
Copy link
Contributor

niklaskorz commented Oct 22, 2016

Current situation:
Shaka player offers a restriction configuration, the interesting fields in this case being maxWidth and maxHeight.

Behavior of current situation:
The list of available video streams, for ABR and for the user, will be filtered by the restrictions set in the configuration.

Problem with current situation:

  1. If no stream matches the configured restrictions, the user will be presented a black screen with audio.
  2. Streams that do not pass the filter will no longer be selectable by the user, although this is required if the user knows that the viewport size will change and therefore wants to pre-buffer the desired resolution.

Solutions:

  1. A more user-friendly behavior would be to make maxWidth and maxHeight more flexible, choosing the first video stream that at least fits the viewport (stream.width >= maxWidth || stream.height >= maxHeight) as the maximum size.
  2. Only apply the restrictions on the stream set passed to the ABR manager. Do not filter the track list that's exposed and selectable by shaka-player.

Possible implementation:
https://github.com/alugha/shaka-player/tree/feature-abr-resolution

Issues with my possible implementation:
I am experiencing a bug where a specific video is not playable if the ABR manager chooses, based on the viewport, the second best resolution in Google Chrome. In this case, and only for this video, the buffering starts at 0.04 instead of 0.00, therefore making it impossible to start playback from the beginning. Seeking to 0.1 or changing the video stream works.
In comparison, dash.js has no issues playing this video with setLimitBitrateToPortal set to true.

@niklaskorz
Copy link
Contributor Author

Any comments on this?

@joeyparrish joeyparrish added this to the vLater milestone Nov 9, 2016
@joeyparrish joeyparrish added the type: enhancement New feature or request label Nov 9, 2016
@joeyparrish
Copy link
Member

Sorry for the delay.

The bug where you seek to 0.1 is likely caused by Chrome's mixup of DTS/PTS. If the higher res streams use more b-frames, it's possible that this bug in Chrome would affect those streams, but not the lower-res or lower-profile ones. A solution to #555 could be an effective workaround for this Chrome bug.

Problem with current situation:

  1. If no stream matches the configured restrictions, the user will be presented a black screen with audio.

Restricting all streams should be an error. There is no reason to restrict all streams, and no reason to expect playback if all streams are restricted.

Before we go too far into redesigning Restrictions, let me give you some history. The reason we created the Restrictions object was to allow apps to give us hints about things we can't know.

For example, say your content has a policy that forbids HD playback on a certain device. Your license server may be able to communicate to your application that it does not meet the requirements for HD. If there's no other indication that the HD streams are not playable (key statuses, etc), then your application needs to tell the library to forbid those HD streams.

If they are selected, playback will definitely fail. Not only should they not be selected by AbrManager, the user should not be able to select them, either.

That is what Restrictions was originally designed for: app-communicated, DRM-based restrictions. The first extension of Restrictions beyond that was by @sanbornhnewyyz in #32 to add maxBandwidth. @sanbornhnewyyz, what is the high-level purpose of this parameter for you? Are you trying to limit data usage for mobile users?

@niklaskorz, here are my suggestions. These options don't break the concept of serious, DRM-based restrictions:

  1. Introduce a second Restrictions object that only applies to AbrManager. The original Restrictions object restricts the availability of tracks for critical reasons. The second Restrictions object (maybe nested under abr in config) will only restrict which tracks AbrManager can select. The user can override these second restrictions. This option is more flexible, and might be a better fit for things like limiting mobile data use. It would be a little more complex, though, since it requires the app to update the config every time the video changes size. At least the app should know exactly when the video changes size, because it is in control of going fullscreen.
  2. Add a config flag for AbrManager that is similar to dash.js's setLimitBitrateToPortal. If set, AbrManager will use the size of the video element as an additional limitation. This flag won't affect any other part of the system, such as the track list. This option is simpler for the app developer, but less flexible. It also requires the library to poll the video element's size, which is ugly.
  3. A mix of the two. A second Restrictions object, which can be auto-updated with resolution restrictions based on the video element if a certain flag is set.

@niklaskorz, @sanbornhnewyyz, thoughts on these options? @sanbornhnewyyz, am I guessing correctly that you want to limit mobile data usage?

@niklaskorz
Copy link
Contributor Author

Thanks for your response, @joeyparrish :)
I fully agree that the current restriction set should only fulfill its original purpose.
As for your suggestions, I would opt for the first one, as an additional config object is more flexible (e.g., it would also allow user settings on the application's side) than enforcing the player's size. The third one would also be a possibility, but not really necessary as the program usually knows about any size changes happening to its video element, thus being able to apply them to the Shaka configuration if needed.

@TheModMaker TheModMaker self-assigned this Jan 5, 2017
@ondrejpar
Copy link

Hi @joeyparrish ,
As a side effect, the last commit removed minAudioBandwidth, maxAudioBandwidth and the corresponding Video restrictions - they were replaced with minBandwidth and maxBandwidth.
This is the situation we were in with Shaka 1.6.x. The "Shaka v2 Upgrade Guide" still states that the restrictions were split.
Is this how the ABR is supposed to work from now on? Actually, I quite liked the possibility to restrict audio and video streams separately.
If this is expected behavior, the upgrade.md should probably be changed too.

@joeyparrish
Copy link
Member

We will update the guide for v2.1. In v2.1, we will support HLS, where we don't know the individual bandwidth requirements of video and audio. Instead, we will only be able to use the bandwidth of the variant (video+audio combination).

The addition of a second restrictions object will only be available in v2.1 and will not be backported to v2.0.x. I apologize for the inconvenience.

@dobrusev
Copy link

Hi @joeyparrish I think adding restrictions to the AbrManager is good idea.

I was looking into a way how to change BANDWIDTH_UPGRADE_TARGET_ in SimpleAbrManager to value I think fits best to my streams specifics. I have a few design restrictions and implementing custom abrManager is not an option for me.

Just a few questions:

  • Is there a way how SimpleAbrManager to be parametrized to be able to set BANDWIDTH_UPGRADE_TARGET_?
  • Is there a way to add BANDWIDTH_UPGRADE_TARGET_, BANDWIDTH_DOWNGRADE_TARGET_ and SWITCH_INTERVAL_MS to abr restrictions?
  • Is there a way how to use maxBandwidth to influence chooseVideoStream_ algorithm?

Thank you in advance!

@joeyparrish
Copy link
Member

@dobrusev,

I've filed a separate issue (#744) to make those things configurable.

As for maxBandwidth, that would restrict which video streams are available for AbrManager to choose.

@joeyparrish joeyparrish modified the milestones: Backlog, v2.1.0 Mar 31, 2017
@dobrusev
Copy link

dobrusev commented Apr 6, 2017

@joeyparrish thank you very much!

@leandromoreira
Copy link
Contributor

leandromoreira commented Feb 16, 2018

@joeyparrish I liked your suggestion number 2.

Add a config flag for AbrManager that is similar to dash.js's setLimitBitrateToPortal. If set, AbrManager will use the size of the video element as an additional limitation. This flag won't affect any other part of the system, such as the track list. This option is simpler for the app developer but less flexible. It also requires the library to poll the video element's size, which is ugly.

I was browsing through the docs looking for something similar but I didn't find it, the only thing I found was to restrict statically the max|min Height|Width|Pixels, anyway, is anyone working on this? Or are you still interested in this feature?

Btw, HLS.js also has this feature under the config capLevelToPlayerSize.

For sure HLS.js also needs to query the size from time to time.

@shaka-project shaka-project locked and limited conversation to collaborators Mar 22, 2018
@shaka-bot shaka-bot added the status: archived Archived and locked; will not be updated label Apr 15, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
status: archived Archived and locked; will not be updated type: enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

7 participants