Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Automatic ABR quality restrictions based on size #2333

Closed
alejandroiglesias opened this issue Jan 8, 2020 · 15 comments · Fixed by #4404 or #4273
Closed

Automatic ABR quality restrictions based on size #2333

alejandroiglesias opened this issue Jan 8, 2020 · 15 comments · Fixed by #4404 or #4273
Assignees
Labels
priority: P3 Useful but not urgent status: archived Archived and locked; will not be updated type: enhancement New feature or request
Milestone

Comments

@alejandroiglesias
Copy link

alejandroiglesias commented Jan 8, 2020

Have you read the FAQ and checked for duplicate open issues?
This was discussed as a part of #565, but the limit based on video container size wasn't implemented.

Is your feature request related to a problem? Please describe.

A functionality that Dash.js and hls.js have is the ability to limit which quality levels are selected based on video element size.

Describe the solution you'd like

On the ABR settings, an extra flag could be added to instruct the ABR manager to limit the quality level based on the video element. For example:

{
  abr: {
    capLevelToPlayerSize: true
  }
}

Or:

{
  abr: {
    limitBitrateByPortal: true
  }
}

Those are the options used in Dash.js and hls.js. You can choose any name fits for this project.

Describe alternatives you've considered

Without this feature, a possible solution would be to introduce a ResizeObserver on the video element to react to changes in size and dynamically set the ABR restrictions on Shaka player.

Additional context

This would allow saving time (and potential problems with the orchestration of the ResizeObserver and the dynamic setting of restrictions) that can arise in the implementation of a video player that handles bandwidth in a smart way not to load a video quality that is higher than required at a given moment.

@joeyparrish
Copy link
Member

ResizeObserver is not available on Edge (until it becomes Chromium-based), IE (which we will drop support for some day), or Safari (which is a bigger issue). If we restrict ABR based on the size of the video element at the level of Shaka Player, our options seem to be limited to:

  1. ResizeObserver (except on IE, Edge, Safari)
  2. Polling getClientRects() (fallback for IE, Edge, & Safari perhaps?)

I worry about the expense of polling, especially on embedded devices.

A cheap and easy time to check the size is when we go into or out of fullscreen mode, which there are universal events for.

It's not clear to me in what other circumstances an application would change the size of the video element. But it does seem clear the application always knows when that happens, because it is the thing changing the size of the element.

So I have always felt that the most efficient way to do this is at the app level. But I'll look into what dash.js and hls.js do. Perhaps there's something else I've overlooked.

@joeyparrish
Copy link
Member

hls.js polls the video size once per second to implement the capLevelToPlayerSize option. They use clientWidth and clientHeight to determine the size.


dash.js adds a resize event listener on the window, which only reacts to changes in the window size, not other layout changes. This could be useful in some cases, but it wouldn't, for example, be able to account for pure app-driven layout changes, such as something equivalent to YouTube's "theater mode" button.

It's not obvious that this is connected to their limitBitrateByPortal option, though. It seems that their approach is just to recompute the size every time they make an ABR decision, which is pretty clever compared to polling every second.


Neither relies on ResizeObserver, which I've just found will be supported in Safari TP according to caniuse.com.

One last note from my research: I've also discovered that there is a resize event on the video element, which only reacts to changes in the decoded frame size, not layout changes. This is equivalent to getting an event fired when you make an ABR decision, so this is misleading and not useful.

@joeyparrish
Copy link
Member

In general, I'm in favor of this option, and I'm in favor of the default being true. It seems very helpful to the service provider to help them more easily avoid wasting bandwidth on higher resolutions just to downscale them on the device, not to mention saving the end-user's battery.

An event-driven solution, such as ResizeObserver, would seem ideal. It would require no changes to the plugin interface, just a new option. But we probably can't get that on all platforms, especially when we consider TVs.

I'd really like to avoid polling, since polling turns out to be surprisingly expensive in specific cases on devices. (See #2252)

Another alternative is to make AbrManager instances aware of the video element and measure its size every time it makes a decision, as dash.js does. This is probably our best option for robustness, but may require changing the plugin interface.

@joeyparrish joeyparrish added type: enhancement New feature or request and removed needs triage labels Jan 8, 2020
@alejandroiglesias
Copy link
Author

alejandroiglesias commented Jan 15, 2020

@joeyparrish I appreciate the time you took into investigating this. Thank you. I looked at the smallest ResizeObserver ponyfill I found, the resize-observer npm module, which is based on polling window.getComputedStyle() on requestAnimationFrame callbacks. Do you think this could be a performance issue on certain devices? Why? Shouldn't requestAnimationFrame callbacks happen on spare cycles?

Otherwise, a mix of ResizeObserver for supporting browsers and making AbrManager instances aware of the video element and measure its size every time it makes a decision, as dash.js does, for non-supporting browsers would be ideal. I assume this would mean reacting instantly on supporting browsers and reacting with a delay on non-supporting ones. Regarding the latter, how often are ABR decisions made? And are they constantly made or it stops after reaching the upper limits?

@joeyparrish
Copy link
Member

I suspect that making AbrManager aware of the video element should be enough. I'm not sure we should need to poll. The default AbrManager (by design) doesn't make decisions all the time, but only once every so many seconds (8 by default).

A custom AbrManager can already make decisions whenever it likes, ignoring the config value for abr.switchInterval, and it can also poll the video element size or add ResizeObservers.

So I think checking the element size when we make decisions should be good enough for the default AbrManager. The only risk in this approach is changing a plugin interface, which must be done carefully and with an eye toward backward compatibility if possible.

@alejandroiglesias
Copy link
Author

alejandroiglesias commented Jan 15, 2020

@joeyparrish I agree that checking the video element size every few seconds (8 by default) seems more than enough. What changes in the interface do you expect for this to happen? And do you expect this functionality to be implemented by you or any member of the team in the short term?
Also, can you provide a quick intro on how to create a custom AbrManager that works as the default one but in which I can add polling of the video element size on every decision so that I can work around this functionality until it's implemented in the core?

@matvp91
Copy link
Contributor

matvp91 commented Jan 21, 2020

@alejandroiglesias if you want to introduce polling outside of shaka, I guess it would make sense to use the restrictions config in AbrConfiguration to match the size of your video element. This way, there is no need to extend the abr manager's functionality.

@joeyparrish joeyparrish added this to the Backlog milestone Jan 29, 2020
@Maxou44
Copy link

Maxou44 commented Jul 21, 2020

This feature will be amazing! 🥳

I think a mix of ResizeObserver and a fallback could be a great solution to get the player size 🤔

With a dash manifest with 720p + 1080p streams and a player resolution of 850p, it could be nice to choose if we want to use the biggest or the smallest video stream (if we don't match the exact resolution) 🙂

@Wuschba
Copy link

Wuschba commented Aug 6, 2020

Thanks @TheModMaker to point me to this issue. He suggested using:

player.configure({
  	abr: {
		restrictions: {
  			maxWidth: 400,
                        maxHeight: 300
  		}
  	}
  });

to limit the size of the video choosen by the abr-manager. This works inside a webpage which is not changing in size, but doesn't work for example when switching to fullscreen-mode.

In my opinion a polling or continuous tracking of size-changes isn't necessary, it would be greate if maxWidth and maxHeight just adapt when shaka.abr.SimpleAbrManager.chooseVariant is called the next time.

@joeyparrish
Copy link
Member

You make a good point. That would update the restrictions every abr decision interval, which by default is every 8 seconds. That would be sure to be more performant than polling, and at such a long interval, probably close enough in battery performance to listening for size change events.

@Doolali
Copy link

Doolali commented Oct 6, 2020

I fixed the problem using restrictions:

this.player.configure({
    restrictions: {
        maxHeight: height
    }
});

Where height comes from a window resize event.

@TheModMaker TheModMaker added the priority: P3 Useful but not urgent label Sep 29, 2021
@sunknudsen
Copy link

Did any of you manage to set ABR restrictions that adapt to player size?

@joeyparrish
Copy link
Member

ResizeObserver is not available on all supported platforms, but is now available on all evergreen browsers. And the platforms that don't support ResizeObserver (TVs, set top boxes, etc) also don't support resizing.

So I think it would be reasonable to add this as a built-in feature, inside something like if (window.ResizeObserver).

@EyMaddis
Copy link

EyMaddis commented Jun 2, 2022

@Doolali the problem is that the restrictions might actually prevent playback if the video element is smaller.
This is not relevant for full screen players, but for example for small preview videos.

In a perfect world I would like to give a suggested or target resolution where shaka can for example decide to use a larger version if the screen is around 1000px in height, it could choose 1080p.

@joeyparrish
Copy link
Member

If you use abr.restrictions instead, playback will still be possible. These are restrictions on adaptation, whereas restrictions is a hard playback restriction (meant to match up with DRM policies the player wouldn't know about).

@EyMaddis, in your example, if abr.restrictions.maxHeight is set to 1000, but the smallest resolution available is 1080p, then 1080p will still be played.

@github-actions github-actions bot locked as resolved and limited conversation to collaborators Oct 11, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
priority: P3 Useful but not urgent status: archived Archived and locked; will not be updated type: enhancement New feature or request
Projects
None yet