Allow EwmaBandwidthEstimator values to be configured

[kchudoba]

Have you read the Tutorials?
yes

Have you read the FAQ and checked for duplicate open issues?
yes

What version of Shaka Player are you using?
3.1.0

Please ask your question
I have spent hours analyzing the network traffic and shaka player's behavior. I also compared it with other players.

My findings/conclusions:

Shaka's ABR is extremely conservative. My 1080p stream can play fine and has some seconds in the buffer, yet shaka decides to switch to 360p out of nowhere.

I monitored the estimated bandwidth every second and noticed that when I select 1080p manually, the estimated BW goes up significantly, compared to the lower quality streams. I assume that this is because smaller chunks have the same initial lag as the big ones, before the actual data is being transferred. Hence, if you simply use the chunk size and download time to estimate the bandwidth, you will get lower estimates. It then results in a vicious cycle, as the lower the estimate, the lower the stream quality and lower chunk sizes => even lower BW estimates etc etc.

I am not sure how the other players do it, but e.g. bitmovin player plays the same stream with 1080p without problems and changes in bitrate.

Perhaps, instead of using a hypercomplicated bandwidth estimation formula, it would be better to simply monitor the buffer level? Say, if the buffer length is increasing, or is at its max level, the player can try to switch to a higher bitrate variant. If the buffer level then starts degrading, it would switch back to the lower bitrate.

Another question - do you use AVERAGE-BANDWIDTH from the HLS manifest in the switching algorithm or the maximum one?

And finally - is there a way to change the bandwidth estimator function by means of configuration (without messing with the sources)? For example, I would like to try and make the EWMAs longer, or even change to SMAs.
in my case, some of the video segments are not cached in the CDN, which results in longer download times - but they are not many and thus it wouldn't really affect the playback, except for the ABR, which panics and switches to a low bitrate stream. I would thus like to smoothen out the bandwidth estimation, to take more segments into account.

[joeyparrish]

You have made some great observations. Replies inline:

Shaka's ABR is extremely conservative. My 1080p stream can play fine and has some seconds in the buffer, yet shaka decides to switch to 360p out of nowhere.

It is very admittedly conservative! We would always prefer to downgrade instead of buffering. There are ways to do streaming and bandwidth estimation that are capable of utilizing a larger percentage of the user's actual bandwidth, but we designed ours with a goal to keep it simple and easy to maintain.

For an example of a way to get higher utilization, you could measure the startup latency of each connection (time to first byte or TTFB), and exclude that time from the throughput measurements. Then, you could overlap segment requests such that the startup latency of segment N+1 overlaps with the end of the download of segment N. This would achieve higher throughput for the same bandwidth, and therefore higher quality. But it's also more complicated.

Instead, we count the startup latency in our throughput measurements, and build our bandwidth estimate from those. We overlap audio, video, and text segment requests, but not multiple requests from any one stream. The result is a lower estimate of bandwidth, but it's an estimate that is an accurate reflection of the way we stream content. It's not so much an estimate of the user's true bandwidth, as much as it is an estimate of how much bandwidth our simplistic streaming system is capable of utilizing.

That said, our overall ABR algorithm has not been revisited since our launch in December 2014, and very few changes to our streaming/fetching algorithm have been made in that time. We're very open to changing these things.

I monitored the estimated bandwidth every second and noticed that when I select 1080p manually, the estimated BW goes up significantly, compared to the lower quality streams. I assume that this is because smaller chunks have the same initial lag as the big ones, before the actual data is being transferred. Hence, if you simply use the chunk size and download time to estimate the bandwidth, you will get lower estimates. It then results in a vicious cycle, as the lower the estimate, the lower the stream quality and lower chunk sizes => even lower BW estimates etc etc.

This is absolutely true. What I would suggest is that get the bitrate requirements of your content (when 1080p is selected manually, check player.getVariantTracks().find(t => t.active).bandwidth), and compare that to our bandwidth estimate (which can be found via player.getStats().estimatedBandwidth). Both numbers are in bits per second. If the bandwidth estimate is sometimes lower than the bitrate requirement of the 1080p stream, then today's player and default plugins won't play it consistently.

However, if the bandwidth estimate is consistently higher, you may be able to tweak the configuration to improve the quality we select. By default, and again, because we are very conservative, we only upgrade if that will result in us using 85% or less of the estimated bandwidth. And if the active stream is using more than 95% of the estimated bandwidth, we downgrade to avoid buffering. But both of these numbers are configurable:

// Downgrade if more than 98% of estimate is used
player.configure('abr.bandwidthDowngradeTarget', 0.98);
// Upgrade if it would use less than 95% of the estimate
player.configure('abr.bandwidthUpgradeTarget', 0.95);

Perhaps, instead of using a hypercomplicated bandwidth estimation formula, it would be better to simply monitor the buffer level? Say, if the buffer length is increasing, or is at its max level, the player can try to switch to a higher bitrate variant. If the buffer level then starts degrading, it would switch back to the lower bitrate.

That sounds very reasonable. The only caution I would give is to add some kind of hysteresis (like our upgrade/downgrade targets) to avoid frequent bitrate switching. Since we stop buffering for a while when a certain goal is met (config streaming.bufferingGoal), there will always be times when the buffer fullness drops. If you react to every change in buffer fullness, you would downgrade and upgrade on every segment or two.

Another question - do you use AVERAGE-BANDWIDTH from the HLS manifest in the switching algorithm or the maximum one?

A quick search of lib/hls/ shows that we are using the BANDWIDTH attribute. Though in DASH, we use an attribute that is an average. So that should probably be fixed. But AVERAGE-BANDWIDTH is optional according to the spec, so we would need to fallback to BANDWIDTH if it's missing. Would you like to send a quick PR for this?

And finally - is there a way to change the bandwidth estimator function by means of configuration (without messing with the sources)? For example, I would like to try and make the EWMAs longer, or even change to SMAs.

Absolutely. You can implement an interface (shaka.extern.AbrManager) and configure a factory (player.configure('abrFactory', () => new MyAbrManager())) to plug in your own ABR algorithm, completely from scratch. You can monitor the video element's buffered attribute or our recently-added player.getBufferFullness() method and react to that instead of estimating bandwidth. (That method was just added and isn't in a release build yet. It's in the latest source, though.)

Here's a shell of how a buffer-based AbrManager could be structured (untested), to help you get started:

class MyAbrManager {
  constructor(getBufferFullness) {
    this.variants_ = [];
    this.enabled _ = false;
    this.switch_ = null;
    this.config_ = null;
    this.getBufferFullness_ = getBufferFullness;
  }

  init(switchCallback) {
    this.switch_ = switchCallback;
  }

  setVariants(variants) {
    this.variants_ = variants;
  }

  configure(config) {
    // Store and use the ABR config if you're going to make it configurable.
    // You can reuse bandwidthDowngradeTarget and bandwidthUpgradeTarget,
    // or you can add new config parameters to Shaka Player that are more fit to purpose.
    this.config_ = config;
  }

  enable() {
    this.enabled_ = true;
  }

  disable() {
    this.enabled_ = false;
  }

  stop() {
    this.enabled_ = false;
    this.switch_ = null;
    this.variants_ = [];
    this.config_ = null;
    this.getBufferFullness_ = null;
  }

  playbackRateChanged(rate) {
    // A no-op if you're monitoring the buffer fullness.
    // A higher playback rate will naturally result in depleting the buffer more quickly.
  }

  getBandwidthEstimate() {
    // If we're not estimating bandwidth...
    return 0;
  }

  chooseVariant() {
    // Choose an initial variant and return it.
    // But don't do this, do something better than this.  :-)
    return variants[0];
  }

  segmentDownloaded(deltaTimeMs, numBytes) {
    // This can be a no-op if you're monitoring buffer fullness,
    // or you can use this method as a prompt to compute the
    // buffer fullness  and react to changes.  I recommend the latter.

    const fullness = this.getBufferFullness_();
    if (fullness is increasing) {
      this.switch_(higher quality variant);
    } else if (fullness is decreasing) {
      this.switch_(lower quality variant);
    }
  }
}

player.configure('abrFactory', () => {
  return new MyAbrFactory(() => player.getBufferFullness());
});

If you can make this work well, we would be more than happy to have a buffer-based ABR system added alongside shaka.abr.SimpleAbrManager as an alternative. (Though perhaps SimpleAbrManager should get renamed at that point to better describe the difference.) If you want to work on this, we would be happy to collaborate or provide guidance.

in my case, some of the video segments are not cached in the CDN, which results in longer download times - but they are not many and thus it wouldn't really affect the playback, except for the ABR, which panics and switches to a low bitrate stream. I would thus like to smoothen out the bandwidth estimation, to take more segments into account.

In addition to monitoring buffer fullness, you could also consider making some small changes to SimpleAbrManager and then subclassing it or configuring it to tweak some parameters in the bandwidth estimation. For example, making bandwidthEstimator_ replaceable or configurable. Currently, the EwmaBandwidthEstimator class contains these hard-coded constants you might want to change:

https://github.com/google/shaka-player/blob/9709086e9898300324d742fb39deb9afcb386813/lib/abr/ewma_bandwidth_estimator.js#L22-L34

[avelad]

I think latency should be taken into account. I agree with @joeyparrish about complexity, but in many countries maybe the network is not the best, and the latency can be high and this penalizes the reproduction.

@joeyparrish, are you open to making EwmaBandwidthEstimator configurable? If yes... -> The problem is that the EwmaBandwidthEstimator is created in the constructor when we don't have the configuration yet, do you have any ideas for this? If you give me any ideas I can do a PR for this.

[third774]

Glad to see great discussion on this! Is there currently a way to write a custom ABR manager that takes latency into account? I'm not sure how to capture latency since segmentDownloaded only accepts deltaTimeMs and numBytes. I may be overlooking something simple, but it would seem to depend on the NetworkingEngine providing this information?

[joeyparrish]

ABR plugins have been possible in Shaka for a long time. Any implementation of the shaka.extern.AbrManager interface can be plugged in via the configure interface:

player.configure('player.abrFactory', () => new MyCustomAbrManager(myCustomConstructorArgs));

The networking system doesn't capture time-to-first-byte latency, but we would be happy to review PRs to improve this so that network latency info is available to ABR plugins. We would also be happy to review PRs to use that latency info in our default ABR manager.

[shaka-bot]

@kchudoba Does this answer all your questions? Can we close the issue?

[shaka-bot]

Closing due to inactivity. If this is still an issue for you or if you have further questions, you can ask us to reopen or have the bot reopen it by including @shaka-bot reopen in a comment.

[joeyparrish]

@avelad, I overlooked a question you asked here. Sorry! Yes, we can make EwmaBandwidthEstimator configurable. Feel free to create a PR that expands the ABR section of the player config and feed it to the estimator.

[avelad]

@joeyparrish I have sent the PR. I think the label of this issue should be changed, so that it better reflects reality.