Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use Media Capabilities Decoding Info API #1391

Closed
beaufortfrancois opened this issue Apr 4, 2018 · 13 comments
Closed

Use Media Capabilities Decoding Info API #1391

beaufortfrancois opened this issue Apr 4, 2018 · 13 comments
Assignees
Labels
status: archived Archived and locked; will not be updated type: enhancement New feature or request
Milestone

Comments

@beaufortfrancois
Copy link
Contributor

The Media Capabilities API is an API for querying the user agent about decoding and encoding capabilities of the device, providing better information about the display properties, and real-time feedback about playback so that adaptive streaming can better alter the quality of content based on user perceived quality. The intent of this API is to give more control for content delivery while providing a better user experience.

I think this would be great to integrate it into Shaka Player as it is going to be enabled by default in Chrome 66 and is already used by YouTube player.

Example:

const mediaConfig = {
  type: 'media-source', // or 'file'
  audio: {
    contentType: 'audio/webm; codecs=opus',
    channels: '2', // audio channels used by the track
    bitrate: 132266, // number of bits used to encode a second of audio
    samplerate: 48000 // number of samples of audio carried per second
  },
  video: {
    contentType: 'video/webm; codecs="vp09.00.10.08"',
    width: 1920,
    height: 1080,
    bitrate: 2646242, // number of bits used to encode a second of video
    framerate: '25' // number of frames used in one second
  }
};

navigator.mediaCapabilities.decodingInfo(mediaConfig).then(result => {
  log('This configuration is' +
      (result.supported ? '' : ' NOT') + ' supported,' +
      (result.smooth ? '' : ' NOT') + ' smooth and' +
      (result.powerEfficient ? '' : ' NOT') + ' power efficient.');
});

Resources:

@joeyparrish joeyparrish added type: enhancement New feature or request and removed needs triage labels Apr 5, 2018
@joeyparrish joeyparrish added this to the v2.5 milestone Apr 5, 2018
@joeyparrish
Copy link
Member

Scheduled for v2.5. Thanks!

@joeyparrish
Copy link
Member

Rough plan: Use this API (if present) to drive an additional filtering pass on the media. If any variants are smooth, we avoid all non-smooth variants.

We may want to consider using the powerEfficient field as well if the data-saver flag is on. See #855.

@beaufortfrancois
Copy link
Contributor Author

That's great to hear!
FYI @chcunningham @mounirlamouri

@mounirlamouri
Copy link

Out of curiosity, how would you use powerEfficient with data-saver?

@joeyparrish
Copy link
Member

To clarify, I was thinking data-saver being enabled would be a proxy to tell us it's a mobile device.

But now I realize that there's a battery API as well. MDN says it's obsolete, but it's not clear why, and they don't recommend an alternative.

Mounir, I see your name on the battery status spec. Can you clarify its status? Is there another way we should determine that powerEfficient is an important bit?

@joeyparrish joeyparrish self-assigned this May 31, 2018
@joeyparrish
Copy link
Member

I tried out MediaCapabilities in Shaka Player this week, but I think my plan to use it for an additional filtering pass needs to change.

For one, MC is async, and our existing filtering is synchronous. This means we may want to refactor a bit before introducing it.

For another, this is different from our filtering based on isTypeSupported, codec compatibility (to avoid unsupported codec switches in MSE), and DRM restrictions (to remove things we can't decrypt). Those things are "hard" restrictions that we should never be flexible with. If a codec is not supported at all, we fail if we try. But just because something is smooth doesn't mean we should never play it. smooth is a "soft" signal, indicating what we should do.

We could still use it as a filter, in some circumstances. For example, if playback were smooth for 480p & 720p, but not 1080p, we could easily exclude 1080p without much harm.

But what if stereo audio were smooth and 5.1 surround were not? We don't know if that would ever happen in practice, but from the API itself, this is a possible outcome. Now what if the 5.1 surround track were the original English language version of a TV show, but the French dub were only available in stereo? I've seen this sort of thing on some region 2 DVDs. In those cases, we might be filtering out an entire language based on smoothness.

So the only safe way to use this as a filter is to first split up the tracks by things like language (and probably other metadata), then filter in each group. If nothing in a group is smooth, you can't get rid of the whole group. This sounds like a giant pain.

The alternative is to use MC to give ABR hints, rather than filter tracks. For example, we could annotate the variants with smoothness, and we could disallow AbrManager from choosing anything that is not smooth. (If MC is missing, we would assume everything to be smooth.)

I'm leaning toward using MC for ABR hints, as this is a much smaller refactor.

@chcunningham
Copy link

But what if stereo audio were smooth and 5.1 surround were not? We don't know if that would ever happen in practice, but from the API itself, this is a possible outcome.

Adding a little color here - in chrome, especially for unencrypted, the audio configuration will not affect smoothness.

There is a hypothetical scenario where UAs in very constrained devices (e.g. TVs, cast devices, etc) could theoretically be impacted by the audio configuration. For instance, say audioConfigA means can choose internal pipeline A which leverages their hw chip to smoothly decode the corresponding video configuration. But audioConfigB might only be available in a SW pipeline and they don't support running both pipelines in parallel -> less smooth video. But this is pretty contrived and the real world may never hit this problem.

You might just use the API in whatever way is most valuable to you and keep some metrics to indicate whether you ever encounter this hypothetical audio filtering problem. I think chances are low, maybe zero, but I'd be excited to learn otherwise.

@joeyparrish joeyparrish removed their assignment Nov 21, 2018
@joeyparrish joeyparrish modified the milestones: v2.5, Backlog Jan 2, 2019
@joeyparrish joeyparrish modified the milestones: Backlog, Backlog 2 Jan 28, 2020
shaka-bot pushed a commit that referenced this issue Jun 5, 2020
This is a change in preparation for the adoption of the
MediaCapabilities API.

Related to #1391

Change-Id: If0988aedafe15aa3be794fb977fd8e04e91ca10e
@michellezhuogg michellezhuogg self-assigned this Feb 4, 2021
shaka-bot pushed a commit that referenced this issue Feb 6, 2021
Player.filterManifest() does two things:
1. Filter the manifest with Drm, codecs and currentVariant's
   compatibility in StreamUtil.
2. Filter the manifest with PlayerConfiguration.restrictions.

In applyConfig_(), only the restrictions config affects filtering
the manifests. We don't need to call StreamUtils.filterManifest().

2. Make the filterManifest function async.

Issue #1391

Change-Id: I2fba9cabb3f3a3e89f5d990c4eeb19cc74924de9
shaka-bot pushed a commit that referenced this issue Feb 9, 2021
Breaking the StreamUtils.filterManifest() function into four steps.

Issue #1391

Change-Id: I3badc3662bad12375e38f563af1a78437c23a737
shaka-bot pushed a commit that referenced this issue Feb 10, 2021
StreamUtils.filterManifest() filters the manifest with four steps:
1. Filter out the variants unsupported by the platform
2. Filter out the variants unsupported by DRM
3. Filter out the variants uncompatible with the current variant
4. Filter out unsupported text streams.

We re-filter the manifest after the streams are chosen, and we only
need to filter out the variants uncompatible with the current
variant. Skip step 1,2,4 in StreamUtils.filterManifest(), to
avoid duplicate calls to MediaCapabilities.decodingInfo().

Issue #1391

Change-Id: Ia792f94d4ac3d93267741ef63db0872f30189fda
@joeyparrish joeyparrish modified the milestones: Backlog, v3.2 Feb 18, 2021
@joeyparrish
Copy link
Member

As @michellezhuogg has been working on MediaCapabilities for a couple weeks now, I decided to test the platforms in our lab to see which ones have MediaCapabilities.

I found that IE, Tizen 3, and Xbox One do not have the API. IE is not a big deal, since we are dropping support for IE anyway. But the other two will need a polyfill to fall back on.

What's sadder, though, is that Chromecast has MediaCapabilities, but I found that it is broken for encrypted content. Though both PlayReady and Widevine are present on the device, MediaCapabilities reports that neither is. So we will need to deny-list the implementation on Chromecast based on the firmware version number until MediaCapabilities in the Cast firmware gets fixed.

All other platforms that supported MediaCapabilities seem to provide correct results for encrypted content, meaning that the EME-related parts of the API are actually supported. This means our polyfill will not have to deal with platforms that only have MediaCapabilities for clear content.

@avelad
Copy link
Collaborator

avelad commented Feb 19, 2021

Regarding Tizen, the newer versions are based on a Chromium version that should support this. To avoid banning these versions if you need to do tests I can do them to see if it really works.

Links of interest:
https://caniuse.com/mdn-api_mediacapabilities_decodinginfo
image

https://developer.samsung.com/smarttv/develop/specifications/web-engine-specifications.html
image

@joeyparrish
Copy link
Member

joeyparrish commented Feb 19, 2021

@avelad, we probably won't have to ban anything by version after all. Normally, we will detect the API itself. If navigator.mediaCapabilities is null, we will have to install a polyfill to provide the necessary info.

The results I'm seeing on Chromecast via our test framework are not reproducible in the JS debugger on my Chromecast at home. Not sure what's going on there yet, but it probably means we won't have any Cast-specific bans in the long term.

@joeyparrish
Copy link
Member

Just for a quick update, the weird results I get on Chromecast are limited to our test environment. It seems to be triggered by the iframe used to run our tests on the device. I'm talking to the Cast team about this to get it fixed.

shaka-bot pushed a commit that referenced this issue Mar 12, 2021
Issue #1391

Change-Id: I35edc4ce5b84777687f0e1a91c4be1710a85c07a
shaka-bot pushed a commit that referenced this issue Mar 12, 2021
In StreamUtils, use MediaCapabilities.decodingInfo() instead of
MediaSource.isTypeSupported() to check if the stream is supported.

MediaCapabilities.decodingInfo() takes an
MediaDecodingConfiguration object as input, and returns a Promise
with a MediaCapabilitiesInfo object. The returned object tells us
whether decoding the media is supported, smooth, and
powerefficient.

Steps:
1. Create a MediaDecodingConfiguration object for each variant as
   the input.
2. Query the decodingInfo API with the config.
3. Get the 'supported' info from the decodingInfo result, to know
   whether the variant is supported.

Issue #1391

Change-Id: I8fc2d3ec6a9868f38269d550d35f45c298faae98
shaka-bot pushed a commit that referenced this issue Mar 13, 2021
If navigator.mediaCapabilties is not supported by the browser, use
MediaSource.isTypeSupported to check if the stream is supported.

Issue #1391

Change-Id: Iee0c7e339add2a07028ed14a03df14e4d36a203e
shaka-bot pushed a commit that referenced this issue Mar 18, 2021
Adding MediaCapabilities polyfill to uncompiled.js, so that the
polyfill can be used in unit tests.

Fixed the tests failure in Tizen, Safari and other browsers.

Issue #1391

Change-Id: Ia9e220d00ac48bf4a024fb780e4f20e9e111ecc7
shaka-bot pushed a commit that referenced this issue Mar 18, 2021
When querying MediaCapabilities.decodingInfo(), if the video width
or height is not present, set 1 as default. Linux Firefox would
claim the variant is unsupported via decodingInfo() if the values
are falsy.

Issue #1391

Change-Id: Idc27e6869a1276585a0ca35bd380e1b7ad1db242
shaka-bot pushed a commit that referenced this issue Mar 23, 2021
Previously, we fill in the variants' drmInfos with the drm
configurations of |clearKeys|, |servers| and |advanced| during
initializing the DrmEngine, before we query the media keys.

Now we need to call |MediaCapabilities.decodingInfo()| to get the
mediaKeySystemAccess of the variants after the DrmEngine is
created and configured, and the drm values are filled in for the
variants, and before |DrmEngine.queryMediaKey_()|.

The steps would be:
0. StreamUtils.setDecodingInfo() should not be called before
   DrmEngine is initialized and configured.
1. Create and configure DrmEngine.
2. Fill in drm values for the variants with configurations.
3. Call StreamUtils.setDecodingInfo() to get the decodingInfo of
   the variants.
4. Use the decodingInfo results to set up the Drm mediaKeys in
   DrmEngine.
5. When StreamUtils.filterManifest() is called, we can reuse the
   decodingInfo results instead of calling decodingInfo again.

Previously we call filterManifest() when parsing the manifest, to
filter out the unsupported streams.
Now decodingInfo can tell us if a variant is supported and its
MediaKeys at once. When initializing the DrmEngine, we get the
decodingInfo results of the variants, and only use the supported
streams to set up the MediaKeys. After that, we filter the manifest
right after DrmEngine is initialized.
Thus, we can skip filterManifest() in manifest parsers.

Using decodingInfo to get media keys will be in the next CL.

Issue #1391

Change-Id: Ieb401a1e4dfbcc958f7a14fa96df546237b0f446
shaka-bot pushed a commit that referenced this issue Mar 23, 2021
If we have a multiplexd content with audio and video, we combine
the audio and video codecs in the manifest parser.
For example, a multiplexed stream would be with:
mimeType="video/mp4", codecs="avc1.64001e,mp4a.40.2".

When sending the request to MediaCapabilities.decodingInfo(), we
need to have the config for both audio and video to get the result
as "supported".
The video config would be:
contentType='video/mp4, codecs="avc1.64001e"'.
The audio config would be:
contentType='audio/mp4, codecs="mp4a.40.2"'.

Issue #1391

Change-Id: I74a8580c07228e9600dc40c611ebd5d34f8cd7ee
shaka-bot pushed a commit that referenced this issue Mar 23, 2021
The developers should be able to set the config to use
MediaCapabilities.
If set to true, we'll use mediaCapabilities.decodingInfo() to
get the variants' decodingInfo.
If set to false, we'll fallback to use
MediaSource.isTypeSupported() and
navigator.requestMediaKeyAccess().

Issue #1391

Change-Id: I1acbfb12780291e998d56a91156ae2c47a87c7ae
shaka-bot pushed a commit that referenced this issue Mar 30, 2021
In StreamUtils, add a MediaCapabilitiesKeySystemConfiguration
for each key system for the encrypted variant passed to the
decodingInfo API, to get the mediaKeySystemAccess as a part of
the decodingInfo results.

We create a list of mediaDecodingConfigurations for each variant,
and call decodingInfo() with each mediaDecodingConfiguration to
get the mediaKeySystemAccess.

Eventually, we'll use the mediaKeySystemAccess from the
decodingInfo results to replace the call of
navigator.requestMediaKeySystemAccess() in DrmEngine. That will be
in the next CL.

Also, adding MediaCapabilties polyfill with mediaKeysSystemAccess.

Issue #1391

Change-Id: Ied4a27dd8a1ade43209bcf07f21f0c9b31c2693c
shaka-bot pushed a commit that referenced this issue Mar 30, 2021
If we have a multiplexd stream with audio and video, the video
codecs are a combined string containing both audio and video codecs.
Before sending the request to decodingInfo(), we should check which
codec is audio and which one is video.

This is a follow-up of commit 966a756.

Issue #1391

Change-Id: Ic9b1c5972a99f63a715c74ae068b85f43efac447
shaka-bot pushed a commit that referenced this issue Apr 7, 2021
In DrmEngine, previously we created MediaKeySystemConfiguration for the
variants, and called navigator.requestMediaKeySystemAccess() to get the
mediaKeySystemAccess, and set up MediaKeys.

Now we can use the mediaKeySystemAccess from the decodingInfo results of
the variants directly to set up MediaKeys.

Issue #1391

Change-Id: Id93a5e2fed7f6827317ae11644967185fc0cffbd
shaka-bot pushed a commit that referenced this issue Apr 7, 2021
Issue #1391

Change-Id: Ifdf4477d3798ddacf77036f7d85d9cb29438becc
shaka-bot pushed a commit that referenced this issue Apr 20, 2021
When we use decodingInfo() with the drmInfo of the variants to get media
keys, the decodingInfo result can tell us whether the variant's DRM is
supported by the platform. Thus, filterManifestByDrm_() is no longer
needed with MediaCapabilities enabled.

Issue #1391
Closes #3334

Change-Id: I34fbb3e11877f02cae1d435e9dbf274ce0e691dc
shaka-bot pushed a commit that referenced this issue Apr 20, 2021
Issue #1391

Change-Id: I9f990b4e77950b436a0f007999f55bb160d1673b
shaka-bot pushed a commit that referenced this issue Apr 22, 2021
Prepare and use configsByKeySystem only with MediaCapabilities disabled.

Issue #1391

Change-Id: I54ca4a61265bc4b5cd98e869d4be037824c7631e
joeyparrish pushed a commit that referenced this issue Apr 22, 2021
Issue #1391

Change-Id: I9f990b4e77950b436a0f007999f55bb160d1673b
shaka-bot pushed a commit that referenced this issue Apr 29, 2021
Since we got some unexpected decodingInfo results from MediaCapabilities
on Chromecast, we should fall back to MediaSource.isTypeSupported() on
Chromecast.

Issue #1391

Change-Id: I4947ec78624e98c7039df64ad691e14ece2588e0
shaka-bot pushed a commit that referenced this issue May 11, 2021
We'll allow users to configure the decoding attributes they prefer when
choosing codecs through the configuration. The attributes include
'smooth', 'powerEfficient' and 'bandwidth'.

For example, if the user configures the field as ['smooth',
'powerEfficient'], we'll filter the variants and keep the smooth ones
first, and if we have more than one available variants, we'll filter and
keep the power efficient variants.
After that, we choose the codecs with lowest bandwidth if we have
multiple codecs available.

Issue #1391

Change-Id: Ief3f6d8ff98fabff5ec99bb0365cdc6a36d2ab2d
ismena pushed a commit that referenced this issue May 17, 2021
We'll allow users to configure the decoding attributes they prefer when
choosing codecs through the configuration. The attributes include
'smooth', 'powerEfficient' and 'bandwidth'.

For example, if the user configures the field as ['smooth',
'powerEfficient'], we'll filter the variants and keep the smooth ones
first, and if we have more than one available variants, we'll filter and
keep the power efficient variants.
After that, we choose the codecs with lowest bandwidth if we have
multiple codecs available.

Issue #1391

Change-Id: Ief3f6d8ff98fabff5ec99bb0365cdc6a36d2ab2d
shaka-bot pushed a commit that referenced this issue May 21, 2021
Remove "useMediaCapabilities" configuration in the code base.

Issue #1391

Change-Id: I5fc74e31666840828e9dc04a5733db3eaed0d21a
@avelad
Copy link
Collaborator

avelad commented Jun 23, 2021

@michellezhuogg Is the integration of this functionality already complete?

@joeyparrish
Copy link
Member

This is complete, too.

@shaka-bot shaka-bot added the status: archived Archived and locked; will not be updated label Aug 23, 2021
@shaka-project shaka-project locked and limited conversation to collaborators Aug 23, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
status: archived Archived and locked; will not be updated type: enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

7 participants