Claiming a hardware decoder with play()

[JohnRiv]

Migrated from w3c/webmediaporting#30:

When play() is called on a media element that needs to use a hardware decoder, at which step in which algorithm does the UA attempt to “claim” that hardware decoder?

What happens if the hardware decoder is being used by

1. something outside the HTML5 UA, or
2. another HTML5 media element (in the same page) that is playing, or
3. another HTML5 media element (in the same page) that is stopped, or
4. another HTML5 media element in a page that’s loaded in another tab in the browser?

If the HTML5 UA is unable to “claim” the hardware decoder, how would the method call fail and what state would the media element be left in?

[fghilardi]

Chrome's Media Stack (c/o @sandersdan from w3c/webmediaporting#30 (comment) and w3c/webmediaporting#30 (comment))

The web does not have explicit resource management for media. This is a consistent problem and we employ a number of tricks and optimizations to work around it, which seems to be a qualified success.

Chrome's primary intervention is to discard codecs for idle players, then resume playback by an internal seek-to-previous-keyframe when required. (We call this Suspend/Resume, it is unrelated to network suspend.)

We may pause playing video to force it to become idle. A variety of heuristics may be used to select players for this.

I would very much like to see explicit mechanisms added. For WebCodecs I expect that codec instances will be fallible, allowing us to take their resources at any time. A more sophisticated approach is likely required for new use cases, such as background transcoding. (Eg. some way to inform apps when resources become available.)

We do not, in general, have information about platform (hardware OR software) codec availability/constraints. We merely try to obtain them, and this can fail. Failure cases are typically decode errors, and they may be recoverable via load().

Loading to HAVE_METADATA requires acquisition of a codec to decode the poster frame. play() from a lower readyState will force acquisition of a codec.

We change our workarounds somewhat routinely, but there may be a core set of principles we can document. (Approximately 'always make forward progress eventually, fake events if you have to'.)

Comparatively recent changes like MediaSession (media controls in notifications) and the play() promise have helped, in that new players are more flexible when the browser asserts control over video elements. In the past a majority of player implementations would call play() in a loop if we didn't enter a playing state, but that's not a problem we have with modern players.

[sandersdan]

To more precisely answer the questions above:

at which step in which algorithm does the UA attempt to “claim” that hardware decoder?

Chrome decodes the frame at the current time before reaching HAVE_CURRENT_DATA.

Chrome historically required a decoder to reach HAVE_METADATA, but recently has started to avoid this when a poster image is provided.

What happens if the hardware decoder is being used by

something outside the HTML5 UA, or

Decode error.

another HTML5 media element (in the same page) that is playing, or

Chrome may try to pause other elements in the same page. In certain race conditions, you can get a decode error.

another HTML5 media element (in the same page) that is stopped, or

This shouldn't be a conflict in Chrome. In certain race conditions, you can get a decode error.

another HTML5 media element in a page that’s loaded in another tab in the browser?

On Android, Chrome suspends all media playback in background tabs.

In certain race conditions, you can get a decode error.

If the HTML5 UA is unable to “claim” the hardware decoder, how would the method call fail and what state would the media element be left in?

Decode error.

There's a legitimate reason web developers don't like debugging decode errors.