Add support for DisplayCapabilities.

[jernoble]

See the modified version of the spec here: https://jernoble.github.io/media-capabilities/

This pull request adds support for interrogation-style queries for display capabilities through a new method on MediaCapabilities: displayInfo(), similar to the existing decodingInfo() and encodingInfo() methods. This new displayInfo() method takes a DisplayConfiguration dictionary, containing three values: colorGamut (as per CSS and the existing spec), transferFunction, and colorDepth.

Justification:

What we found in our own HDR playback investigations was that screen luminance was not sufficient in determining whether HDR playback was supported on a given device. For example, some devices may support one kind of transfer function but not the other, so specifying the transfer function used by the variant in question is necessary. Also, some devices may support 10-bit color but not 12-bit, making that query may be necessary as well.

What we have also found is that, between color gamut, transfer function, and color depth, we no longer require accurate information about the device luminance in order to answer whether HDR playback is supported. Eliminating device device luminance has the benefit of reducing the fingerprinting surface by reducing the entropy to four additional bits: two for the 'pq' and 'hlg' transferFunction parameters and two for the values of 10-bit and 12-bit colorDepth.

The new displayInfo() method returns a Promise<DisplayCapabilitiesInfo>, which will resolve to a dictionary with a single supported parameter. This dictionary can be extended in the future to add additional parameters (such as powerEfficient) if that becomes necessary.

---

Preview | Diff

[jernoble]

Note that this PR addresses issue #10 (hopefully).

[mwatson2]

@jernoble This looks good to me, except that there is some ambiguity around whether the color gamut field is referring to the capability to interpret pixel data encoded in that gamut or the capability to render (most of the) colors within that gamut.

The ambiguity I think comes mostly from the phrase "describes the approximate range of color supported", which suggests the second interpretation.

I think we are most interested in the first interpretation, although we would obviously like it to be the case that devices can in fact render most of the colors within any gamut that they advertise support for.

Perhaps the definition should be "describes the color gamut encodings supported" and we should add somewhere that "devices should not advertise support for P3 or rec.2020 color gamut encoding unless they support accurate rendering of at least some colors not included within sRGB."

[jernoble]

For colorGamut language, I tried to hew (hue?) as closely as possible to the language in the CSS Media Queries spec, which is definitely more about displaying colors than interpreting them. That said, any device which advertises support for a given color gamut must by definition also be able to understand and interpret that color gamut.

Is there a use case for having both pieces of information available? E.g., is it useful to know that the device understands Rec.2020 but can only display sRGB? Would you really want to deliver Rec.2020 content to such a display and have it clip unrenderable colors? If so, I think we could convey that information in the DisplayCapabilitiesInfo with something like (strawmanning) { supported: true, color: 'clipped' }.

[jernoble]

@mwatson2 I tried to address some of your feedback in a second commit. PTAL.

[mwatson2]

The changes looks good, although I would say "accurately render" since someone might argue that aggressive clipping is still "rendering".

I may be worrying about something that is too obvious to need clarification, but I think it should be clear that if a UA advertises a given gamut then it must be able to interpret data encoded that way.

As written, theoretically, a device could advertise P3 here, because that is the gamut closest to its physical rendering capability, but may only be able to interpret data encoded as 2020. This of course would be odd.

Separately, it's easy to imagine a scenario where a device can interpret 2020 but only render sRGB (new P3-capable laptop connected to sRGB external display). I can only think of two reasons why a site might deliver 2020 content to this device:
(1) the site doesn't have a version encoded in a different gamut
(2) the site wants the user to see the new colors when they drag the window to the integrated P3-capable display

(1) is arguably the site's problem for only providing a version of the content supported on a small % of devices
(2) would be better addressed by capability change notifications which a site could respond to by restarting playback with the appropriate alternative version

So based on the above, I don't think UAs should advertise gamuts they can interpret but not render usefully.

[chcunningham]

Thanks for working on this! Give me a few days to review - I'm looping in some folks who know color better than I do and some player authors to assess how it serves their use case.

[mounirlamouri]

Thanks for tackling this @jernoble ! :)

Like @chcunningham I do not have the cycles just now to do a full review but after a quick look at the change, I wonder why this had to be Navigator instead of Screen? I realise that today, it's fairly similar but assuming websites can detect screen changes or even gather connected screens, this could make the API more powerful.

Similarly, I wonder if we could simply use the color gamut we have in CSS and simply expose it on the Screen object if we want to make it easier to access. We could probably change the definition in CSS unless those are two different use cases?

[jernoble]

Having this API on WorkerNavigator allows access from workers; moving it from Navigator to Screen would imply removing it from WorkerNavigator as well. I guess I'm fine with that, but I suspect that player authors who have been pushing to move as much as possible into Workers may have reservations.

Having a simple accessor for colorGamut on Screen is attractive, but odd. colorGamut is not a single property, but an overlapping set; defining its value in IDL could be weird (is it an enum? a list of enums? etc.). Also, it would close off the possibility of distinguishing displays which support, e.g., a Rec.2020 color gamut but can only support P3 when used with an HLG transfer function. So even if there were a general-purpose colorGamut accessor on Screen we would probably want to have it in the DisplayConfiguration dictionary as well.

[chcunningham]

Haven't forgotten about this! I'm meeting with some color folks tomorrow. Will post feedback this week.

[chcunningham]

Thanks for waiting! Apologies for the formatting - hopefully its at least orderly :)

1]
Transfer function enum looks good to me.

2]
Color depth is already on Screen and has different semantics (~describes the sum of all channels) than proposed for displayInfo(). What motivates adding it here? Also, what motivates the semantics change?

3]
Similar for colorGamut: already lives in CSS via media queries. What motivates adding it here?

4]
Maybe the answer to 2 and 3 is partly that it's nice to group display properties into a common API. On that note, let me chime in on the Screen discussion.

Screen comes with the benefit that it already solves the "which screen" in multiple display scenarios (and I suspect it will be augmented for upcoming cases like VR etc) and it's already the home for colorDepth.

Also, it would close off the possibility of distinguishing displays which support, e.g., a Rec.2020 color gamut but can only support P3 when used with an HLG transfer function.

This is a compelling reason to group the properties with eotf, but, I've so far not been able to confirm this dependency exists in practice. It definitely doesn't exist in Chrome. There was some worry that it might be true for hw backed devices (e.g. Chromecast), but so far I've not found that to be the case. Do we have examples of this in the wild?

Having a simple accessor for colorGamut on Screen is attractive, but odd. colorGamut is not a single property, but an overlapping set; defining its value in IDL could be weird (is it an enum? a list of enums? etc.).

This may not matter in practice. As proposed, each set is a superset of the other (srgb < p3 < rec2020). If we put it on Screen it might be fair to return the highest supported.

5]

Any device which advertises support for a given color gamut must by definition also be able to understand and interpret that color gamut.

This resonates with me (admittedly not a color expert).

Is there a use case for having both pieces of information available? E.g., is it useful to know that the device understands Rec.2020 but can only display sRGB? Would you really want to deliver Rec.2020 content to such a display and have it clip unrenderable colors?

Hubbe@chromium (color expert) pointed out that we may (eventually) desire a way to signal that we don't really have a capable display, but we can do better than clipping (via some clever scaling such that HDR appears as SDR). He was ambivalent about how/if the UA should expose this to web authors. I'm not really making a proposal - just opening up discussion on the idea.

[chcunningham]

Forgot this point...

Having this API on WorkerNavigator allows access from workers; moving it from Navigator to Screen would imply removing it from WorkerNavigator as well. I guess I'm fine with that, but I suspect that player authors who have been pushing to move as much as possible into Workers may have reservations.

I generally agree - but maybe there are workarounds. Even if we put this in displayInfo(), we need a means of deciding which screen to describe in multi screen circumstances. Perhaps this is the screen of the script that spawned the worker (going up to root if need be). Technically this approach could also work for Screen - not sure what ergonomics folks (e.g. TAG) would prefer (e.g. maybe this is a lesser evil vs duplicating things between APIs). For SharedWorkers, I think neither Screen nor displayInfo has a way to say "which screen" to describe (no notion of a root window).

[mwatson2]

@chcunningham The reason for having the various color volume / transfer function attributes here, separately from CSS Screen, is that a devices video rendering capabilities may be different from its graphics (CCS) rendering capabilities. This is the case for all web devices that can render HDR video today (laptops, ChromeCast, CastTV) since there is no HDR in CSS and yet that render HDR video.

It will remain the case for the foreseeable future for devices based on TV SoCs as (AFAIK) noone has very advanced plans for HDR or wide color graphics support on those devices. But of course they all support HDR / wide color video.

[mounirlamouri]

@chcunningham The reason for having the various color volume / transfer function attributes here, separately from CSS Screen, is that a devices video rendering capabilities may be different from its graphics (CCS) rendering capabilities. This is the case for all web devices that can render HDR video today (laptops, ChromeCast, CastTV) since there is no HDR in CSS and yet that render HDR video.

It will remain the case for the foreseeable future for devices based on TV SoCs as (AFAIK) noone has very advanced plans for HDR or wide color graphics support on those devices. But of course they all support HDR / wide color video.

This is coming back to #25 which we can try to resolve but if we want, we should probably also handle the size as today one would most likely use window or screen to find out the output size.

This said, even if moving outside of the Screen object disambiguate graphics vs video rendering, it still makes things impossible to judge for cases where these are the same. If you have a laptop connected to a screen, a browser would have to pick either the worse or the lowest setup. Having the API living on Screen would follow the current expectations that exist on the Web Platform.

[jernoble]

@mounir, I am becoming convinced that the Display Capabilities part of this spec should move to Screen. Should the entire MediaCapabilities object move to screen? And if so, should we define a more limited MediaCapabilities object exposed to Worker?

[mwatson2]

Moving it to Screen is fine for me. I am just concerned that we have the ability to distinguish graphics from video capabilities, so my point is that the existing fields on Screen are not a replacement for this Display Capabilities aspect.

Even on devices that have HDR graphics capabilities, they may not be exposed to the web except for video. This is the current situation on laptops today and I'm rather losing hope that CSS will ever add HDR support so we may be living with it for some time to come, even on laptops.

[mounirlamouri]

@jernoble the decoding info part of Media Capabilities doesn't depend on the active screen so it can be in navigator and exposed to workers. Chrome didn't do this (yet) but it's already done in Firefox for example.

@mwatson2 I understand the concern and I'm fine with having this API be specific for video but instead of having this new API for video rendering only and the rest of the platform being generally oriented for graphics rendering, I wonder if we should try to see how this would fit into a larger design.

[chcunningham]

instead of having this new API for video rendering only and the rest of the platform being generally oriented for graphics rendering, I wonder if we should try to see how this would fit into a larger design.

In that vein, here's a strawman

• window.screen.videoColorDepth
• window.screen.videoColorGamut

Or perhaps a dedicated video subsection is better? Are there other properties (e.g. width && height) that need this treatment?

• window.screen.video.colorDepth
• window.screen.video.colorGamut

Still thinking about EOTF

[chcunningham]

@mwatson2 @jernoble curious if you have thoughts on the strawman from that last message. I'd like to run it (or some evolution) by window.Screen folks if you don't object.

[jernoble]

@chcunningham, Display color gamut capabilities may depend on transfer function and color depth. (I.e., "you only get 'rec2020' with '8 bit' color depth, and 'p3' with '10 bit'.) So static accessors aren't capable of conveying that nuance. (In the previous i.e., you'd get 'rec2020' and '10' for the screen.colorGamut and screen.colorDepth properties, but if you tried to display 10-bit 'rec2020' content, it'd look wrong.)

Also, if your screen.videoColorDepth is '10', does that mean the screen supports 1-bit color? Does it support 9-bit color? (Probably not.)

IOW, the point of a query-based API is the same here for display capabilities as it is for decoder capabilities; static properties can't cover all the edge cases.

[jernoble]

That said, I don’t object to adding a property to Screen exposing the current gamut and color depth.

[jernoble]

Somehow this request got closed; I'd like it re-opened if possible.