Should the VideoDisplayConfiguration width & height be in CSS or device pixels

[gregwhitworth]

Note: I am opening this issue specifically to track the resolution of this discussion so that we don't have to dig through PR feedback to find it.

This is a quick recap of an issue on PR #124 - referincing the definition of width & height with regards to utilizing CSS or device pixels. Here is the current spec text

The width member represents the width of the display in physical pixels, not CSS pixels.

The height member represents the height of the display in physical pixels, not CSS pixels.

@chcunningham provided the following feedback:

This seems not quite right. The important thing we're trying to capture here is that these are the dimensions of the video plane (ie not the graphics plane). I think we still want to measure those dimensions in CSS pixels rather than physical device pixels. @mounirlamouri @hober - double check me

@mwatson2 provided this response:

The main use-case for this is to decide whether it is worth fetching 4K or not. Don't we want physical pixels for that ?

@chcunningham response

I follow you, but I wonder if we should expose it in that way. My worry is whether if its semantically consistent for high density displays.

As I understand the goal, we want to signal when a device's video plane is a different resolution from its graphics plane. For devices where the planes match (everywhere except TVs / dongles), I was thinking window.screen.height should = window.screen.video.height (same for width). Would you agree? If we use device pixels for video the two will not match for displays with device-pixel-ratio > 1. I worry this introduces inconsistent layout semantics.

For example, take a pixel-doubled 4K retina dispaly. The bottom right corner in terms of physical pixels is at 4096,2304, but in terms of CSS px you really only half that area to layout your page. If you try to size a video using CSS from 0,0 to 4096,2304 thinking you're filling the screen, you will actually be viewing only the top left quadrant.

Now take a TV with a device-pixel-ratio=1 and with video plane dimensions matching its native resolution of 4096x2304. Say its graphics plane is only half that. Again, try using html to size a video from 0,0 to 4096,2304 - this time (if I understand how these planes work) I expect the video is actually filling the screen?

Do I have that last part right? I'm not sure I understand how folks do layout for a device with different planes. If I have understood it correctly, it would mean mean screen.video has different semantics in different settings. If we instead use CSS px (as a unit, not referring to a plane) the values would represent a layout area for both settings.

[jernoble]

If the main use case is, as @mwatson2 says, to determine which resolution variant to choose based on device pixels, then this working group is the wrong place to specify that. CSS has made a choice about how they represent the device pixel ratio, and I don’t want to be in the business of undermining their specification choices. If content providers feel that the device pixel ratio is not precise enough, they should file an issue against CSS.

So, is there another use case here? Is there an example of a device which cannot display video at all resolutions it advertises decode support for?

[vi-dot-cpp]

@chcunningham
As I understand the goal, we want to signal when a device's video plane is a different resolution from its graphics plane.

Agreed. In addition, we also want to expose the device's physical resolution so that, for instance, users with pixel-doubled 4K displays are able to receive 4K video content.

Using the pixel-doubled 4K display example, window.screen.height would return 1080 (CSS pixels). Sites like Netflix can derive the physical resolution using window.devicePixelRatio. However, I am under the impression that (window.screen.height)*(window.devicePixelRatio) does not always equate to the device's physical pixel due to window.devicePixelRatio's definition varying from platform to platform. Is my understanding correct?

[lhalliwell]

The TV platforms being discussed have two physical pixel resolutions (one for graphics, one for video). On Chromecast today, window.screen.height is 720, device pixel ratio is 1.5 (1080 physical pixels height for graphics), and 4K video may be supported (so existing CSS facilities don't provide a way to query this, we currently provide a Chromecast-specific API to detect whether 4K is supported or not).

[mwatson2]

A CSS pixel is defined as a unit of physical length, so the size of the graphics and video planes in CSS pixels will always be the same, even where they support different resolutions.

The CSS definition of CSS pixels was originally based on a comfortable pixel size for laptop / desktop viewing distances. As far as I know, there has been no separate definition of CSS pixel for TVs so I imagine that browser implementors in the TV space have made their own choices based on reverse engineering the appearance of web pages, defaulting to physical pixels or some other approach.

@jernoble If it is necessary to follow CSS's approach exactly, then what is needed is a separate devicePixelRatio for video and graphics. However, providing the width and height in device pixels exactly equivalent. It's certainly the case that there will be devices that can decode 4K but are connected to a HD screen (e.g. 4K-decode-capable laptop outputting to HD monitor).

[jernoble]

I think I understand, but let me restate my understanding so y'all can tell me why I'm wrong:

For devices that use a single plane for both graphics and video, the existing CSS facilities are sufficient. But for devices with a separate plane for non-video rendered web content and another plane for video content, the existing CSS facilities may give a false-negative signal about support for particular video resolutions.

Does Is my understanding correct, or have I missed something?

[jpiesing]

The TV platforms being discussed have two physical pixel resolutions (one for graphics, one for video). On Chromecast today, window.screen.height is 720, device pixel ratio is 1.5 (1080 physical pixels height for graphics), and 4K video may be supported (so existing CSS facilities don't provide a way to query this, we currently provide a Chromecast-specific API to detect whether 4K is supported or not).

It's not just Chromecast. There are TV sets with an HD panel, HD graphics but which can decode and down-sample UHD. The ability for what is otherwise an HD TVs to decode and downsample UHD video may even become a legal requirement in some countries.

[jernoble]

@jpiesing, Hang on, that seems to be the inverse of the use case listed earlier. @mwatson2's use case was to discover the physical pixels available to the display in order to choose a variant which most closely matched that physical resolution. What I understand your use case to be is the ability of the UA (the TV) to down-sample the decoded video to from the original resolution to the resolution of the display. If a UA couldn't do that downscaling, I'd argue it shouldn't advertise support for decoding it in the first place, or that's the consensus we came to in #118 earlier with a similar issue (see @chcunningham's comment in that thread).

[gregwfreedman]

For devices that use a single plane for both graphics and video, the existing CSS facilities are sufficient.

i thought i could simply calculate device pixels by multiplying screen.height (and width) by devicePixelRatio. however, i discovered this was not a reliable way to determine device pixels since zoom level affects devicePixelRatio and screen.height (and width) differently on the major browsers.

[jernoble]

@gregwfreedman I'm not sure what you mean by "zoom level", but regardless that doesn't seem to be a problem we should solve in this spec. It's either something to be taken on by CSS or it's an implementation issue in UAs.

[jpiesing]

@jpiesing, Hang on, that seems to be the inverse of the use case listed earlier. @mwatson2's use case was to discover the physical pixels available to the display in order to choose a variant which most closely matched that physical resolution. What I understand your use case to be is the ability of the UA (the TV) to down-sample the decoded video to from the original resolution to the resolution of the display. If a UA couldn't do that downscaling, I'd argue it shouldn't advertise support for decoding it in the first place, or that's the consensus we came to in #118 earlier with a similar issue (see @chcunningham's comment in that thread).

@jernoble I agree it's a different use-case but it's another example of video pixels and graphics pixels being different.

[jernoble]

@jpiesing It is another example, but one that can be addressed through an entirely different mechanism. @gregwfreedman and @mwatson2 specifically want to know the device's physical pixel dimensions so as to decide which variant to send the client; but knowing the device's physical pixel dimensions does not address your use case (whether the device is capable of downsampling UHD media to a HD display resolution). And contrariwise, the solution to your use case (knowing that a device is capable of downsampling UHD to HD) does nothing to help achieve theirs.

I also wonder how a <video> element ends up rendering to a "graphics plane" in the first place. Is it automatic? Does such a <video> element participate in CSS? Does it participate in layout at all?

[chcunningham]

I also wonder how a

I think I have some clarity on this from Chrome Cast folks (can't say how well this describes smartTVs). Here's how it works on the HD cast device

• they have a 1080p graphics buffer. this means they always rasterize graphics to 1080p irrespective of the resolution they've negotiated with the display
• if cast negotiates a 4K HDMI output resolution, the 1080p raster gets upscaled by the output hardware
• the <video> element is still positioned and sized like any other element, but it isn't part of the 1080p raster. It's somehow sent to output hardware at a higher resolution.

@lhalliwell - sound right?

[gregwhitworth]

This was discussed in today's telecon, minutes are here: https://www.w3.org/2019/10/02-mediawg-minutes.html

Resolution was:

Remove the screen width & height from VideoDisplayConfiguration PR until action is done and we can make an informed direction

[lhalliwell]

I also wonder how a element ends up rendering to a "graphics plane" in the first place. Is it automatic? Does such a element participate in CSS? Does it participate in layout at all?

It participates in CSS and layout, although there are some restrictions (see notes below). chcunningham's description above is accurate enough.

• the <video> element is still positioned and sized like any other element, but it isn't part of the 1080p raster. It's somehow sent to output hardware at a higher resolution.

@lhalliwell - sound right?

That's right. The video plane lies 'beneath' the graphics plane, so in graphics buffer,

This whole setup is obviously a bit yucky and not fully featured, i.e. there are things expressible on the web that are impossible to display this way (examples: rotating video at arbitrary angles, applying transparency to video with other content visible beneath, using a video as a WebGL texture).

[chcunningham]

Thanks @lhalliwell. @jpiesing - can you say if this is more or less the same behavior on smartTVs?

[vi-dot-cpp]

Thanks for chiming in on this issue here and on the call. While we discuss, I have removed width and height from PR #124 via this commit.

[jpiesing]

That's right. The video plane lies 'beneath' the graphics plane, so in graphics buffer, element needs to have a transparent "hole" drawn. The video decoder is set up to decode video into a quad that lines up with this hole.

This whole setup is obviously a bit yucky and not fully featured, i.e. there are things expressible on the web that are impossible to display this way (examples: rotating video at arbitrary angles, applying transparency to video with other content visible beneath, using a video as a WebGL texture).

Thanks @lhalliwell. @jpiesing - can you say if this is more or less the same behavior on smartTVs?

@vi-dot-cpp More or less yes.

[chcunningham]

During the call we discussed using device pixel ratio. I'm still in favor of that. Here's a link to a discussion of the tradeoffs in this comment on a separate issue.

[chrisn]

I'll close this, as the remaining work is in w3c/csswg-drafts#6891. Please re-open if there's anything I've missed that's not covered by that CSS issue.