Add HDR compositing of HTMLCanvasElement proposal #15

ccameron-chromium · 2021-03-10T11:49:29Z

Add a new proposal for HDR compositing of HTMLCanvasElement.

simontWork

Added some comments on clamping, low power devices, definitions, OOTF.

simontWork · 2021-03-16T11:23:12Z

hdr_html_canvas_element.md

+There exist the following constraints.
+
+* The exact maximum luminance of the output display is not known and not knowable.
+  * The [CSS Media Queries Level 5 Specification](https://www.w3.org/TR/mediaqueries-5/#valdef-media-dynamic-range-high) allows the application to query the ``'dynamic-range'``. The resulting values are ``'standard'`` and ``'high'``.


For low powered devices, e.g. television embedded browser used for rendering VOD applications, it may be preferable to request the video plane current mode and request the graphics plane sets itself to the same mode using CSS to request the correct set of objects, rather than transforming the rendered objects.

It will be a tradeoff between fingerprinting and usability on low-powered devices.

This and pal's comments are both suggesting a need for a PQ or HLG color space (which would probably pair with 10-bit-per-channel formats).

simontWork · 2021-03-16T11:29:17Z

hdr_html_canvas_element.md

+(Sometimes this range is normalized to [0, 12], but we do not use that convention).
+This step is independent of the output display.
+
+The second step is to apply the normalized OOTF to transform scene light into normalized display light.


The OOTF is designed to be applied in the monitor or TV. The value of the OOTF depends upon the screen's maximum luminance (which can be changed by the end user) and the ambient lighting.

The signal output from the browser should be scene-referred and should not have the OOTF applied.

On OSes that accept an HLG signal (or linearized HLG signal), we will want to do exactly that -- the browser will apply no OOTF. The buck is then passed to the OS, which may pass it to the monitor or TV.

But, not all OSes accept HLG signals, so then the buck stops with the browser. In that case, if the browser is to allow HLG content, then the browser must be the one do whatever transcoding is needed to get the intended HLG content into whatever format the OS accepts.

simontWork · 2021-03-16T11:34:43Z

hdr_html_canvas_element.md

+  * Remaining free parameters in the PQ mapping are chosen to make math easier.
+* If the application can overcome the fingerprinting limitations (e.g, by just asking the user), any desired behavior can be accomplished, using appropriate math.
+
+## Definitions


Definitions should match those used in the ITU as:

ITU is the United Nations standards defining organisation covering international video exchange and has defined terms for high dynamic range video

Moving away from ITU definitions will increase the likelihood of confusion as the terms will be used differently in this document compared to the video standard it references.

ITU standards and definition automatically become law in some UN member states meaning that the definitions can't easily be changed

In that case we should link to the definitions that the ITU provides -- please provide links where the ITU defines these terms and I'll include them.

I haven't found any place where ITU defines what "scene light" is supposed to mean (is it in nits, is it normalized, is it only the measurable light or can it have alterations done, etc, and the usage of the term in BT2100 is flexible).

simontWork · 2021-03-16T11:45:52Z

hdr_html_canvas_element.md

+
+For completeness, there does exist one more HDR mode, the no-HDR HDR mode (which is the current default behavior).
+
+In this mode, scene light is clamped to the range [0, 1], and the SDR OOTF is applied.


This comment is valid for SDR and HDR video and still image modes which use nominal-range signals.

Signals use nominal-range (in 10 bits: 64 - 940) to allow temporary excursions outside the video signal range (e.g. filter overshoots or fast exposure change) without causing signal clipping. Signal clipping is considered bad as when transformed to the frequency domain it has very high frequency components. This is difficult to encode and can cause further ringing artefacts if other video filters are applied (e.g. scaling in a display).

Could we suggest that video and image elements which use nominal-range are clamped in accordance with EBU R.103 https://tech.ebu.ch/publications/r103

hdr_html_canvas_element.md

simontWork · 2021-03-16T11:54:15Z

hdr_html_canvas_element.md

+
+  enum HTMLCanvasHDRCompositingMode {
+    'disabled',
+    'extended', // Call this "relative-luminance"?


'extended-srgb'? Then people would know what's being extended or are we allowing this to be used with Rec2020 too?

I think extended means in general allowing component values greater than 100% and less than 0%, to cope with out of gamut colors. So therefore it is not specific to sRGB.

Correct, the "extended" refers to "extending the SDR range", and can be applied to any color space (srgb, linear-srgb, display-p3, linear-rec2020, etc).

hdr_html_canvas_element.md

chrisn · 2021-03-10T18:16:17Z

hdr_html_canvas_element.md

+* To draw PQ encoded HDR images as they are intended, displaying the specified nits.
+  * E.g, displaying PQ content.
+  * E.g, working in physical luminance.
+* To draw HLG encoded HDR images as the are intended, mapping the HLG signal space to the full display device luminance range.


typo: the are -> they are

Reworded most of the section.

chrisn · 2021-03-10T18:20:37Z

hdr_html_canvas_element.md

+* For input PQ images drawn to canvases, we have to map nits to color values.
+* For outputting canvases in this compositing mode, we have to map color values to nits.
+
+We choose to assign a the same mapping for both of these, so that simply drawing a PQ image to a canvas that is composited in this mode will behave as expected.


typo: a the -> the

Reworded most of the section.

chrisn · 2021-03-10T18:21:24Z

hdr_html_canvas_element.md

+Such parameterization allows for the user to create an unexpected mismatch, which would be undesirable.
+
+The mapping we decide is that the color ``color(srgb 1 1 1)`` is to map to 100 nits.
+The motivation is that it's reasonable and that it the makes math easier for anyone who desires a different value (and many applications will want many other values).


typo: the makes -> makes the

chrisn · 2021-03-10T18:25:31Z

hdr_html_canvas_element.md

+This step is independent of the output display.
+
+The second step is to apply the OOTF to transform scene light into display light.
+This function is a gamma ramp, parameterized by the maximum luminance of the display, with a domain and range are [0, 1].


typo: with a -> where the?

Changed slightly

chrisn · 2021-03-16T11:27:49Z

hdr_html_canvas_element.md

+
+## Proposal summary / TLDR
+
+This summary will use terminology from the below definitions section.


suggest: definitions section below

hdr_html_canvas_element.md

kenrussell

Reviewed this mainly from the WebGL standpoint; it looks good to me overall. A few relatively minor comments throughout.

kenrussell · 2021-03-16T21:09:26Z

hdr_html_canvas_element.md

+
+    // This green will match the color in MyGreenUI.
+    var gl = canvas.getContext('webgl2');
+    gl.configureSwapChain({ format: gl.RGBA16F, colorSpace: 'srgb-linear' });


The exact API here is still in discussion. It's likely to be:

gl.drawingBufferStorage(gl.RGBA16F, canvas.width, canvas.height); gl.colorSpace = 'srgb-linear';

See KhronosGroup/WebGL#3222 .

kenrussell · 2021-03-16T21:09:55Z

hdr_html_canvas_element.md

+    // Configure the swap chain to be floating-point. Leave the color space as
+    // the default of 'srgb'. This will leave all colors in [0, 1] unchanged,
+    // but allow for specifying colors outside of the range of [0, 1].
+    gl.configureSwapChain({ format: gl.RGBA16F });


Needs updating per above comment about exact form of the WebGL API.

hdr_html_canvas_element.md

kenrussell · 2021-03-16T21:17:25Z

hdr_html_canvas_element.md

+
+```idl
+  partial interface HTMLCanvasElement {
+    void configureHDR(HTMLCanvasCompositingMode mode, optional HTMLCanvasHDRMetadata metadata);


Should this be specified for OffscreenCanvas as well? Presumably that would affect both ImageBitmap objects created via transferToImageBitmap, and if the OffscreenCanvas was controlling an HTMLCanvasElement, then would also affect the canvas, similarly to how resizing an OffscreenCanvas eventually causes the HTMLCanvasElement to resize.

Suppose we have a canvas that has had an image drawn into it (SDR, PQ, or HLG) and nothing else. The goal that I have in mind is for the result of transferToImageBitmap to have the same contents as an ImageBitmap created from the original image would. Because of how everything is defined in "device independent light" (in the new upload), this ends up being the case -- no compositing mode information needed(!).

The only place where this could fall apart is if the ImageBitmap can be assigned to an tag. In that case, the resulting would not look the same as the canvas that it came from. Is that an option?

Suppose we have a canvas that has had an image drawn into it (SDR, PQ, or HLG) and nothing else. The goal that I have in mind is for the result of transferToImageBitmap to have the same contents as an ImageBitmap created from the original image would. Because of how everything is defined in "device independent light" (in the new upload), this ends up being the case -- no compositing mode information needed(!).

The only place where this could fall apart is if the ImageBitmap can be assigned to an tag. In that case, the resulting would not look the same as the canvas that it came from. Is that an option?

(Sorry, which tag? <img>? I think Github's editor deleted the angle-bracketed-tag name you referenced.)

HTMLImageElement's src attribute can not be set to an ImageBitmap. ImageBitmap in general can only be used with the imperative JavaScript APIs like the canvas rendering contexts, OffscreenCanvas, etc.

kenrussell · 2021-03-16T21:18:31Z

hdr_html_canvas_element.md

+    void configureHDR(HTMLCanvasCompositingMode mode, optional HTMLCanvasHDRMetadata metadata);
+  }
+
+  enum HTMLCanvasHDRCompositingMode {


I think this needs [Exposed=(Window,Worker)].

Going to defer fixing Exposed for the moment (my understanding of it isn't deep enough)

hdr_html_canvas_element.md

simontWork · 2021-03-24T16:43:54Z

hdr_html_canvas_element.md

+For this OETF, the domain (scene light) is normalized to the interval [0, 1], with maximum brightness at 1 (which we will treat as matching our convention).
+
+Another example of an OETF is the PQ OETF, specified in Table 4 of the [BT.2100 specification](https://www.itu.int/dms_pubrec/itu-r/rec/bt/R-REC-BT.2100-2-201807-I!!PDF-E.pdf).
+For this OETF, the domain (scene light) is luminance values in nits (which does not not match our convention, and we will discuss how to match this in detail).


Is this correct? In table 4, the OETF maps relative scene linear light into the non-linear PQ signal value.

This is a good example of the definition problem that I was discussing. (Short answer: "you're right").

What is "relative scene linear light"? BT2100 uses it, but gives no definition. From its use, and with some detective work, we can piece together what might be a definition that is consistent with its usage.

In Table 4 that "Input signal to PQ opto-optical transfer function (OOTF)" is "Scene linear light. The OOTF maps relative scene linear light to display linear light." Well, which is it, "scene linear light" or "relative scene linear light"? They're different words, so presumably they have different meanings? What does "relative" mean here? Why is the word "linear" in there?

The next row in Table 4 contains the symbols "OOTF[E]", so I guess this means that E is the input signal to the OOTF which we established is either "scene linear light" or "relative scene linear light". (let's just go with "relative scene linear light"). And then E is defined as "E is the signal determined by scene light and scaled by camera exposure". No definition for "scene light" either, but at least that one is intuitive.

Table 5 has the same text, except defines E as "E is a signal for each colour component ... proportional to scene linear light normalized to the range [0:1]".

If we go back to Table 4, we can see that in the "Reference PQ OOTF" row, "E′ = G_{709}[E]" is defined only for values of E in the half-open interval [0, 1). So that supports the hypothesis that relative scene linear light is limited to the unit interval (perhaps closed or perhaps half-open).

Putting all of this together, we could say that "Relative scene linear light (denoted by E in this document), is a value in the interval [0, 1] that is linearly proportional to the physically measurable light in the scene being captured. The precise scaling factor used in mapping from physically measurable light to relative scene linear light depends on the particular capture setup, including such variables as camera exposure".

This is the sort of definition that I want to put at the top of the proposal -- I want it to be readable to people who have a basic understanding of compute graphics and the physics of light, but who are not steeped in broadcast terminology.

simontWork · 2021-03-24T16:46:27Z

hdr_html_canvas_element.md

+
+The HLG OOTF is parameterized by the maximum luminance of the display device. It maps a fixed range of scene light to cover the full range (up to maximum luminance) of the display device.
+
+The PQ OOTF can be thought of as the identity function, with scene light equaling display light.


Is this correct? BT.2100 defines the OOTF as approx. a gamma 1.2 applied on RGB.

The "reference PQ OOTF" defines OOTF[E], for E in (0.0003024, 1)

OOTF[E] = 100 * pow(1.099 * pow(59.5208 * E, 0.45) - 0.099, 2.4)

This is approximately 10,000*pow(E, 1.08). But yes, for the reference display the OOTF isn't the identity.

simontWork · 2021-03-24T17:11:34Z

hdr_html_canvas_element.md

+This step is independent of the output display.
+
+The second step is to apply the normalized OOTF to transform scene light into normalized display light.
+This function is a gamma ramp, parameterized by the maximum luminance of the display, with a domain and range that are [0, 1].


Gamma ramp applied on Luminance, not RGB.

chrisn · 2021-03-24T17:28:41Z

hdr_html_canvas_element.md

+          style='position:absolute; width: 95%;'></canvas>
+  <script>
+    var canvas = document.getElementById('MyCanvas');
+    canvas.configureHDR('pq-compatible'});


Typo: extra }

chrisn · 2021-03-24T17:28:50Z

hdr_html_canvas_element.md

+In this example, we make a mistake, and use ``'extended'`` in the above exmple.
+
+```javascript
+  canvas.configureHDR('extended'});


Typo: extra }

chrisn · 2021-03-24T17:29:13Z

hdr_html_canvas_element.md

+          style='position:absolute; width: 95%;'></canvas>
+  <script>
+    var canvas = document.getElementById('MyCanvas');
+    canvas.configureHDR('hybrid-log-gamma'});


Typo: extra }

chrisn · 2021-03-24T17:29:44Z

hdr_html_canvas_element.md

+  }
+```
+
+It is a convention in HLG to use a signal value of 0.75 as diffuse wwhite.


Typo: white

simontWork

Added a couple more comments.

ccameron-chromium · 2021-03-29T17:23:28Z

Merging this with a clarification about "display independent light"

ccameron-chromium force-pushed the add_hdr branch 5 times, most recently from 63aeeb9 to 6522398 Compare March 16, 2021 00:54

simontWork reviewed Mar 16, 2021

View reviewed changes

chrisn requested changes Mar 16, 2021

View reviewed changes

chrisn reviewed Mar 16, 2021

View reviewed changes

hdr_html_canvas_element.md Outdated Show resolved Hide resolved

kenrussell approved these changes Mar 16, 2021

View reviewed changes

simontWork reviewed Mar 24, 2021

View reviewed changes

chrisn requested changes Mar 24, 2021

View reviewed changes

simontWork reviewed Mar 25, 2021

View reviewed changes

ccameron-chromium force-pushed the add_hdr branch 4 times, most recently from 789cfba to 731e1d6 Compare March 29, 2021 17:08

Add HDR HTMLCanvasElement proposal

c4853c3

ccameron-chromium force-pushed the add_hdr branch from 731e1d6 to c4853c3 Compare March 29, 2021 17:22

ccameron-chromium merged commit 5e818b5 into w3c:master Mar 29, 2021


		For completeness, there does exist one more HDR mode, the no-HDR HDR mode (which is the current default behavior).

		In this mode, scene light is clamped to the range [0, 1], and the SDR OOTF is applied.


		## Proposal summary / TLDR

		This summary will use terminology from the below definitions section.


		The HLG OOTF is parameterized by the maximum luminance of the display device. It maps a fixed range of scene light to cover the full range (up to maximum luminance) of the display device.

		The PQ OOTF can be thought of as the identity function, with scene light equaling display light.

Add HDR compositing of HTMLCanvasElement proposal #15

Add HDR compositing of HTMLCanvasElement proposal #15

Conversation

ccameron-chromium commented Mar 10, 2021

simontWork left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kenrussell left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

simontWork left a comment

Choose a reason for hiding this comment

ccameron-chromium commented Mar 29, 2021