Precise color transformation matrices

[willmtemple]

What's the problem?

This issue has been reported before, but I'm proposing a specific solution for discussion. This is a problem that involves three pieces:

• Hyprland receives CTMs over hyprland-ctm-control-v1 (present version 2), where the definition of the CTM is underspecified and overconstrained to not allow negative elements, and the elements are quantized to 1/256 due to the wl_fixed encoding only having 8 bits of fractional precision. A proper chromatic adaptation needs a defined color space, negative off-diagonal elements, and finer precision.
• Hyprland applies the CTM at the CRTC through KMS. The KMS CTM applies to values already encoded by the output transfer function (typically sRGB for SDR, PQ for HDR). This is wrong for a few reasons and particularly bad for multi-monitor SDR/HDR side-by-side setups.
• Hyprsunset constructs the CTM for sRGB using an approximation, so it's very wrong on outputs that use a very different transfer function (PQ), and chromatically inaccurate on connectors that use different primaries.

You notice this when using cm = hdr/hdredid the moment you enable a CTM. An HDR monitor using BT.2020 + PQ goes way more red than an SDR monitor, and it also gets a slight magenta cast. That is the effect of the sRGB matrix being applied to the encoded PQ values in the primaries of BT.2020.

What's your proposed solution?

My proposal is for "Precise CTM" and it comes in three parts.

1. Extend hyprland-ctm-control-v1 to version 3 with a new request set_ctm_for_output_precise. This request would define the CTM to be computed in linear light, in the primaries of BT.709, with 1e-6 precision (wire format defined to be the matrix elements multiplied by 1e6 and transmitted as an integer).
2. In Hyprland, since it already has a compositing stage in that exact description, applying the precise CTM is as easy as multiplying pixels by the matrix in-shader in the general case. In specific cases, where the hardware and drivers can support it, we can offload to the DRM Color Pipeline API. We may be able to continue using KMS approximates in cases where a v2 matrix is supplied, or where the CM shader is disabled.
3. In hyprsunset, rewrite the CTM construction using colorimetric linear algebra that adapts in Bradford space. This will yield a matrix that is a real chromatic adaptation in linear BT.709, and the compositor can cheaply conjugate it in other primaries using mathematically well-defined matrix operations.

I have a working prototype of this pipeline including the hyprland-protocols revision, hyprsunset matrix rewrite, and in-shader CTM for Hyprland. I will post them as PRs for review and reference. I haven't dug into the DRM Color Pipeline stuff much yet (notes in attachments below).

Caveats

An in-shader CTM cannot apply to direct scanout (but when offloaded to a DRM Color Pipeline, it can work), so a precise CTM applied in-shader either blocks DS or vice versa, depending on whether the user prefers chromaticity to be consistent vs. having direct scanout when the night light is active. There could be render settings and windowrules for customization of how CTMs interact with DS and fullscreen.

Attachments

Image of the defect. Left monitor is BT.2020 + PQ running with cm = hdredid, right is sRGB (camera white balance is targeting the window on the right monitor).

DRM Color pipeline API notes (click to expand)

These ideas are based on what's on my machine specifically, need to do research about what exists on AMD/intel. Probed on NVIDIA 610.43.03, kernel 7.1.5. I'm probably wrong/confused about some of the following.

• This path is an offload where the general case is that the CTM can always be applied in linear light in the CM shader when the hardware does not support what we need. Falling back to in-shader CTM is always correct for indirect scanout. The CRTC path may not be needed at all, but might still be a useful fallback for v2 imprecise matrices.
• The cursor plane on my machine exposes no color pipeline at all. Hardware cursor CTM could still be accomplished by pre-tinting the cursor bitmaps.
• The way color pipeline API works is that the driver tells you which pipelines you support, and you bypass stages you don't want and program the ones you do want.
• My hardware has a prebuilt pipeline called "Lite" that only contains the front segment of a full color pipeline. This is the only prebuilt pipeline offered for the overlay plane (fine, since Lite has everything we need). It has a floating point mode also. Lite contains everything we need for precise CTM.
• PQ framebuffers: conjugate the precise CTM in the primaries of the framebuffer, use the Lite pipeline or equivalent, unbypass PQ decode, apply CTM, bypass the rest.
• SDR framebuffers: same, but decode goes through a LUT programmed for the output eotf (usually sRGB piecewise, NVIDIA only exposes fixed function hardware for PQ and a LUT is good enough for 8-bit power laws)
• fp16 scRGB framebuffers: FP Lite, no decode needed. the CTM stage will also carry a basis conversion from BT.709 to blend space if needed. Pipeline has a MULT stage that allows white scaling to be fixed function if needed.
• On my hardware, a plane pipeline ends in blend space where the planes are composited (linear light, but this still confuses me a bit...), and the driver will encode to the wire on its own, so we don't have to program re-encoding back to PQ. A cheap test on any hardware to see if blend space has different primaries from the input is to select a pipeline with every op bypassed and see what happens to the picture.
• You cannot configure a custom pipeline (that's what shaders are for). These pipelines are fixed-function. The general approach is to walk the list of pipelines the driver supports and pick one that has all the functions we need (bypass the rest that we don't need).

I acknowledge that:

• I have searched all (open and closed) discussions and issues, checked the wiki, and my feature is not there.

• I have read the Code of Conduct and agree to follow it.

• I wrote this request with my own hands, my own keyboard, and in my own words.

[vaxerski]

This sounds reasonable to me. It's a shame we can't use the CTM from KMS, but it's probably not going to change much since we already do the final offloaded pass in hl anyways pretty much unconditionally nowadays