Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Black crush and color deviations with some ICC profiles #2815

Closed
haasn opened this issue Feb 13, 2016 · 233 comments
Closed

Black crush and color deviations with some ICC profiles #2815

haasn opened this issue Feb 13, 2016 · 233 comments

Comments

@haasn
Copy link
Member

haasn commented Feb 13, 2016

Continuation of the discussion in #534.

Some ICC profiles are experiencing black crush when used in mpv, in particular ones that use complex transfer curves rather than simple pure powers.

Proposed solutions including reworking the 3DLUT/lcms mechanism to pick a connection space more gracefully.

@UliZappe
Copy link

To get some terminology clear, there still is an intermediate space here - that space is just BT.709 in your example. There has to be some intermediate space by design, because the 3DLUT is a function from one space to another. The former space is the “intermediate space” that we generate the 3DLUT against.

OK, every color space conversion, by its very essence, requires two color spaces. But in my terminology, an intermediate color space would be a space that is neither the source nor the target space (nor, strictly speaking, the PCS that is used in all non “device link” conversions).

So clearly, if the video material is in BT.709, BT.709 is no intermediate space in the way I used the terminology. (It would be if you used it for all video material, BT.709 or not).

Are you implicitly suggesting we generate the 3DLUT against XYZ or Lab? Because both of those would be possible.

No I wasn’t, although, now you say it, this might at least work better than using BT.2020 for this purpose. But I still think the optimal solution would be a dedicated 3DLUT for a specific video color space – monitor profile combination. Just as a (well-constructed) device link profile is superior to two profiles connected via PCS.

It was working fine for me, which may be simply because I have a wide gamut profile that uses a pure power curve - both of which are ideal combinations for minimizing the error introduced by the change.

Probably – just like my test with Pro Photo RGB as the monitor profile, where everything worked fine.

@haasn
Copy link
Member Author

haasn commented Feb 13, 2016

OK, every color space conversion, by its very essence, requires two color spaces. But in my terminology, an intermediate color space would be a space that is neither the source nor the target space (nor, strictly speaking, the PCS that is used in all non “device link” conversions).

There are many different conversions going on here, each one of the spaces that is not at the very beginning or very end of the pipeline is an “intermediate” space in my books.

During typical video playback, the source video will undergo the following color spaces:

  1. space=Y'CbCr prim=BT.709 trc=BT.1886
  2. space=RGB prim=BT.709 trc=BT.18886
  3. space=RGB prim=BT.709 trc=linear
  4. space=RGB prim=BT.709 trc=sigmoid
  5. space=RGB prim=BT.709 trc=linear
  6. space=XYZ prim=BT.709 trc=linear
  7. space=XYZ prim=BT.2020 trc=linear
  8. space=RGB prim=BT.2020 trc=linear
  9. space=RGB prim=BT.2020 trc=gamma2.4
  10. space=RGB prim=monitor trc=monitor

I have highlighted steps 5 which is where CMS begins and step 9 which is what gets fed into the 3DLUT. We could use a different set of conversions here.

@UliZappe
Copy link

@4ad

So, in the end, I think we're looking at two problems here? One is that the intermediate BT.2020 color space conversion introduces color distorsion. We can see this by measuring the colors with various profiles.

The other problem is that either LittleCMS itself, or the way mpv uses LittleCMS can't handle complex profiles properly. We can see this by measuring black levels with simple and complex profiles.

Yep, it seems so.

black levels are good with simple profiles (but colors are off)

Which confirms that it’s two issues.

@4ad
Copy link

4ad commented Feb 13, 2016

Repeating the experiments with BT.709 gamma 1.961 (uploaded in the previous thread), which uses a simple power function for the transfer curve.

I am reading native values. This test also measures color.

Program black#1 black#7 black#8 black#15 white#15
QuickTime 1-1-1 8-8-8 9-9-9 17-17-17 238-238-238
mpv (recent) 1-1-0 7-8-7 8-9-8 17-17-16 239-238-240
mpv (before) 1-1-1 8-8-7 9-9-8 17-17-17 239-238-240
mpv (after) 1-1-1 8-8-7 9-9-8 17-17-17 239-238-240
Program bright red bright green bright blue medium red medium green medium blue
QuickTime 255-0-0 0-255-1 0-0-255 191-0-0 0-191-0 0-0-192
mpv (recent) 254-5-4 12-254-7 4-4-254 192-5-2 7-191-5 3-3-193
mpv (before) 255-1-1 0-255-1 0-1-255 192-0-0 0-191-0 0-1-193
mpv (after) 255-1-1 0-255-1 0-1-255 192-0-0 0-191-0 0-1-193

@UliZappe
Copy link

There are many different conversions going on here, each one of the spaces that is not at the very beginning or very end of the pipeline is an “intermediate” space in my books.

Ah, OK. But then I would differentiate between primaries conversions and TRC conversions. From my experience, the critical conversions are the primaries conversions.

So if we focus on these, your list becomes:

BT.709
BT.2020
Monitor

From this POV, there is one intermediate space, and this is BT.2020. This should remain BT.709, so that we get

BT.709
Monitor

i.e. no intermediate space in this sense.

I have highlighted steps 3 which is where CMS begins and step 7 which is what gets fed into the 3DLUT.

Uhm, the highlighted steps were 5 and 9?

@haasn
Copy link
Member Author

haasn commented Feb 13, 2016

Concerning the pipeline I just outlined, I have some two important observations to make:

The majority of these conversions are done using exact computations with 32 bit precision, so the margin of error is negligible both in theory and in practice.

The video gets rounded and clipped while in some spaces, due to various reasons:

1: inherently clipped to 8-10 bit precision (input video depth)
4: clipped to 16-bit precision (or more/less depending on texture depth) while upscaling, for performance reasons
9: “reduced” to ~8-bit precision while passing through the 3DLUT.
10: Clipped to 8-bit precision with dithering (monitor depth)

Of these operations, the most significant are the two 8-bit precision clips. That's most likely a big part of the reason why we are introducing so much distortion, because clipping to 8-bit BT.709 values and then again to 8-bit BT.2020 values reduces the overall bit depth below 8 bits because the “round numbers” are not aligned.

(There is another 8-bit clip in step 10 but this is less significant since it's dithered, so we retain much of the precision)

So if we want to minimize distortion, we have to make sure that space 9 and space 1 are as close as possible, to eliminate the effects of “double clipping”. This is the idea behind using a profile similar to the video source as the intermediate space. (Basically, it would be the same as in step 2)

But looking at this pipeline it seems obvious to me that we could eliminate distortion even more by going all the way back to the space used in step 1 - the input space - and generate the 3DLUT against this. That way the 3DLUT would essentially be a mapping from source Y'CbCr to monitor RGB.

(The internal calculations are performed with 64-bit floating point precision by LittleCMS, so the extra conversion inside LittleCMS should be essentially free)

@UliZappe
Copy link

But looking at this pipeline it seems obvious to me that we could eliminate distortion even more by going all the way back to the space used in step 1 - the input space - and generate the 3DLUT against this. That way the 3DLUT would essentially be a mapping from source Y'CbCr to monitor RGB.

Yes, exactly.

@haasn
Copy link
Member Author

haasn commented Feb 13, 2016

From my experience, the critical conversions are the primaries conversions.

Clearly, from the evidence in this discussion, the TRC selection matters just as much as the selection of the primaries - that's what's causing the black crush.

Uhm, the highlighted steps were 5 and 9?

Oops, I had duplicates in my numbering while editing and GitHub automatically changed them to count consistently.

@haasn
Copy link
Member Author

haasn commented Feb 13, 2016

How much does this patch improve things? (haasn@2ed5e5a)

diff --git a/video/out/opengl/lcms.c b/video/out/opengl/lcms.c
index c956127..db93e08 100644
--- a/video/out/opengl/lcms.c
+++ b/video/out/opengl/lcms.c
@@ -197,7 +197,7 @@ bool gl_lcms_get_lut3d(struct gl_lcms *p, struct lut3d **result_lut3d)
         // because we may change the parameter in the future or make it
         // customizable, same for the primaries.
         char *cache_info = talloc_asprintf(tmp,
-                "ver=1.1, intent=%d, size=%dx%dx%d, gamma=2.4, prim=bt2020\n",
+                "ver=1.1, intent=%d, size=%dx%dx%d, gamma=1.961, prim=bt709\n",
                 p->opts.intent, s_r, s_g, s_b);

         uint8_t hash[32];
@@ -242,9 +242,9 @@ bool gl_lcms_get_lut3d(struct gl_lcms *p, struct lut3d **result_lut3d)
     if (!profile)
         goto error_exit;

-    // We always generate the 3DLUT against BT.2020, and transform into this
+    // We always generate the 3DLUT against BT.709, and transform into this
     // space inside the shader if the source differs.
-    struct mp_csp_primaries csp = mp_get_csp_primaries(MP_CSP_PRIM_BT_2020);
+    struct mp_csp_primaries csp = mp_get_csp_primaries(MP_CSP_PRIM_BT_709);

     cmsCIExyY wp = {csp.white.x, csp.white.y, 1.0};
     cmsCIExyYTRIPLE prim = {
@@ -253,9 +253,9 @@ bool gl_lcms_get_lut3d(struct gl_lcms *p, struct lut3d **result_lut3d)
         .Blue  = {csp.blue.x,  csp.blue.y,  1.0},
     };

-    // 2.4 is arbitrarily used as a gamma compression factor for the 3DLUT,
+    // 1.961 is arbitrarily used as a gamma compression factor for the 3DLUT,
     // reducing artifacts due to rounding errors on wide gamut profiles
-    cmsToneCurve *tonecurve = cmsBuildGamma(cms, 2.4);
+    cmsToneCurve *tonecurve = cmsBuildGamma(cms, 1.961);
     cmsHPROFILE vid_profile = cmsCreateRGBProfileTHR(cms, &wp, &prim,
                         (cmsToneCurve*[3]){tonecurve, tonecurve, tonecurve});
     cmsFreeToneCurve(tonecurve);
diff --git a/video/out/opengl/video.c b/video/out/opengl/video.c
index d13cd30..3a0c2c4 100644
--- a/video/out/opengl/video.c
+++ b/video/out/opengl/video.c
@@ -1768,9 +1768,9 @@ static void pass_colormanage(struct gl_video *p, enum mp_csp_prim prim_src,
     enum mp_csp_prim prim_dst = p->opts.target_prim;

     if (p->use_lut_3d) {
-        // The 3DLUT is hard-coded against BT.2020's gamut during creation, and
+        // The 3DLUT is hard-coded against BT.709's gamut during creation, and
         // we never want to adjust its output (so treat it as linear)
-        prim_dst = MP_CSP_PRIM_BT_2020;
+        prim_dst = MP_CSP_PRIM_BT_709;
         trc_dst = MP_CSP_TRC_LINEAR;
     }

@@ -1800,10 +1800,10 @@ static void pass_colormanage(struct gl_video *p, enum mp_csp_prim prim_src,
     }
     if (p->use_lut_3d) {
         gl_sc_uniform_sampler(p->sc, "lut_3d", GL_TEXTURE_3D, TEXUNIT_3DLUT);
-        // For the 3DLUT we are arbitrarily using 2.4 as input gamma to reduce
+        // For the 3DLUT we are arbitrarily using 1.961 as input gamma to reduce
         // the severity of quantization errors.
         GLSL(color.rgb = clamp(color.rgb, 0.0, 1.0);)
-        GLSL(color.rgb = pow(color.rgb, vec3(1.0/2.4));)
+        GLSL(color.rgb = pow(color.rgb, vec3(1.0/1.961));)
         GLSL(color.rgb = texture3D(lut_3d, color.rgb).rgb;)
     }
     if (need_gamma)

It's not the full idea I proposed (using the input space), but for BT.709 videos this should be similar, perhaps.

@4ad
Copy link

4ad commented Feb 13, 2016

With haasn/mp@2ed5e5a.

First with BT.709 gamma 1.961. I am measuring native values.

Program black#1 black#7 black#8 black#15 white#15
QuickTime 1-1-1 8-8-8 9-9-9 17-17-17 238-238-238
mpv (recent) 1-1-0 7-8-7 8-9-8 17-17-16 239-238-240
mpv (2ed5e5a) 0-1-0 7-8-6 8-9-7 17-17-16 238-238-239
Program bright red bright green bright blue medium red medium green medium blue
QuickTime 255-0-0 0-255-1 0-0-255 191-0-0 0-191-0 0-0-192
mpv (recent) 254-5-4 12-254-7 4-4-254 192-5-2 7-191-5 3-3-193
mpv (2ed5e5a) 255-0-1 0-255-1 0-1-255 192-0-1 0-191-1 0-0-192

With iMac color profile.

Program black#1 black#7 black#8 black#15 white#15
QuickTime 1-1-1 7-7-7 8-8-8 16-16-16 241-241-241
mpv (recent) 0-0-0 3-4-3 4-5-4 15-16-14 241-240-242
mpv (2ed5e5a) 0-0-0 3-4-3 5-5-4 15-15-14 240-240-241
Program bright red bright green bright blue medium red medium green medium blue
QuickTime 235-51-36 117-251-76 0-26-246 183-37-25 90-195-57 0-17-191
mpv (recent) 235-50-34 115-251-75 1-26-247 183-36-24 89-194-56 2-17-192
mpv (2ed5e5a) 234-51-35 117-250-76 0-26-245 183-37-25 89-194-57 0-18-191

With BT.709 gamma 1.961, your patch improves color accuracy significantly.

With iMac color profile, your patch improves color accuracy, but less so, since the recent build wasn't too bad with this profile in the first place. Still, an improvement.

Black crush is not improved by your patch; of course, this is expected, since the problem is in LittleCMS, or with LittleCMS interaction which was not changed by your patch.

@haasn
Copy link
Member Author

haasn commented Feb 13, 2016

Wasn't black crush improved when using an sRGB input curve for the 3DLUT as well?

Another thing I would like to try is generating the 3DLUT against your monitor profile (i.e. extract a gamut and TRC from the profile and use that) - if that has similar performance, we could consider just doing that.

That would have the benefit of preserving the status quo of “one 3DLUT per ICC profile”.

We can also test generating it against XYZ or Lab, but I doubt this is going to improve black crush at all.

@4ad
Copy link

4ad commented Feb 13, 2016

Wasn't black crush improved when using an sRGB input curve for the 3DLUT as well?

In haasn/mp@114f69a? No, that made black crush slightly worse.

@4ad
Copy link

4ad commented Feb 13, 2016

@haasn

I rebased my DCI-P3 patch on haasn/mp@2ed5e5a, and now vo=opengl:target-prim=DCI-P3:target-trc=bt.1886 matches QuickTime within +/-1 if I set P3 gamma 1.961 as the display profile.

Of course it's not usable because this uses gamma=1.961 which is not right for my display (in other words P3 gamma 1.961 doesn't match my display, nor the "real" Display P3.icc).

My patch is at 4ad/mpv-colorfix@dbed5e0.

@haasn
Copy link
Member Author

haasn commented Feb 13, 2016

I rebased my DCI-P3 patch on haasn/mp@2ed5e5a, and now vo=opengl:target-prim=DCI-P3:target-trc=bt.1886 matches QuickTime within +/-1 if I set P3 gamma 1.961 as the display profile.

This does not surprise me, since in the absence of a 3dlut all operations are done with 16-32 bit precision, which is more than enough to meet the 8-bit requirement you gave it. (In fact, due to mpv's good dithering, the mpv result will be virtually exact overall)

@haasn
Copy link
Member Author

haasn commented Feb 13, 2016

A possibility that is somewhat far out there but possibly worth consideration in general is to use the 3DLUT for less and do more with exact shader computations - e.g. we could implement parts of the ICC spec in mpv directly, such as the bits concerning the transfer characteristics.

(Theoretically it would be possible to implement all of ICC in pure shader code and get a virtually perfect result that way, but the implementation overhead of such a feat would be monstrous - and the potential for errors due to bugs or negligence far greater.)

@4ad
Copy link

4ad commented Feb 13, 2016

@haasn

Hmm, I have an interesting result.

So, I added target-trc=DCI-P3. Now, I don't have access to the standard, so I implemented the complex parametric transfer function by looking inside the color profile. In other words, I read the transfer function from the profile, just like LittleCMS does!

And guess what, the result is the same, I get black crush.

In other words, with the Display P3.icc color profile, vo=opengl:icc-profile-auto and vo=opengl:target-prim=DCI-P3:target-trc=DCI-P3, produce the same output, which has the same problem.

My code is here 4ad/mpv-colorfix@5711136.

@haasn
Copy link
Member Author

haasn commented Feb 13, 2016

@4ad Maybe the correct result of a transformation through that profile results in crushed blacks even in theory?

It's not impossible that QuickTime is the one that gets this particular profile “wrong” - or that they have some extra logic that goes beyond simply transforming it in order to actively mitigate black crush.

@4ad
Copy link

4ad commented Feb 13, 2016

I don't think so. On 1-Black Clipping.mp4 I should see 17-25 flashing, but I only see 19-25 flashing. Plus, in real movies, mpv is lacking shadow detail (not noise, genuine detail) that is visible in QuickTime.

@haasn
Copy link
Member Author

haasn commented Feb 13, 2016

And here we are again at the important distinction between “looking good” and being a correct implementation of the spec.

Sure, I don't doubt that crushed blacks are bad. But if that's the way your profile was generated, and if Little CMS correctly implements the standard, then this is the result we would and should expect.

Since you have independently extracted the transfer curve from the profile, implemented, and gotten the same result - I'm inclined to believe the problem here is not LittleCMS but the profile.

@4ad
Copy link

4ad commented Feb 13, 2016

Well, the profile contains both a parametric tone response curve, and a 3*1024 point LUT called the "tone response code". Maybe they don't match, and LittleCMS uses the parametric curve, while QuickTime uses the LUT?

@haasn
Copy link
Member Author

haasn commented Feb 13, 2016

That would be a good thing to investigate. Inconsistencies like these often arise when different programs decide to use different versions of what should be redundant data, but isn't (e.g. due to bad profile generation).

Edit: Though I think the 3x1 “tone response code” might be for per-channel adjustments independent of the parametric tone response curve.

@4ad
Copy link

4ad commented Feb 13, 2016

Well that's why I want to get the real standard.

@UliZappe
Copy link

QuickTime Player, mpv, ColorSync and Little CMS

In the following I’m trying to more clearly locate the root of the deviations we see in the reproduction of the dark patches.

Video Players

For mpv, I’m using my “working version” in this test that has no BT.2020 intermediate space and of which we know that it reproduces the colors very close to QuickTime Player (which it again did in my following tests). So I’m concentrating on the dark patches. Specifically, I look at the 9th patch which we can assume is RGB 8-8-8 in BT.709 gamma 1.961. I use the DigitalColor Meter to measure the colors.

CMMs

To evaluate the ColorSync and Little CMS CMMs which QuickTime Player and mpv use, respectively, I calculate the conversion from 8-8-8 in BT.709 gamma 1.961 to the tested monitor space using the Calculator in the ColorSync Utility application included with OS X for ColorSync and the transicc command line utility included with Little CMS for Little CMS.

Monitor Profiles

I have tried to use a representative collection of monitor profiles. I will list native monitor RGB values, since we are interested in the (relative) differences between the color values obtained in the four different ways, not the absolute values. Using native monitor RGB values minimizes the required calculations and focuses on what we want to know. The absolute values in the following table do not matter, only the differences.

I have omitted profiles with simple tone response curves, as we know that these work well.

We have not much discussed yet that when it comes to complex transfer curves, there are still two different variants: matrix profiles with such curves, and LUT profiles.

Note that as soon as we deal with LUT monitor profiles, rendering intents become relevant (whereas for matrix monitor profiles, only the chromatic adaptation of the absolute colorimetric intent might make a difference). In the table, I use p = perceptual, r = relative colorimetric, a = absolute colorimetric.

Here’s the rationale for the monitor profiles I used:

ICCv2, Matrix, complex tone response curve

sRGB (well known ICCv2 standard profile)
eciRGBv2 ICCv2 (from the European Color Initiative; valid ICCv2 standard profile with an unusual tone response curve (L*)

ICCv4, Matrix, complex tone response curve

P3 (taken from OS X El Capitan, ICCv4 standard profile)
eciRGBv2 ICCv4 (valid ICCv4 and supposed to deliver identical results to the ICCv2 version above)

outside of ICCv2/ICCv4 spec, Matrix, complex tone response curve

iMac (tagged as ICCv2, using ICCv4 features, real life relevance as it’s default profile in recent iMacs)

ICCv2 LUT profile

spring-LED.LUT (individual profile created with Quato iColor Display)

ICCv4 LUT profile

LED ipLUT (individual profile created with X-Rite i1 Profiler)

Test Results for the dark patch with RGB 8-8-8 (BT.709 gamma 1.961)

Nr./RI Monitor Profile QuickTime Player mpv ColorSync Little CMS
1pr eciRGB v2 ICCv2 6-6-6 3-4-3 4-4-4 3-3-3
1a 4-4-6 3-3-3
2pr eciRGB v2 ICCv4 6-6-6 3-4-3 4-4-4 3-3-3
2a 4-4-6 3-3-3
3pr sRGB 8-8-8 5-5-4 6-6-6 4-4-4
3a 7-6-6 4-4-4
4pr P3 8-8-8 5-5-4 6-6-6 4-4-4
4a 7-6-6 4-4-3
5pr iMac 8-8-8 5-5-4 6-6-6 4-4-4
5a 7-6-6 4-4-3
6p spring-LED.LUT 16-16-16 1-1-0 15-16-14 12-12-12
6r 7-0-0 0-0-0
6a 4-0-0 0-0-0
7p LED ipLUT 2-3-4 10-11-0 0-0-1 9-10-11
7r 10-11-0 7-8-0
7a 10-11-0 7-8-0

Comments

  1. For all matrix profiles (nr. 1 – 5), the calculated results for both ColorSync and Little CMS are always darker than the actual colors in QuickTime Player and mpv. Maybe that’s because my assumption is wrong that the 16 dark patches in the video have RGB values from 0 – 15, or maybe my self-created BT.709 gamma 1.961 profile (which the calculated results use, in contrast to QuickTime Player and mpv) is somehow flawed. But we should not ruminate too much about that, because for our problem it’s only helpful if QuickTime Player and mpv display the dark tones a bit brighter than calculated.
  2. For all matrix profiles (nr. 1 – 5), mpv and Little CMS are always darker than QuickTime Player and ColorSync. So unfortunately, this seems simply to be a characteristic of Little CMS, and nothing that mpv can improve upon.
  3. Profiles 1 and 2 (an ICCv2 and an ICCv4 profile which are otherwise identical) sport exactly the same result. So Little CMS and thereby mpv has no issues with ICCv4 profiles.
  4. Little CMS has problems with the absolute colorimetric rendering intent. For instance, eciRGB (profiles 1 and 2) has a D50 white point. Displaying a D65 color source such as Bt.709 in an absolute manner means eciRGB must shift the colors to blue. ColorSync does this, but Little CMS does not. From a lengthy discussion with the creator of Little CMS I know that this is a conscious decision of his. I think he’s wrong, but he thinks Apple and myself are wrong. So we should not go into this, and I think we don’t have to, as the absolute color rendering intent should be irrelevant for watching videos.
  5. If we ignore the absolute rendering intent, profiles 3 – 5 produce exactly identical results. This means that the profiles are obviously very similar, but especially, it means that Little CMS and thereby mpv has no problems to deal with the out-of-spec profile nr. 5 correctly.
  6. The differences in behavior for the two LUT profiles (nr. 6 and 7) are huge and almost the opposite: in profile 6, mpv is completely off, in profile 7, QuickTime Player is completely off. Looking at the results of Little CMS, it seems that mpv uses the relative colorimetric rendering intent as the default, when the perceptual intent would provide much better results (and it wouldn’t at least hurt in any other of the 7 test cases, though this only refers to this one dark patch, of course). An alternative might be to use black point compensation for the relative colorimetric rendering intent, which in case of profile 6 changes 0-0-0 into 13-12-2. Of course, QuickTime Player does not offer any way for this kind of fine tuning. What completely amazes me is the huge difference between these two LUT profiles; it seems anything is possible here. Nr. 6 is a ICCv2 profile created with software from the (now defunct) Quato; nr. 7 is a profile created with software from X-Rite. Whether it’s ICCv2 vs. ICCv4 or the different profiling software which makes the difference is not clear from this test.

Summary

For matrix profiles with complex tone response curves, it simply seems that Little CMS does not handle dark tones too gracefully, and that there is little mpv can do. The only thing I can think of is to replace Little CMS with ColorSync for the Mac platform, which would be technically possible, but of course a huge platform specific deviation.

For LUT profiles, the possible differences in behavior are baffling. The options for fine tuning that mpv offers (rendering intent, black point compensation) become much more relevant here than I would have thought. Perceptual might be a better default rendering intent than relative colorimetric (but this would need further confirmation).

@UliZappe
Copy link

Regarding comment nr. 1 above: Assuming that, for whatever reason, it is the 8th (not the 9th) patch in the test movie that has RGB 8-8-8, the table would look as follows (different values for QuickTime Player and mpv):

Nr./RI Monitor Profile QuickTime Player mpv ColorSync Little CMS
1pr eciRGB v2 ICCv2 5-5-5 3-3-2 4-4-4 3-3-3
1a 4-4-6 3-3-3
2pr eciRGB v2 ICCv4 5-5-5 3-3-2 4-4-4 3-3-3
2a 4-4-6 3-3-3
3pr sRGB 7-7-7 4-4-3 6-6-6 4-4-4
3a 7-6-6 4-4-4
4pr P3 7-7-7 4-4-3 6-6-6 4-4-4
4a 7-6-6 4-4-3
5pr iMac 7-7-7 4-4-3 6-6-6 4-4-4
5a 7-6-6 4-4-3
6p spring-LED.LUT 15-15-15 0-0-0 15-16-14 12-12-12
6r 7-0-0 0-0-0
6a 4-0-0 0-0-0
7p LED ipLUT 0-1-0 9-10-0 0-0-1 9-10-11
7r 10-11-0 7-8-0
7a 10-11-0 7-8-0

@haasn
Copy link
Member Author

haasn commented Feb 14, 2016

Why are you listing mpv under pr and nowhere else? mpv's default intent is relative colorimetric, and you can choose your own.

Regarding perceptual vs colorimetric, I use a LUT-based profile (created by ArgyllCMS) and I see no difference whatsoever between the two intents.

@haasn
Copy link
Member Author

haasn commented Feb 14, 2016

Does this patch improve things?

diff --git a/video/out/opengl/video_shaders.c b/video/out/opengl/video_shaders.c
index a083364..1b01d87 100644
--- a/video/out/opengl/video_shaders.c
+++ b/video/out/opengl/video_shaders.c
@@ -251,8 +251,18 @@ void pass_linearize(struct gl_shader_cache *sc, enum mp_csp_trc trc)
                              pow((color.rgb + vec3(0.055))/vec3(1.055), vec3(2.4)),
                              lessThan(vec3(0.04045), color.rgb));)
         break;
-    case MP_CSP_TRC_BT_1886:
-        GLSL(color.rgb = pow(color.rgb, vec3(1.961));)
+    case MP_CSP_TRC_BT_1886: {
+        double white = 1.0;
+        double black = 0.05 / 120; // 0.05cd/m² blacks on a 120 cd/m² display
+        double gamma = 1.961;
+
+        double dif = pow(white, 1/gamma) - pow(black, 1/gamma);
+        double gain = pow(dif, gamma);
+        double lift = pow(black, 1/gamma) / dif;
+
+        GLSLF("color.rgb = %f * pow(color.rgb + %f, vec3(%f));\n",
+                gain, lift, gamma);
+        }
         break;
     case MP_CSP_TRC_GAMMA18:
         GLSL(color.rgb = pow(color.rgb, vec3(1.8));)

The idea is to implement the “full” BT.1886 spec, by simulating a device with a real black point (one which the user could possible tune directly in the further). That way we basically get our own “black point compensation” into mpv.

Note that you can get a similar result by playing with the “brightness” and “contrast” controls in mpv, i.e. making the overall image slightly darker or brighter. That method actually has the benefit of preserving image values that were technically below the minimum, i.e. “super-blacks”, if your source has any.

@haasn
Copy link
Member Author

haasn commented Feb 14, 2016

A good test-pattern for testing both black and white clipping at the same time is https://github.com/haasn/cms/raw/8286d4ed3b4a00e5a44e086e931bd2eae669921c/MP4-2c/Basic%20Settings/2-APL%20Clipping.mp4

@4ad
Copy link

4ad commented Feb 14, 2016

@haasn

Does this patch improve things?

It certainly changes things, but doesn't improve them. With iMac color profile and measured native values.

Program black#1 black#7 #black8 #black15 #white15
QuickTime 1-1-1 7-7-7 8-8-8 16-16-16 241-241-241
mpv (recent) 0-0-0 3-4-3 4-5-4 15-16-14 241-240-242
mpv (patch) 2-2-1 8-9-7 10-10-9 21-22-20 241-240-242
mpv (patch + black=1/5000) 1-1-1 7-7-6 8-8-7 19-20-18 241-240-241

I also changed the black level to various values between 1/700 to 1/15000, the best for black#7 and black#8 seems to be around 1/5000, but #black15 is way off.

@haasn
Copy link
Member Author

haasn commented Feb 14, 2016

Well, the idea behind this change is only to raise the black levels overall in order to prevent black crush, it's not to make the transfer curve exactly match QuickTimes.

As far as we can tell, mpv's transfer curve for this profile was fine in theory, but the black crush caused problems in practice.

This patch fixes those practical problems.

@4ad
Copy link

4ad commented Feb 14, 2016

But with simple color profiles, QuickTime and mpv agree, and we agree that it's producing correct output.

This patch not only changes output with complex profile, it changes output with simple profiles too. So if they were right before, now with this patch it means that they must be wrong.

Also, while this patch changes black levels, I don't think the changes are correct. I compare mpv and QuickTime on sRGB-like monitor, with simple profile. I'm watching a real movie, shadow-detail seems good, and consistent between mpv and QuickTime.

Now I watch the same movie on the iMac. Shadow detail is missing (quite obviously I must add, people unaware of this bug here have complained to me about the shadows missing!). When I add your patch (with blacks set to 1/5000), something changes, however, it doesn't look like the output on the sRGB monitor, while the QuickTime output does look the same as mpv and QuickTime on the sRGB monitor.

Also if the problem is related to black-point compensation, surely that must be addressed in the final transformation from the intermediary color space (now BT.709) to the target display space, and not in the intermediary space, no?

haasn added a commit to haasn/mp that referenced this issue Mar 29, 2016
This commit refactors the 3DLUT loading mechanism to build the 3DLUT
against the original source characteristics of the file. This allows us,
among other things, to use a real BT.1886 profile for the source. This
also allows us to actually use perceptual mappings. Finally, this
reduces errors on standard gamut displays (where the previous 3DLUT
target of BT.2020 was unreasonably wide).

This also improves the overall accuracy of the 3DLUT due to eliminating
rounding errors where possible, and allows for more accurate use of
LUT-based ICC profiles.

The current code is somewhat more ugly than necessary, because the idea
was to implement this commit in a working state first, and then maybe
refactor the profile loading mechanism in a later commit.

Fixes mpv-player#2815.
@fhoech
Copy link

fhoech commented Mar 29, 2016

As an aside, with this code layout we could support loading custom “3DLUT” files

The easiest/most straightforward way to facilitate something like this (if desired) is probably to add support for ICC devicelink profiles via LittleCMS. This approach could possibly also used as a general way to cache transforms to disk (as devicelinks) ie. using cmsTransform2DeviceLink to turn the transform into a devicelink which can then be stored for later (re-)use.

@UliZappe
Copy link

This approach could possibly also used as a general way to cache transforms to disk (as devicelinks) ie. using cmsTransform2DeviceLink to turn the transform into a devicelink which can then be stored for later (re-)use.

I’m afraid this would collide with Slope Limiting (contrary to 3DLUTs). I doubt there is a way to encode such behavior into an ICC device link profile, is it?

@fhoech
Copy link

fhoech commented Mar 29, 2016

I’m afraid this would collide with Slope Limiting (contrary to 3DLUTs). I doubt there is a way to encode such behavior into an ICC device link profile, is it?

A parametric tone curve that incorporates the slope limit should work. Of course if the slope limit were to be implemented directly in littleCMS then this wouldn't be needed.

@UliZappe
Copy link

A parametric tone curve that incorporates the slope limit should work.

You mean additional profiles in the profile chain for the device link, or as as a replacement for the original TRCs of the profiles in the profile chain?

Of course if the slope limit were to be implemented directly in littleCMS then this wouldn't be needed.

But how could a CMM apply slope limiting to an existing device link profile? The slope limit algorithm is applied to the conversions to and from the PCS, but there is no PCS in device link profiles. Because of this, my Slope Limit patch for Little CMS ignores device link profiles so far.

@fhoech
Copy link

fhoech commented Mar 29, 2016

as as a replacement for the original TRCs of the profiles in the profile chain?

Yes, a replacement for the original source profile TRC (with the only difference being the slope limit).

But how could a CMM apply slope limiting to an existing device link profile?

Alright, that wouldn't be possible.

haasn added a commit to haasn/mp that referenced this issue Mar 29, 2016
This commit refactors the 3DLUT loading mechanism to build the 3DLUT
against the original source characteristics of the file. This allows us,
among other things, to use a real BT.1886 profile for the source. This
also allows us to actually use perceptual mappings. Finally, this
reduces errors on standard gamut displays (where the previous 3DLUT
target of BT.2020 was unreasonably wide).

This also improves the overall accuracy of the 3DLUT due to eliminating
rounding errors where possible, and allows for more accurate use of
LUT-based ICC profiles.

The current code is somewhat more ugly than necessary, because the idea
was to implement this commit in a working state first, and then maybe
refactor the profile loading mechanism in a later commit.

Fixes mpv-player#2815.
@UliZappe
Copy link

Yes, a replacement for the original source profile TRC (with the only difference being the slope limit).

That was my first approach to the slope limit issue, but unfortunately, from what I can see, it only works in specific cases that we cannot assume. (Which is why in the end I decided to patch Little CMS.)

To build a replacement TRC for a parametric or a gamma TRC is certainly possible. But what do you do if a profile uses a 1DLUT for its TRC? While you could try to translate it to a parametric curve, this is neither trivial nor guaranteed to produce a precise result, whereas the slope limit algorithm I used in Little CMS leaves the original TRC completely untouched apart from the dark area where it intentionally intervenes. So I think the solution I used is more robust and produces superior results, but unfortunately, it does not lend itself to your suggestion with the device link profiles.

@fhoech
Copy link

fhoech commented Mar 29, 2016

But what do you do if a profile uses a 1DLUT for its TRC?

I see. Yes, that case is not easily supported if you don't know exactly how the original 1D LUT was arrived at.

@haasn
Copy link
Member Author

haasn commented Mar 29, 2016

(Surely instead of trying to reconstruct a parametric response curve from the 1D LUT and adding slope limiting to that, you can just directly modify the 1D LUT to incorporate the slope limiting?)

@UliZappe
Copy link

(Surely instead of trying to reconstruct a parametric response curve from the 1D LUT and adding slope limiting to that, you can just directly modify the 1D LUT to incorporate the slope limiting?)

True, but by then a simple one liner to incorporate slope limiting has become an unwieldy amount of code – recalculate and replace gamma curves, parametric curves, and 1DLUTs, for v2 and v4 ICC profiles …

I don’t think that’s worth it just to be able to use device link profiles to be able to write 3DLUTs to disk in ICC format. (And it would be less probable that such an amount of code would make it into the official Little CMS code base.)

@ghost ghost closed this as completed in 2dcf18c Apr 1, 2016
@ghost
Copy link

ghost commented Apr 30, 2016

(This is the wrong issue for this and offtopic, deleting posts.)

@4ad
Copy link

4ad commented May 1, 2016

I am back after some work-related absence.

@UliZappe

So what is the state regarding Adobe CMM emulation in LittleCMS?

In other words, do we have a way to fix the black crush problem?

@haasn
Copy link
Member Author

haasn commented May 1, 2016

@4ad The new BT.1886 emulation should fix black crush for the majority of profiles. (It would still fail if the profiles don't contain accurate reverse tables, since then the code can't estimate the display's black point accurately)

I'm not sure what to do in that situation

@UliZappe
Copy link

UliZappe commented May 2, 2016

First of all, sorry for the long silence; unfortunately, I have very little time currently.

I have now checked the most recent mpv build for its new color management implementation.

First, again the reference image from Melancholia with which it all started. Above is the reference still image, below is mpv:

melancholia-bt1886

To little surprise, the mpv image is again too dark/has too much contrast; but of course, this is what you would expect from a BT.1886 implementation that implies a contrast enhancement.

Second, the QuickTime test movie (QuickTime Player above, mpv below):

quicktime test pattern hd bt 1886

If anything, the current implementation makes the black crush issue worse; it certainly does not fix it.

I have no interest in rekindling the ideological debates about color management again. Maybe it’s a question of semantics:

  1. If you consider color management to be a technology that controls the color reproduction of one specific application and if you assume a dim viewing environment as a given so that you can “hardwire” the corresponding contrast enhancement, then the current implementation might work at least partly (apart from the black crush issue).
  2. If, in contrast, you consider color management to be a technology to provide color consistency between applications, then the current approach definitely does not work. (mpv would have to meet the Melancholia still image which is displayed by another application, but does not.)

If you use the term “color management” in the second way (as I do), then the only way to adjust contrast for a dim viewing environment would be to use a color appearance aware display profile which would be used by both applications alike.

Of course, all this isn’t surprising; it’s what I’ve said all along.

Let’s just say that those who understand the term “color management” in the second way and want to make sure the black crush issue is fixed need a different option in mpv which does not exist yet. This option would have to be a emulation of the QuickTime behavior, i.e. a source gamma of 1.961 and LittleCMS with a slope limit of 16.

I will report on that in my next post.

@haasn
Copy link
Member Author

haasn commented May 2, 2016

If anything, the current implementation makes the black crush issue worse; it certainly does not fix it.

Seems like your profile has inaccurate reverse tables. mpv relies on them being accurate in order to map black to black. A profile with accurate reverse tables should have no black crush almost by definition.

If, in contrast, you consider color management to be a technology to provide color consistency between applications, then the current approach definitely does not work.

This in itself is a good reason to provide compatibility / emulation modes where appropriate, but I can't help but wonder whether it's mpv that should be emulating QuickTime or QuickTime that should be emulating mpv.

A priori, mpv follows the BT.1886 spec, QuickTime follows some made-up Apple/Adobe spec, so to match them either one of the two would have to implement support for emulating the other.

But I guess it's not like we can change QuickTime either way, so the best we can hope to provide is implement a compatibility mode inside mpv.

@UliZappe
Copy link

UliZappe commented May 2, 2016

Seems like your profile has inaccurate reverse tables.

It’s a matrix profile which has no tables at all.

@haasn
Copy link
Member Author

haasn commented May 2, 2016

It’s a matrix profile which has no tables at all.

That's a problem, the profile must have reverse tables for mpv to be able to figure out what the display's black point is.

I'm not sure how to work around this. Pure matrix profiles like yours contain no information whatsoever about the target device's contrast, do they?

Edit: Well, I could still add an option like target-contrast to specify the value manually in cases like these.

@UliZappe
Copy link

UliZappe commented May 2, 2016

I'm not sure how to work around this.

Slope limiting. 😎

Pure matrix profiles like yours contain no information whatsoever about the target device's contrast, do they?

No, they don’t. But they are the most common type of display profile (and for good reasons: they are efficient – important for all moving content –, and tables are not really needed for display profiles where non-linearities can be dealt with in the video card LUT).

@UliZappe
Copy link

UliZappe commented May 2, 2016

@4ad

If I am reading this thread correctly, it seems mpv is waiting for the slope-lmiting patch to be merged into LittleCMS before enabling any slope-limiting functionality, correct?

Yes.

Now comes the bad news: Marti Maria (the author of LittleCMS) does not want to incorporate the patch.

The “core patch” is as simple and clean as it gets, but to propagate the user option setting to the function that implements the patch, several other functions must be modified, which Marti dislikes.

But that’s not the main point. The main point is – I can hardly believe it – patents. Marti is afraid that this extremely simple formula might be patent encumbered. He used to work for HP, who supported him in such matters (figuring out the legal situation for him), but does not work there anymore.

Therefore, he asked me to use a plug-in for the implementation instead. I still don’t think that’s fully possible, but promised him to try. Unfortunately, because of my current time restraints, I haven’t gotten too far with it, yet.

But of course, this would only relocate the possible patent issue. It would not be in LittleCMS anymore, but it would be in the plug-in.

@wm4

I would think that a project like mpv had already have to deal with patent related questions. Is there some kind of “standard procedure” how to proceed in cases like this?

Anyway, this leaves us with a situation with (if I get it correctly) the following alternatives:

  1. The patent issue is cleared, and Marti agrees to incorporate my patch. (This would be the best variant by far.)
  2. The patent issue is cleared, and mpv includes a plug-in which I will write that enables slope limiting in the standard LittleCMS distribution. This implies that implementing such a plug-in is technically possible, which I currently doubt.
  3. My LittleCMS fork with the slope limit implementation is used. I would volunteer for the time being to keep it up-to-date with new releases of LittleCMS as long as no cease and desist letters arrive, and mpv would have to look if the slope limit option is defined before applying it. This way, users could enable slope limiting for themselves by using my LittleCMS fork.
  4. mpv uses ColorSync. Works out-of-the-box, but only on OS X.
  5. mpv does not use slope limiting and emulates QuickTime only by using a gamma of 1.961. Black crush then can only be prevented by adequately built monitor profiles (implementing a black point compensation option in mpv would definitely be helpful!).

Have I forgotten anything?

Oh well … what should we do?

@UliZappe
Copy link

UliZappe commented May 2, 2016

@haasn

This in itself is a good reason to provide compatibility / emulation modes where appropriate, but I can't help but wonder whether it's mpv that should be emulating QuickTime or QuickTime that should be emulating mpv.

But in the Melancholia image comparison, the still image is not displayed by QuickTime Player, but by a color managed image editing application (which itself is completely consistent with all other OS X applications which handle still images, and with spectrometers). The point with QuickTime’s approach is that it provides this consistency for moving images, too.

A priori, mpv follows the BT.1886 spec, QuickTime follows some made-up Apple/Adobe spec, so to match them either one of the two would have to implement support for emulating the other.

Purely for the sake of simplicity, I suggest the following terminology:

color management for color reproduction in one specific application (point 1. above)
ICC color management for color reproduction that takes care of color consistency between applications (point 2. above)

Using this terminology, my argument has always been that the BT.1886 approach is not applicable to ICC color management because it displaces the contrast enhancement for a dim viewing environment from the display profile (where it belongs architecturally in ICC color management) to the source application. So from this perspective, QuickTime clearly is the rule and BT.1886 the exception.

@UliZappe
Copy link

UliZappe commented May 2, 2016

@haasn

Edit: Well, I could still add an option like target-contrast to specify the value manually in cases like these.

But how can the user know this value? This reintroduces the guesswork that color management is supposed to remove.

@4ad
Copy link

4ad commented May 2, 2016

I remember a previous patch that had a target-contrast value (or equivalent). It made things worse for me at every value, albeit the current code is perhaps different, and this previous observation is meaningless.

I posted previous measurements earlier in this thread. To recap, at realistic contrast ratios it made the problem imperceptibly worse, and at ridiculous contrast ratios (1:10000), black crush was solved at the expense of huge deviations of midtones.

@4ad
Copy link

4ad commented May 2, 2016

Why can't we implement slope limiting by editing the LUT generated by LittleCMS?

@UliZappe
Copy link

UliZappe commented May 2, 2016

Why can't we implement slope limiting by editing the LUT generated by LittleCMS?

I would assume that once a LUT is created, it’s opaque and hardly editable, but I’m no expert on that.

Anyway, we do have the correct implementation that emulates ColorSync’s behavior perfectly and in every situation. It seems to be more of a (potentially) legal issue, and I would assume this would apply to LUT editing as well.

@4ad
Copy link

4ad commented May 2, 2016

Oh boy, please disregard the previous message (deleted). I am very tired. It was intended for a very different project.

@haasn
Copy link
Member Author

haasn commented May 3, 2016

Why can't we implement slope limiting by editing the LUT generated by LittleCMS?

It would be doable but complicated. Inside the 3DLUT there are three steps:

  1. Convert from input space to PCS
  2. Desaturate inside PCS to make it fit into the target gamut
  3. Encode from PCS to the target space (via ICC LUT etc.)

Slope limiting needs access to information from step 2. Specifically, the algorithm does something like out = min( ICC(in), in * V ). (Although I'm not entirely sure why it's ‘min’ here and not ‘max’, when the goal is to eliminate black crush..)

Anyway, to replicate the algorithm we would probably need to reimplement LittleCMS' desaturation algorithm, although we could theoretically get away with it due to the fact that it only matters for things close to black (which are very desaturated either way).

@UliZappe
Copy link

UliZappe commented May 3, 2016

(Although I'm not entirely sure why it's ‘min’ here and not ‘max’, when the goal is to eliminate black crush..)

Slope limiting applies to both directions, and depends on the direction (F = slope limit factor):

for the RGB to PCS direction:

  out = max ( ICC(in) , in / F )

for the PCS to RGB direction:

  out = min ( ICC(in) , in * F )

It is the RGB to PCS direction where the black crush is prevented.

This issue was closed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants