Color Management (OS X): Image is too dark

[UliZappe]

I have compared the icc-profile= setting of mpv to Apple’s native QuickTime Player X on OS X 10.9.1 and found that the resulting colors are basically correct, but the image is too dark.

The following images are from a scene from Lars von Trier’s Melancholia. The first image is from QuickTime Player X:

This image is from a h.264 file (in an mp4 container) extracted from the BluRay version of the movie. I have compared it to the DVD version (played in Apple’s DVD Player) and to the version from the iTunes store, and the colors are exactly the same in these three versions and also match high quality still images I have from this movie.

So this image is arguably the reference and shows how the colors should look like.

The next image is from the same mp4 file, played in mpv without color management:

This image makes it very obvious why color management is a must; aesthetically, it’s basically another image (technically, it’s too red and too saturated).

The third image is again from the same mp4 file, played in mpv with color management (using the default rendering intent, which is absolute – choosing another intent does not fix the problem discussed here):

As you can see, compared to the QuickTime version, the colors seem to be more or less correct now, but the image is obviously too dark.

Now I wonder if this is a deficiency of the liblcms2.2.dylib library that mpv uses, or if it's just that mpv uses liblcms2.2.dylib with incorrect or suboptimal parameters?

Could somebody familiar with this part of the mpv code base comment?

[haasn]

Hey, thanks for reporting this. I'd like to ask what devices you're displaying each of these images on. Are you using the same monitor for everything, including the Apple DVD Player? The first thing that came to mind is the difference between television sets and computer monitors (which tend to be brighter), so it's possible that some program is making an incorrect assumption here.

It might also be possible that mpv is making an incorrect assumption about what the expected input gamma of the movie is. I wasn't quite sure about this when implementing it so any feedback is much appreciated.

[UliZappe]

Yes, these are all screenshots from the very same monitor (a 30" Apple Cinema Display in my case). I also used this monitor when cross-checking with Apple’s DVD Player and the iTunes version.

The first thing that came to mind is the difference between television sets and computer monitors (which tend to be brighter)

Hm, yes, but the very idea of color management is to abstract from any hardware. The task of the monitor profile is exactly to convert the colors such that they look exactly the same on any output device with a corresponding monitor profile.

That some devices are "brighter" than others is pre color management thinking, so to say; this shouldn't matter.

You have a point in that usually there are no factory monitor profiles for TV sets available, and it’s very hard to create them by yourself (almost no suitable measurement devices available). That’s why personally, I prefer either computer displays or projectors to watch movies. Both can be profiled very exactly.

It might also be possible that mpv is making an incorrect assumption about what the expected input gamma of the movie is.

Hm, unfortunately, I have little knowledge about the video side of things, but I would have guessed that the BT.709 and BT.601 profiles define the gamma of the video source (just as ICC profiles embedded in images document the gamma of the image’s color space). The gamma of the targeted display is included in its monitor profile, it will most probably (but not necessarily) be either 1.8 or 2.2. (My display is calibrated to gamma 2.2, which is the OS X default since Snow Leopard (and has always been the Windows default)).

So basically, Little CMS must gamma match the video source gamma as specified in the video profile to the display gamma as specified in the display profile.

From what it looks like, the issue might well be gamma related. (The other, more remote possibility that comes to my mind would be some incorrect blackpoint compensation setting.)

I use a prebuilt version of mpv as I didn't want to deal with all the dependencies of the code base, but if I can help by checking modified binary versions, I’d be glad to do so.

[Pacoup]

Actually, it's the opposite. This is well-known issue with QuickTime (all versions on OS X for as long as I can remember) where the gamma is set too high for H.264.

(Additional notes: This does not happen with mastering codecs such as ProRes because they are tagged with a colour profile which QuickTime understands, e.g. when coming out of Premiere Pro, but by default, QuickTime and any QuickTime-based playback on OS X will happen with a slightly lifted gamma.)

[UliZappe]

Actually, it's the opposite. This is well-known issue with QuickTime (all versions on OS X for as long as I can remember) where the gamma is set too high for H.264.

This contradicts my findings that the H.264 version, the DVD version (MPEG-2, no H.264 involved) and a still image (TIFF format in Adobe RGB (1998)) sport exactly the same colors.

Also, I’ve never heard of this issue in connection with QuickTime Player X, which is 64 bit and based on the new AV Foundation framework. (It kind of existed with QuickTime Player 7, which was not color managed, was 32 bit and built with the old QuickTime framework.) Do you have any (current, i.e. referring to OS X 10.8 or 10.9) link about this issue?

[Pacoup]

My apologies, saying it's related to H.264 may have been confusing matters. I believe it applies to any codec, as will be demonstrated below. (I was wrong in the previous post)

The fact of the matter is, you are observing the brighter so-called reference image in three Apple software using exactly the same video playback framework in OS X:

• QuickTime Player
• DVD Player
• iTunes

This is by no means reference matterial. You're just replicating the issue thrice.

The fact that QuickTime Player X is based on new APIs does not change the fact that Apple software does not handle gamma correctly. From OS X 10.6 and on, the underlying APIs have changed, but the issue has not.

Let me demonstrate (all samples produced under OS X 10.9 Mavericks).

In the following sample, the same H.264 video shows a slightly brighter yellow in QuickTime than the other players.

In the same breadth of ideas, this static colour video rendered in Adobe Premiere Pro shows wrong colours in QuickTime when encoded in both H.264 and ProRes, plus, to top it all off, both H.264 and ProRes render differently. (I could not get H.264 and ProRes in MPV simultaneously in the same screenshot, but H.264 renders the same in it, i.e. correctly)

This is notable because, just as the same video will render the same colours on the same machine under Windows and OS X except in QuickTime, the issue is actually replicated in professional media production software.

My video producer friends have been complaining about this forever, mastering their video in Apple software and realizing it's all wrong everywhere else.

[haasn]

Thanks a lot for the additional input.

Hm, yes, but the very idea of color management is to abstract from any hardware. The task of the monitor profile is exactly to convert the colors such that they look exactly the same on any output device with a corresponding monitor profile.

You're absolutely right, in a perfect world monitor profiles would be an exact resolution of this - however in practice it may for example happen that a studio masters their content as too bright to overcompensate for average TV sets being darker than the BT.709 transfer curves, so an implementation aimed at computer monitors or profiled devices may wish to compensate. Regardless, I don't think either of these playback mechanisms are doing that - and the symptoms would be the opposite (video too dark, not too bright). I'm certain we don't do anything of the sort in mpv, as explained a bit below.

Hm, unfortunately, I have little knowledge about the video side of things, but I would have guessed that the BT.609 and BT.701 profiles define the gamma of the video source (just as ICC profiles embedded in images document the gamma of the image’s color space).

Rec BT.709 defines a segmented input gamma curve used for non-linear precorrection, which is approximately equal to gamma=0.45 (1/0.45 = 2.222..). We use 1/0.45 for our gamma precorrection (and linear light scaling). Looking at the code base again, it seems like a tiny accuracy gain could be gained here by using a segmented curve as input for LCMS (the computational complexity of which is abstracted away by the LUT one way or the other), but it should only affect the lowest two or three black levels and have no noticeable effect on the gamma issue you are experiencing whatsoever (since the exponent is the same). I might as well go and patch it in anyway.
Edit: Merged in 257d9f1, I could not notice any difference though.

a still image (TIFF format in Adobe RGB (1998))

Ah, it sounded like you had an officially printed copy, that confuses matters.

I can't comment on the OS X issue but from what it sounds like @Pacoup is reporting, the Apple media frameworks could be bugged? It sounds odd, but could you test it using a DVD player from a different manufacturer perhaps? (I'm not sure how many are color managed or anything like that, though)

[UliZappe]

@Pacoup:

In the following sample, the same H.264 video shows a slightly brighter yellow in QuickTime than the other players.
[…]
In the same breadth of ideas, this static colour video rendered in Adobe Premiere Pro shows wrong colours in QuickTime
[…]
This is notable because, just as the same video will render the same colours on the same machine under Windows and OS X except in QuickTime

So basically, your argument goes like this:

QuickTime renders colors differently to a lot of other software.

So far, this is correct.

Now, either because you assume the majority must be right, or for purely pragmatic reasons, you conclude that QuickTime gets it wrong while all the other software gets it right.

This conclusion is not cogent and is, in fact, incorrect, as I will show below. QuickTime gets it right, the other software doesn’t.

My video producer friends have been complaining about this forever, mastering their video in Apple software and realizing it's all wrong everywhere else.

There has been a lot of confusion about color management among graphic artists (it’s getting better lately), and instead of getting the color management workflow right, many people tinker with some other settings, only making things worse.

I’ve spent this morning with reading some discussions among videographers and got the impression that the confusion about color management is even worse among this group of users.

So I prepared a test case to clear things up. I did all tests on OS X 10.9.1.

The following is a 1920 x 1080 pixel PNG image in sRGB color space with six colors, the RGB values of which are specified in the image:

(The image was created in Pixelmator, which uses the OS X system wide CMM, ColorSync.)

You can open this image in Preview and check the colors with the DigitalColor Meter application included with OS X:

The DigitalColor Meter will show you the sRGB color values of the color under the mouse cursor. If you have a decent monitor profile, you should find that the colors are correct (= as specified in the image). Note that this works independently of the monitor profile you use. Also note that the values might deviate +-3; this is because the DigitalColor Meter actually measures the screen pixel color, i.e. the color is converted from sRGB (the image color space) to the display color space and then (within DigitalColor Meter) is converted back to sRGB. Since ICC color conversion isn’t lossless, rounding errors are bound to occur.

My actual measurement results were:

201 51 51
52 201 51
51 51 201
202 101 101
101 201 100
101 101 201

This is as good as it gets in color management reality.

Now, I imported (Cmd-I) this image into Final Cut Pro X, which is the first color managed Final Cut Pro version (and possibly even the first (and only?) color managed video NLE in general, I’m not sure about that).

I created a corresponding new project. When dragging the image into the new project, I specified the following video properties:

Then, without any modification, I exported the project by choosing File > Share > Apple Devices 1080p and using the following settings:

This results in a QuickTime movie. Note that it has a video color profile embedded, as the Finder’s Info panel reveals:

Unfortunately, though the movie file is only 240kB (when trimmed to 1s duration), I cannot share it here since embedding movies is not supported in this forum. You can either reproduce the above steps by yourself, or else download it from http://www.datafilehost.com/d/68595d1b.

Anyway, I opened this movie in QuickTime Player X and again measured the colors, just as with the PNG image above. These are my results:

199 50 49
53 202 50
49 49 198
202 102 100
103 200 103
102 102 202

IMHO, for a video file, these are almost perfect colors. It’s video color management at its best.

In comparison, if I open the same movie in mpv, I get these values:

193 39 36
39 194 38
35 47 194
192 87 87
87 194 87
88 88 193

So, as you can easily see, the colors are not only darker in mpv, they are also technically incorrect. The same goes for VLC and the many other players with this flavor of color reproduction. The one software that gets it right is QuickTime.

In the case of the original image from Melancholia, I know that the director intended it to look like the first instance, not like the third one, but obviously, you can believe me or not, or we could endlessly argue about aesthetics. I hope this purely technical demonstration makes it clear which of the color reproductions is the correct one.

[pigoz]

You can use http://www.datafilehost.com/ or Dropbox to host the file

[UliZappe]

OK, you can download the movie file at http://www.datafilehost.com/d/68595d1b.

[UliZappe]

An additional note: The movie is only 1 s long, so you must pause it immediately after opening it in mpv to prevent that mpv quits again.

[pigoz]

I can get out the same results you described, I wonder if it's possible to extract that color profile from the video. Maybe using that as icc-profile with mpv would improve the situation.

[UliZappe]

I wonder if it's possible to extract that color profile from the video. Maybe using that as icc-profile with mpv would improve the situation.

Unfortunately, I don’t know that much about video color management so far, but my guess would be that what is embedded is not an actual profile but rather a tag that describes the color space used (just as many images don’t actually include the sRGB profile, but just indicate that their color space is the – standardized – sRGB. So HD (1–1–1) maybe simply means BT.709. But as I said, so far I’m not sure.

[pigoz]

I opened the png you attached in the discussion with mpv (mpv --paused color-test.png) and the "Display in sRGB" colors in DigitalColor Meter seem to be correct. I suspect the problem is indeed related to the color tagging applied by Final Cut Pro and the fact that only Apple frameworks recognizes it.

[haasn]

Out of curiosity, could you show a screenshot of what QuickTime renders when color management is disabled?

I noticed that the gamma of mpv's color managed version seems similar to its non-color-managed version, so I'm not sure if the difference between QuickTime and mpv here is strictly tied to color management at all or if there's an inherent brightness difference even with it disabled (which would also rule out LittleCMS).

I opened the png you attached in the discussion with mpv

I wouldn't rely on mpv rendering png files correctly (with respect to color management), I have no idea what it does but it probably just decodes the PNG naively and then color manages it as if it was BT.709 input material (regardless of the input profile tag), which for untagged PNG should be correct save for the gamma curve.

I suspect the problem is indeed related to the color tagging applied by Final Cut Pro and the fact that only Apple frameworks recognizes it.

This shouldn't affect the samples obtained from the Blu-ray though, since that's untagged BT.709, unless @UliZappe transcoded it into a different color space (+ tagging).

[UliZappe]

@nandykins

You're absolutely right, in a perfect world monitor profiles would be an exact resolution of this - however in practice it may for example happen that a studio masters their content as too bright to overcompensate for average TV sets being darker than the BT.709 transfer curves, so an implementation aimed at computer monitors or profiled devices may wish to compensate.

I know that in the "real world"® things like this happen, but this is what above I referred to as instead of getting the color management workflow right, many people tinker with some other settings, only making things worse. In the end, you’re just opening a can of worms. Therefore I would argue that in implementing color management in an application like mpv it is crucial to stick to the standards, no matter what the current practices are. Otherwise, the standards will never have a chance to become commonplace.

I'm certain we don't do anything of the sort in mpv, as explained a bit below.

OK; so we agree here. :-)

Rec BT.709 defines a segmented input gamma curve used for non-linear precorrection, which is approximately equal to gamma=0.45 (1/0.45 = 2.222..). We use 1/0.45 for our gamma precorrection (and linear light scaling). Looking at the code base again, it seems like a tiny accuracy gain could be gained here by using a segmented curve as input for LCMS (the computational complexity of which is abstracted away by the LUT one way or the other), but it should only affect the lowest two or three black levels and have no noticeable effect on the gamma issue you are experiencing whatsoever (since the exponent is the same). I might as well go and patch it in anyway.
Edit: Merged in 257d9f1, I could not notice any difference though.

Yes, this is a very subtle difference and certainly not the deviation we’re confronted with here.

a still image (TIFF format in Adobe RGB (1998))

Ah, it sounded like you had an officially printed copy, that confuses matters.

Now I’m confused – not sure where we misunderstood each other. The TIFF is "official" in that’s it’s extracted from an "official" PDF of the production. It was embedded in the PDF and its color space was Adobe RGB (1998).

Not sure how this confuses matters.

I can't comment on the OS X issue but from what it sounds like @Pacoup is reporting, the Apple media frameworks could be bugged?

Hopefully, meanwhile it has become clear that the bugs are on the "other side" and the Apple media frameworks are correct.

It sounds odd, but could you test it using a DVD player from a different manufacturer perhaps? (I'm not sure how many are color managed or anything like that, though)

I can’t because there aren’t any (to the best of my knowledge). QuickTime vs. the rest of the world probably means consistent color management vs. inconsistent color management or no color management at all.

[UliZappe]

@pigoz

I opened the png you attached in the discussion with mpv (mpv --paused color-test.png) and the "Display in sRGB" colors in DigitalColor Meter seem to be correct.

Very interesting! I wasn’t aware mpv can open images.

And you’re right, the colors are rendered 100% correctly.

I suspect the problem is indeed related to the color tagging applied by Final Cut Pro

It’s indeed just tagging:

I found an Automator action that can attach profiles to video files:

If you attach another profile to a copy of a video and make a diff between the two versions, it’s only a few bytes in the file header that have changed.

There are only 3 profile options: SD (Rec. 601) | HD (Rec. 709) | RGB

Additionally, there is an Exported on: setting with the values Mac | PC. I have no real idea what this setting refers to, and it seems to make no difference at all in color reproduction for the SD and HD settings. In the case of RGB, however, the difference is huge (and the colors of the PC version are far off on a Mac).

The Finder reports files generated with these settings as follows:

SD (Rec. 601) – SD (6-1-6)
SD (Rec. 601) [PC] – SD (6-1-6)
HD (Rec. 709) – HD (1-1-1)
HD (Rec. 709) [PC] – HD (1-1-1)
RGB – RGB (2-2-6)
RGB [PC] – RGB (1-2-6)

Again, no difference for SD and HD with regard to the Mac/PC setting.

So our assumption was correct that these profiles basically flag BT./Rec. 601 / BT./Rec. 709 for video files (not sure when RGB is needed).

BTW, does anyone know what the (1-1-1) etc. values refer to? I had found some info earlier on, but cannot remember it or find it now.

and the fact that only Apple frameworks recognizes it.

Here, I’m not so sure. Since it’s basically HD vs. SD, absent a recognizable color profile/flag, a software has to take an educated guess from the video parameters, anyway, has it not?

[UliZappe]

@nandykins

Out of curiosity, could you show a screenshot of what QuickTime renders when color management is disabled?

Unfortunately, no. There is no way to disable color management in QuickTime Player X. The only thing I can think of would be to remove the color profile from the movie file, but 1.) the Automator action described above only allows to change the color profile, not to delete it, and 2.), even if it did, QuickTime Player X would probably take an educated guess.

I noticed that the gamma of mpv's color managed version seems similar to its non-color-managed version, so I'm not sure if the difference between QuickTime and mpv here is strictly tied to color management at all or if there's an inherent brightness difference even with it disabled (which would also rule out LittleCMS).

I can hardly imagine another culprit than video color management, because otherwise, why would mpv handle the PNG image correctly?

I wouldn't rely on mpv rendering png files correctly (with respect to color management), I have no idea what it does but it probably just decodes the PNG naively and then color manages it as if it was BT.709 input material (regardless of the input profile tag), which for untagged PNG should be correct save for the gamma curve.

But the colors are completely correct. I would be amazed if this happened by chance.
You are right. I just tried a variant of the image converted to ProPhoto RGB, and the colors are completely off in mpv in this case.

I suspect the problem is indeed related to the color tagging applied by Final Cut Pro and the fact that only Apple frameworks recognizes it.

This shouldn't affect the samples obtained from the Blu-ray though, since that's untagged BT.709, unless @UliZappe transcoded it into a different color space (+ tagging).

No, I did not transcode it, I just had to exchange the container (mkv > mp4) to be able to open it in QuickTime Player X; I did this using Subler. However, I just cross-checked in the Finder, and apparently, this was already enough to tag the mp4 file as HD (1-1-1). But transcoded it wasn’t.

Unfortunately, I’m not familiar with video programming at all, but in case this helps, there is an Xcode example called From a view to a movie. It seems to do the same thing as mpv in that it builds a color matrix from the color management parameters and then applies this matrix to the OpenGL output. Maybe this code is helpful in understanding what’s going on?

[Pacoup]

Now, either because you assume the majority must be right, or for purely pragmatic reasons, you conclude that QuickTime gets it wrong while all the other software gets it right.

Not really. I was just going on an assumption based on my own experience, albeit limited, with video production on OS X vs video production on Windows, where that problem has not occurred to me.

There has been a lot of confusion about color management among graphic artists (it’s getting better lately), and instead of getting the color management workflow right, many people tinker with some other settings, only making things worse.

That, I agree. I myself do not fully understand the underlying reasons and technicalities behind the issue and really can't bring any further solid data on the subject.

Now, I imported (Cmd-I) this image into Final Cut Pro X, which is the first color managed Final Cut Pro version (and possibly even the first (and only?) color managed video NLE in general, I’m not sure about that).

Well, only in the sense that FCPX has a complete end-to-end color workflow via ColorSync.

When using my Mac with a calibrated display profile, the color is the same end-to-end from Premiere to the export preview to MPV, etc. Except it's different in QuickTime. But the opposite is true if using a workflow starting with FCPX. So it looks likes it's more something along the lines of incompatible color profile data.

Or, well, everybody else could be not fully color managed.

Just a thought.

[UliZappe]

When using my Mac with a calibrated display profile, the color is the same end-to-end from Premiere to the export preview to MPV, etc. Except it's different in QuickTime.

So that means that if you recreate my test from above, but with Premiere (i.e. if you import the PNG image from above into Premiere instead of FCPX to create the corresponding movie), the colors are correct (as reported by DigitalColor Meter) when playing the resulting movie in mpv? Could you check this? (I don’t have Premiere, so I can’t.)

So it looks likes it's more something along the lines of incompatible color profile data.

That would be strange. I mean, you would think that Rec. 709 is well defined, and we found above that the "profile" the Finder reports is probably not much more than the movie file being tagged as Rec. 709.

Or, well, everybody else could be not fully color managed.

This could be checked when you recreate my test with Premiere.

1. Recreate my test with Premiere. As described above, the colors should be correct (as reported by DigitalColor Meter) when playing the resulting movie in mpv.
2. If this test is passed, change your monitor profile to something very different (ProPhoto RGB would be a good choice), for your Mac in System Preferences and at the same time for mpv in its config file. As a result, your screen colors everywhere (including mpv playing the test movie) should look completely off (pale in the case of ProPhoto RGB), but DigitalColor Meter should still report the same, correct sRGB data for the test movie played in mpv.
3. If this test is passed, recreate the test movie a second time in Premiere, while still using ProPhoto RGB as your monitor profile. (I don’t know whether Premiere will automatically use the monitor profile chosen in System Preferences, or whether you’ll have to switch this setting manually – but of course, the monitor profile used in Premiere should be in sync with the rest of your Mac.) Since you’re using an incorrect monitor profile, the colors should look consistently off everywhere (in Preview displaying the PNG test file, in Premiere and in mpv), but DigitalColor Meter should again report the same, correct sRGB data for the second test movie played in mpv.
4. If this test is passed, switch your Mac and mpv back to your original monitor profile and play the second test movie again in mpv. DigitalColor Meter should again report the same, correct sRGB data for the second test movie, and additionally, the colors of the test movie played in mpv should now look correct on the screen (as you’re using a correct monitor profile).

(Tests 3 and 4 are required to prevent a seemingly "correct" test result "by chance", as was the case above with the PNG image seemingly displayed correctly in mpv.)

If all 4 tests are passed successfully (no deviations > 3 in DigitalColor Meter), the workflow from Premiere to mpv is indeed correctly color managed (but incompatible with QuickTime for whatever reason). If not, the workflow is not successfully color managed.

[haasn]

Since I don't have OS X, nor do I have sufficient amounts of information and judgment to discern which of mpv, QTX, etc. are incorrect here, nor can I notice anything obviously wrong with the mpv code base w.r.t color management, I'm going to go ahead and try to construct an exact test file and calculate the correct values by hand.

[pigoz]

I looked for some details about QTX color management and it looks like QTX uses the colr atom of the mov and mp4 containers when present. This has a nclc tag which defines the colorspace of the video.

Source: https://developer.apple.com/library/mac/technotes/tn2227/_index.html

[UliZappe]

@nandykins

Wouldn’t it be easiest if somebody who has Premiere first replicated my test as described in my last post?

If the test fails, then we’ll know which toolchain is correct wrt/ color management, and which isn’t.

If it doesn’t fail, then things would become much trickier, of course, and your suggested procedure would certainly be useful.

[Pacoup]

@UliZappe Okay, I'll replicate the test when I have time tonight, screenshots and all =D

[UliZappe]

@Pacoup

Great! :-)

If all 4 tests succeed, just to be completely sure, you could try a 5th and last one: Just like test 1, but use a copy of the test PNG image which you converted to ProPhoto RGB color space before. (Which means the colors should remain the same on the screen (when the image is displayed in Preview), and DigitalColor Meter should still report the same sRGB values, it’s only that internally the image will be in ProPhoto RGB. If color management is correct, of course this should still work when imported into Premiere – it does with FCPX.)

[haasn]

I've generated the following test clip:

100x100 containing 200/50/50 as YV12 (file consists of 10000 \200 bytes
followed by 5000 \50 bytes), interpreted as BT.709.

You can find this test clip here: http://www.nand.wakku.to/video.yuv

I'm playing it back as follows:

mpv video.yuv --demuxer=rawvideo --demuxer-rawvideo=w=100:h=100:mp-format=yv12 -pause --colormatrix=BT.709 -vo opengl-hq:icc-profile=/mem/AdobeRGB1998.icc:dither-depth=16

• Note that YV12 here represents Y'CbCr, not Y'UV (which has no relevance)

Using AdobeRGB1998.icc obtained from this link: http://www.adobe.com/digitalimag/adobergb.html

I've then captured the raw output of mpv's window (as sent to the GPU, which
further applies three LUTs for color correction and sends it as-is to the
monitor - the GPU LUTs don't affect the output of xwd, though, they're
functionally part of the monitor) as follows:

xwd -nobdrs | xwdtopnm > screen.ppm

Analyzing the screen.ppm using hexdump gives me the raw RGB values (a1,ff,57)
or (161,255,87) which represent the definitive, final AdobeRGB values assuming
everything is correct.

Running this computation manually:

Rec BT.709 specifies the derived signals are as follows:

DY'  = 219 EY'  + 16
DCb' = 224 ECb' + 128
DCr' = 224 ECr' + 128

So, reversing this, we get:

EY'  = (200-16)/219 ≈ 0.84
ECb' = ECr' = (50-128)/224 ≈ -0.348

For conversion from RGB, the rec specifies:

EY'         =  0.2126 ER' + 0.7152 EG' + 0.0722 EB'
ECb'*1.8556 = -0.2126 ER' - 0.7152 EG' + 0.9278 EB'
ECr'*1.5748 =  0.7874 ER' - 0.7152 EG' - 0.0722 EB'

Solving this for ER'/EG'/EB' in matrix form gives us:

ER' = 0.292
EG' = 1.068 (clipped to 1)
EB' = 0.194

As obtained using the following WolframAlpha query:
https://www.wolframalpha.com/input/?i=row+reduce+%7B%7B0.2126%2C0.7152%2C0.0722%2C0.84%7D%2C%7B-0.2126%2C-0.7152%2C0.9278%2C-0.348*1.8556%7D%2C%7B0.7874%2C-0.7152%2C-0.0722%2C-0.348*1.5748%7D%7D

To obtain the actual luminance (R, G and B) we need to calculate according
to the opto-electronic transfer characteristics, defined by the Rec as:

V = 1.099 L^0.45 - 0.099 for L ≥ 0.018
V = 4.500 L              for L < 0.018

Reversing this gives us:

L = (V+0.099)/1.099 ^ (1/0.45) for V ≥ 0.0812
V = L/4.5                      for V < 0.0812

Plugging this into the values for ER'/EG'/EB' gives us:

R = 0.1
G = 1
B = 0.053

Since these are RGB values with respect to BT.709, we need to conver these
to space-independent XYZ values. This is done using the conversion matrix
derived from the specified primaries. I will be using the matrix obtained from
http://www.brucelindbloom.com/index.html?Eqn_XYZ_to_RGB.html as listed for
sRGB (which has the same primaries as BT.709) which I've confirmed previously
are correct.

Plugging it into the R/G/B values gives us:

X = 0.408
Y = 0.740
Z = 0.172

https://www.wolframalpha.com/input/?i=%7B%7B0.4124564%2C0.3575761%2C0.1804375%7D%2C%7B0.2126729%2C0.7151522%2C0.0721750%7D%2C%7B0.0193339%2C0.1191920%2C0.9503041%7D%7D*%7B0.1%2C1%2C0.053%7D

Doing the same in reverse for AdobeRGB 1998 gives us:

R = 0.356
G = 1
B = 0.093

Converting these back to the component R'/G'/B' coordinates using the inverse
of the power factor defined as 2+51/256=563/256, or 256/563:

R' = 0.625
G' = 1
B' = 0.34

And finally, scaling these back to AdobeRGB's 0-255 range:

DR' = 160
DG' = 255
DB' = 87

Which matches mpv's actual output of (161,255,87) to within rounding error.

For additional verification, I've encoded the video.yuv file using x264 as
follows:

x264 video.yuv --input-res 100x100 --input-csp yv12 --input-range tv -o video.mkv --colorprim bt709

You can find the resulting video.mkv file here: http://www.nand.wakku.to/video.mkv

I then played back the mkv as followed:

mpv --colormatrix=BT.709 video.mkv --pause -vo opengl-hq:icc-profile=/mem/AdobeRGB1998.icc:dither-depth=16

And repeated the screenshot procedure from above, which resulted in the
identical values (161,255,87), confirming that either both x264 and mpv handle
this correctly or that x264 and mpv's H.264 decoder have identical errors,
which I rule out as implausible for now.

What does this tell us? That, at any rate, mpv seems to handle BT.709 Y'CbCr
data and ICC profiles correctly, using the raw input or H.264 - which is the
most pressing type of source that we care about getting correctly. It also rules out obvious bugs in LittleCMS.

Of course, all of this is running on Linux. It's possible that OS X does something else to the mpv's output window instead of simply passing the raw values to the device as profiled by the ICC profile. If this is the case, it would certainly explain any color mismatches.

Also, since these tests are based on the standard scenario of simple BT.709 Y'CbCr data with no abnormal tagging or color spaces, it simply confirms that mpv handles the base case correctly. This doesn't rule out mpv doing something completely wrong when encountering nonstandard color matrices or tagging. Theoretically what should be happening for other color matrices like BT.601 is that the color matrix mpv uses internally converts this to BT. 709, which is handled normally from there. I'm not sure how correct these matrices actually are, since I haven't written or verified those myself. Maybe something about the color primaries or white points is not correctly adapted.

Can you confirm that you get the same values out of QTX when playing the .mkv I uploaded above, for starters?

[UliZappe]

Can you confirm that you get the same values out of QTX when playing the .mkv I uploaded above, for starters?

Unfortunately, I cannot even open this movie in QTX. It won’t accept movie files with 0 s duration. Can you provide a version with a duration of >= 1s?

[UliZappe]

Oh, and ideally in an mp4 container, not mkv. QuickTime cannot handle mkv, so I’d have to convert the container first, which would be an additional step for possible errors.

[UliZappe]

In the meantime, I tried what happens if I play video.yuv in mpv with the identical command you described above (the only difference was that I used the AdobeRGB1998.icc profile included with OS X).

The resulting raw RGB values on my screen (again measured with DigitalColor Meter) are:

154 255 74

In contrast, your values were: 160 255 87.

This means that again, mpv displays too dark an image on OS X.

[haasn]

This means that again, mpv displays too dark an image on OS X.

This is actually quite interesting. It might be the case that OS X is indeed messing with mpv's output. Unless the ICC profile is different for some reason (highly doubtful), I can't imagine there being a platform-dependent different in the actual rendering logic used by mpv.

This is where my knowledge of OS X and how it handles colors ends, unfortunately. I'm not going to touch any of the logic that will affect Linux or Windows, where the current behavior seems correct - not to mention not knowing what I'd change even if I were to fix something - but if you can pinpoint the issue and figure out why mpv's output is too dark on OS X specifically (maybe related to some Window flags?), that would be worth looking into.

Here's a longer, 1 second clip in an mp4 container: http://www.nand.wakku.to/video.mp4

mpv also outputs (161,255,87) for this test clip.

Edit: Oops, this clip and the .mkv file earlier were both encoded in Hi10P profile, which QuickTime seems unable to render - which makes it even more interesting that mpv rendered x264's output with the right values, since it means the 10-bit source -> renderer workflow works correctly, too.

I'll update the clips with 8-bit versions as soon as I can.

Edit 2: The video.mp4 link above is now 8 bit, and for comparison I've also uploaded an 8-bit .mkv file at http://www.nand.wakku.to/video8.mkv

[Pacoup]

Hmm, I'm trying to be very thorough this time, I really want to understand this issue, so it's taking much longer than expected to compile the results (e.g. I had to split the first test in two parts, go figure).

Anyway, I can't complete this tonight, so maybe tomorrow!

[4ad]

@haasn

With vo=opengl:target-prim=bt.2020:target-trc=bt.1886, black levels are good.

That should be a no-op on grayscale images.

Seems to work then (at least regarding black levels).

With vo=opengl:target-prim=bt.2020:target-trc=gamma2.2, blacks are not crushed, they are too bright though.

“Too bright” for gamma 2.2?

Not sure what you mean. Since black levels look ok with gamma=1.961, obviously they will be off with gamma 2.2.

So if the source->BT.2020 transform seems to be working, does it mean that the problem is with the BT.2020->target transform? (so in LittleCMS calls).

[haasn]

Not sure what you mean. Since black levels look ok with gamma=1.961, obviously they will be off with gamma 2.2.

What “looks ok” and what “is correct” are two different things. If I set my mpv to output linear light, and it's rendering correctly (i.e. “is correct”) then it will look terrible (i.e. “looks bad”).

The point is not that it should “look ok” but that it should output what you told it to output.

So if the source->BT.2020 transform seems to be working, does it mean that the problem is with the BT.2020->target transform? (so in LittleCMS calls).

I doubt the source->BT.2020 transform was ever broken to begin with, since that's extremely simple code (one matrix multiplication and two calls to pow). 90%+ of the complexity is inside the LittleCMS calls.

[4ad]

I see. When I said "looks correct" I meant "measures correctly in digital color meter".

So vo=opengl:target-prim=bt.2020:target-trc=bt.1886 produces 16 tones of black in the 0-16 range.

[UliZappe]

@haasn

We choose BT.2020 since it's the smallest space that covers all the content. BT.2020 and BT.709 are just linearly related, so any errors introduced should be within the margin of error of the quantization error * the expansion factor, which is low enough (1-2 at most) for this to be a negligible operation in theory.

Oh well, the famous difference between theory and practice ;–) . First, a deviation of 1-2 is not negligible from my POV if you convert twice and also have additional sources of error. Second, an example calculation (with ColorSync, though) for BT.709 gamma 1.961 RGB 6-6-6 to BT.2020 and back to BT.709 gamma 1.961 results in RGB 3-10-7, which is exactly the kind of problem we’re dealing with here.

So yes, this is most probably the culprit. BT.2020 obviously cannot be used as a kind of PCS.

In general, I think you should try hard to keep the number of conversions down to a bare minimum. Why introduce an intermediate space at all? It’s not that there are a terrible lot of video color spaces to deal with. Building two or three 3DLUTs tailored to the few video color spaces you’ll encounter will most probably be a clearly superior solution.

Just to be sure, I’ll also have a look at the other source files you mentioned.

[elmimmo]

Reading the code is beyond my skills and therefore also following exactly what you have tried, found out, and what you haven't.

But just to throw in an idea: might it have something to do by blacks becoming blacker than zero in the intermediate connection space and therefore being thrown out when converted from that to the destination space? See these articles about Working in bounded and unbounded color spaces (some of which are a bit hard to follow, but might be of help if the issue is related to that).

[haasn]

So vo=opengl:target-prim=bt.2020:target-trc=bt.1886 produces 16 tones of black in the 0-16 range.

With target-trc=gamma2.2 the tones 0-16 in the source image should be mapped to the range 0-21.6 in the output image.

So yes, this is most probably the culprit. BT.2020 obviously cannot be used as a kind of PCS.

“Patches welcome”

Also, @4ad said there's no difference before-after commit. Did you test it yourself?

[4ad]

With target-trc=gamma2.2 the tones 0-16 in the source image should be mapped to the range 0-21.6 in the output image.

They are mapped to 0-23.

With target-trc=bt.1886, they are mapped to 0-17, not 16. I haven't measured precisely enough before. So it's not exactly a no-op (albeit close).

[haasn]

They are mapped to 0-23.

I just thought of something: You could try getting a more accurate reading by enabling dithering (:dither=fruit:dither-depth=8) and sampling an average value over the entire region. (e.g. using GIMP's radius-based color selector)

[4ad]

vo=opengl:target-prim=bt.2020:target-trc=bt.1886:dither=fruit:dither-depth=8 gives 0-17 with sampling over region.

vo=opengl:target-prim=bt.2020:target-trc=gamma2.2:dither=fruit:dither-depth=8 gives 0-23 with sampling over region.

[UliZappe]

@haasn

It was introduced in this commit:
[…]
video: Add support for non-BT.709 primaries
[…]
Could try comparing before/after the commit.

I have now taken the source from my correctly working version and applied this commit (and the one before it, video: Add BT.2020-NCL colorspace and transfer function, a dependency).

As I expected, this introduces the issues we now have, i.e. the incorrect colors and the inhomogeneous dark patches.

So you need to revert this commit.

I would strongly suggest to not use another intermediate color space, but to avoid any intermediate color space and thereby additional color conversions (they always make trouble, not necessarily in theory, but definitely in practice …) and build dedicated 3DLUTs for all required video color spaces. That’s really no big deal, after all, and it will make for superior color reproduction and reliability.

[haasn]

If going for a “minimal loss” approach, one of the things we could do in theory is extract the transfer curve from the user profile and use that as the 3DLUT's input curve - that way the 3DLUT should be nearly-linear in terms of the overall transfer characteristics and would only need to encode deviations in color values.

We could also make a best effort to figure out the smallest space that overlaps both the video and the profile's gamut and use that as the connection space, that way the dynamic range would be maximized.

Either way, both of these require some significant rewrites to the littlecms/3dlut mechanism in order to generate the 3DLUT based on video-specific information.

[haasn]

By the way, the “generate 3DLUT against video profile” is how I originally wrote the commit that introduced non-BT.709 support: c85c2b1

But @wm4 didn't like the way the lcms code interacted with the video information (at least not in the way that commit was written), so I moved to the simpler “generate one 3DLUT per profile” approach instead.

[UliZappe]

If going for a “minimal loss” approach, one of the things we could do in theory is extract the transfer curve from the user profile and use that as the 3DLUT's input curve

What do you mean by “user profile” in this context? The monitor profile?

Anyway, I think the original BT.709 only implementation worked reasonably well, so “simply” recreating this original approach without an intermediate color space – but allowing for additional, non BT.709 variants of the 3DLUT – would basically be the way to go.

We could also make a best effort to figure out the smallest space that overlaps both the video and the profile's gamut and use that as the connection space, that way the dynamic range would be maximized.

Maybe, but I’m a bit sceptical about over-optimizations. There’s a reason ICC color management sticks to XYZ and Lab as PCS.

IIRC LittleCMS offers methods for creating device link profiles, which is basically what we create here if we create one 3DLUT per specific video profile – monitor profile combination. I just don’t know by heart if these methods support 3DLUTs.

By the way, the “generate 3DLUT against video profile” is how I originally wrote the commit that introduced non-BT.709 support: c85c2b1

But @wm4 didn't like the way the lcms code interacted with the video information

Ooops … too bad …

[ghost]

The problem is that this would have generated multiple 3dlut cache files, which all need to be managed, reloaded etc. - and in the end a simpler approach was found that didn't require this. So all was well.

So do you think this is absolutely needed for correct output, or are you speaking only theoretically?

[4ad]

@UliZappe

As I expected, this introduces the issues we now have, i.e. the incorrect colors and the inhomogeneous dark patches.

I will retry then, maybe there was something wrong with my settings. Did you use approx-gamma?

@wm4

So do you think this is absolutely needed for correct output, or are you speaking only theoretically?

Well the problem we see now is real and obvious. If the intermediary conversion is the culprit, than that is a very real problem, not only theoretical.

I will retry the before and after commits to check for myself.

[UliZappe]

@wm4

The problem is that this would have generated multiple 3dlut cache files, which all need to be managed, reloaded etc.

I see the point, but I don’t think you can avoid this.

• and in the end a simpler approach was found that didn't require this. So all was well.

Well, it wasn’t – color reproduction suffered.

So do you think this is absolutely needed for correct output,

Yes.

or are you speaking only theoretically?

No. On the contrary, theoretically the current approach sounds like a good idea. But color management practice shows again and again that you should avoid additional color space conversions as much as you can. Color space conversions are never lossless; there’s too many things that can go wrong, and some of them will go wrong.

Case in point: Even Apple, who rightly proud themselves of being the first who have implemented color management on an operating system level, experience issues with their (OS integrated) Core Image image editing technology which needs to convert image color spaces to a predefined, constant working color space, and although they have done a lot to optimize this, sometimes image colors are inadvertently changed simply by enabling Core Image (and thereby the working color space). This is an issue at the core of color management.

Simply spoken: We discuss this issue here for the very reason that practice has shown that mpv, which had already worked flawlessly, now distorts colors because of the intermediate color space approach.

[UliZappe]

@4ad

I will retry then, maybe there was something wrong with my settings.

I’m not even sure that a version similar enough to my working version is still available, or if this was just the one (unpublished) moment in time when everything worked OK. So it might well be the case that you cannot reproduce this. But my test result is unambiguous.

Did you use approx-gamma?

Yes, the versions at that time needed this to work properly.

[4ad]

So, here are the results.

I am using the iMac color profile. I am comparing four programs for black crush. QuickTime 10.4 (855), mpv eff2e57 (recent build), mpv 5b82983 (the before BT.2020 commit), and mpv 86d3d11 (the after commit).

For the recent build, I am using vo=opengl:icc-profile-auto.

For the before and after builds, I am using vo=opengl:icc-profile-auto:approx-gamma.

I am measuring both sRGB values, and native pixels, so we can do comparisons doing one less color space conversion (although, as you will see, for dark blacks they are the same).

We have 16 black bars, from bar#0 to bar#15.

We will be measuring bar#1, the first non-zero bar, the second bar; relatative to the colors row, this is the right bar above the white.

We will also be measuring bar#7 and bar#8. These are the 8th and 9th bars, the bars above the green.

We will be then measuring bar#15, the rightmost black bar.

Finally, we will be measuring the rightmost white bar, white#15.

So, here are the sRGB results:

Program	bar#1	bar#7	#bar8	#bar15	#white15
QuickTime	1-1-1	7-7-7	8-8-8	16-16-16	241-241-241
mpv (recent)	0-0-0	3-4-3	4-5-4	15-16-14	241-240-241
mpv (before)	0-0-0	4-4-3	5-5-4	16-16-15	241-240-242
mpv (after)	0-0-0	4-4-3	5-5-4	16-16-16	241-240-242

And here are the native results:

Program	bar#1	bar#7	#bar8	#bar15	#white15
QuickTime	1-1-1	7-7-7	8-8-8	16-16-16	241-241-241
mpv (recent)	0-0-0	3-4-3	4-5-4	15-16-14	241-240-241
mpv (before)	0-0-0	4-4-3	5-5-4	16-16-15	241-240-242
mpv (after)	0-0-0	4-4-3	5-5-4	16-16-16	241-240-242

I have also attached pictures, so you don't have to believe me.

Unfortunately, even the before BT.2020 version exhibits black crush.

@UliZappe can you upload the source code of your working version. I should test that here too.

QuickTime:

Mpv (recent):

Mpv (before):

Mpv (after):

[UliZappe]

@4ad

@UliZappe can you upload the source code of your working version. I should test that here too.

It’s too big to attach it here, and probably won’t work for you, anyway, as I customized several configuration files to adapt to my already existing installations of ffmpeg etc..

But there should be no need to duplicate efforts, anyway. I would suggest taking one step at a time, i.e. first, let’s rebuild the 3DLUT mechanism without an intermediate color space. Then we can see if you still experience the black crush problem.

From what I can see here, you will, as long as you use the iMac ICC profile. This profile does not strictly adhere to the ICC spec in that it is a v2 profile that nevertheless defines a parametric tone response curve which is only allowed for v4 profiles. ColorSync can handle this, but IIRC, LittleCMS cannot. That’s probably the root of your specific problems.

I have measured the dark patches in my working mpv version with several real world monitor profiles, including your iMac profile. The iMac profile did still exhibit the black crush, but it was the only one; all other monitor profiles worked fine.

[UliZappe]

The iMac profile did still exhibit the black crush, but it was the only one; all other monitor profiles worked fine.

Unfortunately, I have to correct myself in that I found several other monitor profiles which are correct v2 profiles but still exhibit the black crush. These are all LUT profiles (as opposed to matrix profiles). It seems that LittleCMS, at least the way it is used in mpv, cannot handle profiles with complex tone response curves gracefully.

[haasn]

Anyway, I think the original BT.709 only implementation worked reasonably well, so “simply” recreating this original approach without an intermediate color space – but allowing for additional, non BT.709 variants of the 3DLUT – would basically be the way to go.

To get some terminology clear, there still is an intermediate space here - that space is just BT.709 in your example. There has to be some intermediate space by design, because the 3DLUT is a function from one space to another. The former space is the “intermediate space” that we generate the 3DLUT against.

Maybe, but I’m a bit sceptical about over-optimizations. There’s a reason ICC color management sticks to XYZ and Lab as PCS.

Are you implicitly suggesting we generate the 3DLUT against XYZ or Lab? Because both of those would be possible.

Well, it wasn’t – color reproduction suffered.

It was working fine for me, which may be simply because I have a wide gamut profile that uses a pure power curve - both of which are ideal combinations for minimizing the error introduced by the change.

I am using the iMac color profile. I am comparing four programs for black crush. QuickTime 10.4 (855), mpv eff2e57 (recent build), mpv 5b82983 (the before BT.2020 commit), and mpv 86d3d11 (the after commit).

Can you repeat these experiments on a profile that uses a pure power curve? Since sRGB and “native” are so close on your device, it seems like your device is using an sRGB-like gamma curve. That might be a contributing factor to the black crush which I experience none of (using a pure power profile).

[haasn]

By the way, this issue is getting too large for my work PC to handle gracefully, and it's already marked as resolved, so let's open a new one.

[4ad]

@UliZappe

These are all LUT profiles (as opposed to matrix profiles). It seems that LittleCMS, at least the way it is used in mpv, cannot handle profiles with complex tone response curves gracefully.

Aha! well this is a very significant finding.

So, in the end, I think we're looking at two problems here? One is that the intermediate BT.2020 color space conversion introduces color distorsion. We can see this by measuring the colors with various profiles.

The other problem is that either LittleCMS itself, or the way mpv uses LittleCMS can't handle complex profiles properly. We can see this by measuring black levels with simple and complex profiles.

@haasn

Can you repeat these experiments on a profile that uses a pure power curve?

Yes, I can.

Note that I already did test the latest mpv with complex and simple power curve profiles already (mentioned a few times in this thread), and black levels are good with simple profiles (but colors are off). I will repeat experiments with before and after builds.

[jimmyd-sudo]

Please excuse me if this is a stupid question. The BT709 standard has a specific recommended gamma function, so media files that are generated with the BT709 typically have some metadata indicating to use that gamma transfer function. In practice, when converting across the color spaces BT601 (typically tagged with gamma 2.2) to BT709 (typically tagged with gamma around 2.45), does the gamma value specified in the input metadata change from 2.2 to 2.45 at the expense of different light intensity in the output file vs. the input file, or are is it preserved (which would result in a hybrid output file tagged with gamma 2.2 but encoded using BT709 matrix coefficients)? It doesn't seem possible to me to be able to both match the light intensity in the output with the input (which requires using gamma 2.2 in the output) and strictly following the BT709 guidelines (which requires using gamma 2.45 in the output), and I'm not sure which decision is considered best practice. Thank you for your guidance.