vectorscope variant for histogram

[dtorop]

This is a first working pass at a vectorscope variant of the histogram:

Visible changes:

• A vectorscope is drawn in JzAzBz space (showing the a/b axes).
• The vectorscope is in grayscale, density corresponding to quantity of pixels of a given chromaticity.
• There is a "graticule" consisting of the primaries and secondaries of the current histogram profile.
• The color channel buttons are "insensitive" in vectorscope and (now) RGB parade views as it doesn't make sense in these views to only look at particular channels.

Changes to allow for this:

• The histogram button code has been revamped. Previously the "mode/type" and "view" buttons (the two leftmost) were implemented as toggle buttons. As there are now three modes (histogram, waveform, vectorscope), toggle doesn't work anymore. Clicking these buttons instead switches their icons and the state recorded in a data structure. This change shouldn't be visible to the user, but changed a bunch of code. The UI code should now be tidier (e.g. no stack needed to swap buttons anymore, CSS selectors are simplified).
• There is now CSS to set the secondary colors for the graticule (graph_cyan, graph_magenta, graph_yellow). The primaries use the extant CSS (graph_red, graph_green, graph_blue).

Some improvements to make after this PR:

• It is possible to make a "false color" variant which instead of being drawn in light gray, has each pixel colored according to their corresponding chromaticity values.
• I did a bit of optimization of the vectorscope process code, but it is still significantly slower than the corresponding histogram/waveform code. Here it takes about 0.040 secs. to process a vectorscope, compared to 0.002 sec. for histogram or 0.010 sec. for waveform.
• There is a bug which this PR doesn't fix: the tooltips of the buttons don't show except for just after the buttons are clicked. Otherwise the tooltip for the drawable below shows. This bug may have been around since GTK-ify histogram UI #7922.
• There are no concentric target-style circles from the graticule, as they didn't seem to refer to anything. It could be worth giving this more thought, though.
• It would be nice if the graticule actually showed some sort of gamut bounding line.
• As suggested by @sobotka (add RGB parade and vectorscope as histogram alternatives #4149 (comment)), it would be interesting to show other primaries in the graticule (input, working, output).
• The gamma/brightness math may need a bit further thought, and can probably be optimized as well.
• Work on the longstanding questions about how bright the various scopes should be, this included.
• Work on longstanding request so that colorpicker selection alters other scopes besides "regular" histogram.

Closes #4149, though it would still be good to refer to all the good ideas in that discussion.

[dtorop]

A few other notes:

The vectorscope is currently drawn to a pretty large (350x350 px x 1 byte alpha) buffer. This can probably be shrunk (and made a power of 2). But the buffer size doesn't make much of a difference for optimization.

The buffer is square, hence very out-of-gamut colors will visibly clip on the left/right of the display unless the right panel is very narrow. It's possible to fix this, but I'm keeping things simple for now. Here's an example of a clipped display:

The position of the "graticule" dots depends on the primaries of the histogram profile. So the image above is made with an sRGB histogram profile. For comparison, here is Adobe RGB:

And here is Rec 2020:

Also, in looking at the code, I realize that some of the waveform math can be moved from the drawing to the processing code. I added a comment about that, and will look into that after this PR.

[dtorop]

A few other notes:

• I didn't look at other vectorscope implementations -- excepting to try to orient the color axes in what seems to be a traditional way. I'm sure there are lots of good ideas out there... Much of this is implementing ideas/advice from @sobotka from the add RGB parade and vectorscope as histogram alternatives #4149 thread.
• The hope was to make something "darktable"-ish: minimal, with correct colorimetry, and real-time responsive.
• Speaking of responsive, there are no drag regions. I could see dragging (or scrolling?) to resize as useful. (The example images to generate screenshots above have extra-intense color, not all vectorscope graphs will take up so much space.) Or just keeping things simple.
• People have mentioned interest in a separate/pop-out/relocated scope window. This is definitely out of scope for this PR, but I understand that it's a pretty small graph. I'd be curious about the reverse -- a scope split into two panes, one showing a waveform, one showing a vectorscope. That should be straightforward to implement.

[sobotka]

This is some nice stuff. Perhaps my only thought is that it might be extremely useful to start from some of the more canonized plots such as 1976 perhaps?

Even if something like JzAzBz proved to be useful beyond the crazy behavior it can exhibit at points, the more “standard” plots are likely hugely valuable?

[dtorop]

This is some nice stuff.

Thank you!

Perhaps my only thought is that it might be extremely useful to start from some of the more canonized plots such as 1976 perhaps?

It should be trivial to do that, either as a view option via button (just to see the difference for now), or just to switch over. And it might even be faster! I'm happy to try this, for the sake of experimenting. I'm seeing this PR as a way to get baseline usable code (which would include a sensible colorspace), and there'll need to be follow-up refinements.

Even if something like JzAzBz proved to be useful beyond the crazy behavior it can exhibit at points

I didn't know about this! Are there good examples? I'm assuming crazy not in a positive sense.

[sobotka]

Because it is a nonlinear encoding, as values go downward things can go completely bonkers, especially in the case of non-physically plausible primaries. I think it goes bonkers high too.

On the other hand, 1976 is a pretty decent baseline, and has familiar patterns such as where a ColorChecker 24 ends up etc. Easier to plot gamut triangles on as well!

[dtorop]

On the other hand, 1976 is a pretty decent baseline, and has familiar patterns such as where a ColorChecker 24 ends up etc. Easier to plot gamut triangles on as well!

This makes a lot of sense. I added a commit with CIELUV space. It looks much better to me. Right now there's a button (with dummy icons) to switch between JzAzBz and CIELUV, but I wonder about dropping JzAzBz altogether. Also, it's over 2x faster to calculate in CIELUV (about 0.015 sec. vs. 0.040 sec. for JzAzBz).

There are some other questions, for the future...

• I'm finding primaries (and secondaries!) by feeding (1,0,0) etc. back through the profile matrix, as there seems to be no guarantee that they're in a chromaticity tag in the profile. This seems to work (tested at least for primaries), but also seems odd.
• If I have primaries in CIE xyY and connect the three x y coordinates, is this a correct gamut triangle? How does this distort when drawn in CIELUV?
• In most cases, I'm converting pipeline-profile RGB to histogram-profile RGB (w/relative colorimetric intent, via matrix) to XYZ. How is this different from skipping a step and converting pipeline-profile RGB straight to XYZ?
• New code is using CIELUV (u* and v* values). This compensates for whitepoint, and it seems important to center "neutral" in the scope. An alternative is to work with u' v' (or u' v' adjusted by whitepoint). Saves math, but...

[dtorop]

I added "icons" to differentiate CIELUV and JzAzBz plots, and re-re-oriented the JzAzBz plot to be as in the screenshots above. I do realize that at this point adding a CIE 1931 xy chromaticity view probably makes sense, as it's another well-known way to see color...

Also probably worth thinking about whether to scale the vectorscope to fit the histogram primaries (as now) or use an "absolute" scale if this is a "canonized" plot.

[dtorop]

I'm seeing an OS X build error:

2021-01-27T15:24:50.7436380Z /Users/runner/work/darktable/darktable/src/src/dtgtk/thumbtable.c:1163:5: fatal error: implicit declaration of function 'dt_osx_disallow_fullscreen' is invalid in C99 [-Wimplicit-function-declaration]
2021-01-27T15:24:50.7437650Z     dt_osx_disallow_fullscreen(dialog);
2021-01-27T15:24:50.7438020Z     ^
2021-01-27T15:24:50.7438330Z 1 error generated.

Somewhat mystified as to what this means...

[parafin]

It's from another recently merged PR. Really PRs shouldn't be merged with CI failures, that's the whole point of CI...

[dtorop]

It's from another recently merged PR. Really PRs shouldn't be merged with CI failures, that's the whole point of CI...

Ah! Good to know.

I see that 4f0b668 added the suspect line src/dtgtk/thumbtable.c:1163. But this branch vectorscope-2021 is based off of git master before that (d50e30b) and doesn't include that commit. Is something funny with CI that it is pulling in the wrong code to test?

[sobotka]

Nice work Dan.

It might be worth drawing the spectral locus on 1976, given it holds meaningful horizon limits. Colour coding the locus?

If I have primaries in CIE xyY and connect the three x y coordinates, is this a correct gamut triangle?

Correct.

How does this distort when drawn in CIELUV?

It’s one of the nice sides of 1976 is that it has a direct perceptual-like connectivity to 1931 / 2012; the spectral locus remains intact, as does any area gamut triangle.

Be warned that the canonized CIE XYZ to xyY and likewise XYZ to 1976 Luv will fail with many contemporary camera virtual primaries due to negatives. To properly account for negatives, use the absolute, and return the sign after calculations.

[dtorop]

Thank you, helpful... It's nice to see the CIE 1931 xy plot with the spectral locus -- and indeed the xy coordinates do make a nice gamut triangle. I also appreciate that every bit of color math/data I need is already somewhere in the darktable source.

It's nice to see how clipping works in the top image (w/Adobe RGB profile, in this case).

These screenshots have a dot for the white point made by feeding (1,1,1) through the profile matrix to PCS. Unfortunately (presumably because that always emerges in D50) the white point is always calculated as D50, even with ProPhoto. So one piece is missing, and hence the graph is still suspect.

[sobotka]

Given you selected 1931 (2012 should also be an option, which I suspect @aurelienpierre may have added and has the corrected response in shorter wavelengths) it is worth putting an extremely wise quote from one of the colour deities here. From Giorgianni and Madden’s canonical tome:

Odysseus had to go to great lengths to keep his sailors from being bewitched by the sirens. Similarly, people who want to specify color appearance with numbers must be wary of chromaticity diagrams with lovely colors on them. Such illustrations as this tend to imply (or the viewer tends to infer) that a particular perceived color is associated with a particular point on the diagram. This is false!

The chromaticity diagram is a stimulus specification. A given color perception can be elicited by a stimulus represented by just about any point on the chromaticity diagram, depending on the nature of the visual field and the viewer’s state of adaptation.

Although a colored rendition of a chromaticity diagram does, perhaps, help to indicate to a novice that with a common illuminant, blue paint will have a chromaticity in the lower left, etc., that novice is likely to jump to all the wrong conclusions and consequently find the world of color very confusing.

In short, the uses of such illustrations probably should be confined to making lovely wall decorations in places that never will be visited by budding color scientists and engineers.

[dtorop]

In short, the uses of such illustrations probably should be confined to making lovely wall decorations in places that never will be visited by budding color scientists and engineers.

Ah, but imagine seeing a wall decoration with color values from your very own image included on a canonical graph! But yes, it's a good caution...

[dtorop]

This is growing in scope (really, no pun intended), but perhaps the way to go is to split this into two varieties new modes:

• vectorscope: This allows for a sense of the motion of chromaticity in the direction of various primaries. This could be drawn in terms of u*v* of CIELUV, AzBz of JzAzBz, or other spaces of interest. It shows primaries/secondaries of histogram profile, and is scaled so that the plot fits within them. This is centered on something white-point-ish. Code for u*v* and AzBz is already in this PR and should be OK.
• chromaticity diagram: This gives a sense of the image in an "absolute" colorspace, and helps understand gamut/clipping -- and colorchecker swatches. It includes the spectral locus horseshoe and gamut triangle(s) (at least for the histogram profile, ideally for input/work/output profiles as well). The full plot is always displayed (no scaling). This is drawn in terms of CIE 1931 x y, CIELUV u'z', etc. The CIE 1931 code is in this PR, but I still want to think it through more. This could be split out into a follow-up PR.

A hope of these plots would be to give users a way to understand what sort of math/model is working on their image, not least with some of the newer tools (thinking of color balance rgb...). As an example: The curves tool has been so important not just because it runs fast and allows for subtle changes, but also because of how it can be visualized by a histogram -- such that histograms are often even drawn as part of the tool. Even if we're cautious about the scope being 1:1 with the image, it at least has a close relationship to tools which work on the image...

[Nilvus]

Just checked css code, of course, and all good.

Tested this PR and all good for me. I like that and buttons are better and consistent now. Good work @dtorop.

I let other developers check the code!

[dtorop]

@Nilvus Thank you for looking it over, and glad for the positive report.

I think the code as-is is fine, with the code master..c1d1814 being the most stable.

The two following (and most recent) commits add a new feature (chromaticity diagram with gamut plot) which still needs a bit more fleshing out and maybe some color math fixes. I'm not sure if I should roll back those two commits from this PR and work on them separately, or if they could be part of this PR with the caution that they're not-yet-for-precision-work.

[dtorop]

This PR should be ready for further review. I rolled back two commits to get rid of the chromaticity diagram code, and added a commit with cleanups. The main user-visible change is that the cross-hair in the center of the vectorscope is brighter:

I've some mixed feelings about this, as I don't want to make this seem a tool to get neutrals to a particular target...

I'm also referring to the two views as u*v* and AzBz (formerly Luv and JzAzBz) to clarify that this a plot of chromaticity.

I'll work on another branch to get the chromaticity diagram code right, but didn't want to be endlessly fiddling with this PR.

[johnny-bit]

Just ping when it's ready for review and one of janitors will remove "wip" label :)

[dtorop]

@johnny-bit Thank you! Yes, please do remove the "wip" label, whomever might do such work!

(I think I finally cracked the whitepoint problem to make a "correct" chromaticity diagram. But I'm working on that in another branch. There's more than enough in this PR to look at, between UI code changes and adding the vectorscope view.)

[chhil]

Great work Dan, thank you for bringing this functionality to DT.

Would it be possible to get the skin tone line into this?

It would be great to try this with a colorchecker card.

[dtorop]

Great work Dan, thank you for bringing this functionality to DT.

Thank you! Still not merged -- or even fully review -- of course. This PR makes for a relatively big diff.

Would it be possible to get the skin tone line into this?

It's certainly a recurring discussion. See #4149 (comment) and #4149 (comment). It's deeply tricky territory: doesn't seem supported by color science, biology (is dermatology biology?), can get into rough political/cultural territory fast, and besides all that goes against what makes these scopes actually interesting.

The more I get into this, the exciting thing about these scopes seems to be a visual representation of the mass of #'s which iops (especially recent additions) work on, which gives another angle onto what these tools can do, and how they can do the work.

It would be great to try this with a colorchecker card.

Yes... See screenshots at #8005 (comment) (from a lousy JPEG), though the real interest will be to see it in a chromaticity diagram (not scaled to histogram profile) such as the second image in #8005 (comment). That code isn't in this PR, though, still working on getting into right. (It probably goes without saying, though, that the "calibrate with a color checker" section of the new "color calibration" iop appears to be pretty amazing and may subsume any external/visual tool.)

[todd-prior]

For what it is worth this is what RT uses currently.....shown below
Is the vectorscope in the master at all. I did one the other day and it was not included??
Hue Chroma

Hue Sat

[dtorop]

Tested today. Working as expected.

Great! Thank you for testing.

The vector scope in many images I have tested (mostly portrait where colors are important) the displayed data are actually taking a very little part of the center of the scope. So it is hard to guess where the colors are, I would have expected this vector scope to better sustain the color grading step in dt.

It's a fair point. Histogram profile is also a factor -- a smaller gamut histogram profile such as sRGB will show more range to the graph. Sometimes the lack of deflection of the graph from the center actually is the useful piece of information, not least if the goal is to make something neutral.

This might be a usecase for a scalable graph.

I think so. It shouldn't be hard to implement. It's been on my list, so I'll try this.

some kind of log scaling there to make the center area far more readable

I've been trying to think this out. My note on it back #8005 (comment) was: "The trick with log scale for radius is it edges into re-thinking CIELUV or JzAzBz...." I'm still not sure on this, but I'd like to try user-controllable (or auto?) scaling before shifting things to log.

Here are some examples of what I see here with the current code and histogram profile in sRGB (which does exaggerates chroma).

A photograph including an x-rite colorchecker:

A landscape (grass, blue sky):

And here are a couple which is pretty much lack saturation/chroma:

[dtorop]

Looking at the screenshots above also reminds me that I should lighten the backround, as per #8005 (comment):

just ligthen a little black background (in darktable.css file, line 153, just set graph_bg to grey_15 instead of grey_10

[dtorop]

Thinking about this a bit more. The "regular" histogram already scales the y axis automatically, and it's pretty seamless/natural. Perhaps that can be a guidepost for this. I'll work on it a bit.... I wouldn't want to lost the visualization of "maximum" chroma for each hue, though.

[dtorop]

More updates, to try to address #8005 (comment) from @TurboGit:

The vector scope in many images I have tested (mostly portrait where colors are important) the displayed data are actually taking a very little part of the center of the scope. So it is hard to guess where the colors are, I would have expected this vector scope to better sustain the color grading step in dt.

The vectorscope is now scalable via shift-scroll or shift-drag (reset via shift+double-click). I thought about making "auto" scaling by always sizing the graph to fit, but it seems good to keep manual control so that the scale doesn't jump around from image to image. It wouldn't be hard, though, to convert this code to "auto" or add an "auto" option. It would also be possible to make the scope of higher resolution so it offers more detail at high scales.

I also updated the vectorscope drawing code. The lines should be a bit more crisp (drawn in whole-pixel units rather than depending on a ratio).

I realized that for all scopes (histogram, waveform, vectorscope) the border was never drawn, so I fixed this. I'm actually not sure how much I like it with the border -- this may be something good to turn off.

I updated the background of scopes in the "darker" theme from grey_10 to grey_15 as per #8005 (comment) from @Nilvus. This affects all the scopes (histogram, waveform, vectorscope) as well as graphs in various iops: tonecurve, filmicrgb, toneequal, and colorbalancergb.

[TurboGit]

@dtorop : I'm still not 100% convinced by the auto-scaling which most of the time makes the outer colored circle (gamut) outside of the view and so not visible except for some little parts. But I think this can go as-is now. So if you have no more cooking changes on your side I'll merge today for more field testing. Is that ok with you ?

[MStraeten]

the autoscale is ok, but in my opinion now the ability to rescale to see also the border is lacking. Reenabling the manual scaling mode might be a solution.
maybe for a polished user experience it's useful to move the center on demand

[TurboGit]

And why not a log scale as proposed for standard histogram? A manual scaling will probably require adjustment for each image.

[MStraeten]

my UseCase: In general the full view is ok to get a overview if colors seems to be ok - especially skin tones.
If i have an image that just use colors in a part of the scope i want to magnify this part maximizing the graph area to display this.
workflow aspects: if switching to a new image i expect the full view of the scope, but then be able to zoom and pan.

[dtorop]

... I think this can go as-is now. So if you have no more cooking changes on your side I'll merge today for more field testing. Is that ok with you ?

I'm happy if this could merge, and then continue to work to improve! The code should be workable/stable. For reference, things on my mind still:

• Not losing hue legibility when scaled such that hue ring is out of view (see below).
• Manual control of scale/panning (see below).
• I fear there's a race in the binning. Worst case is that the graph will be slightly wrong, and some pixels of the same hue/chroma will be undercounted. Still good to fix.
• Is there a better way to display color on the graph? See vectorscope variant for histogram #8005 (comment).
• Do something nicer when colorpicker is active in area mode -- show graph of whole image with less alpha, graph of selected area with full alpha (similar to current point colorpicker behavior).
• Optimize graph generation and colorspace code. This may also include caching hue ring data per profile.
• Think out memory usage and graph/color resolution -- is the current code the best balance?
• Display spectral locus as well, to help understand how hue ring sits within it. This will require getting some things with white point right.
• Display primaries/gamuts of input/working/output colorspaces. See add RGB parade and vectorscope as histogram alternatives #4149 (comment).

All of these can be subsequent work -- take this list as a roadmap for that.

I'm still not 100% convinced by the auto-scaling which most of the time makes the outer colored circle (gamut) outside of the view and so not visible except for some little parts.

Yes, losing the outer colored circle is a worry. The hope is to make a scope which "just works" -- for neutral images, the color of the graph and the concentric circle scale are adequate to understand hue/chroma, for more intensely colored images, the hue ring will be a further cue. The outer hue ring does provide a gamut warning, but not a great one, as depending on the lightness of an image color, it can be drawn within the hue ring but be out of gamut. Perhaps one of the inner concentric scale circles could show hue gradations as well, to aid legibility?

Reenabling the manual scaling mode might be a solution. maybe for a polished user experience it's useful to move the center on demand

I'm open to a manual mode. I'm also happy if that's a further PR, to move this along.

workflow aspects: if switching to a new image i expect the full view of the scope, but then be able to zoom and pan.

The alternative would be to save the vectorscope scale/pan (for both Luv and JzAzBz mode!) in conf, so it's extra "sticky". Is there a case for someone who wants to work on multiple images viewing the same scope scale/pan?

Maybe the scheme would be on new image to "auto" zoom. Then shift-drag can move center, shift-scroll can scale. (But does everyone have a mouse/trackpad which can scroll? Does there also need to be something like ctrl-shift-drag to scale?) Shift + doubleclick returns to auto mode, or maybe cycles: auto fit image graph -> manual -> auto fit hue ring.

Perhaps the scale mode would be sticky between images, so it's possible to set up a scale/pan which is consistent between a series of images, but in auto mode, scale would remain auto between images? There are also three unused buttons in the top right, one of them could do this work.

I also don't want to over-complicate the UI if there can really be a "just works" solution instead...

[MStraeten]

The alternative would be to save the vectorscope scale/pan (for both Luv and JzAzBz mode!) in conf, so it's extra "sticky". Is there a case for someone who wants to work on multiple images viewing the same scope scale/pan?

Maybe, i'm also fine with keeping this setting in the session but doubt that needs to be persisted in darktablerc. Then a quick reset to full centered view might be a solution.
But thats better to decide if there's more experience with using vectorscope by a broader group, so strong vote for merging.

so my opinion on the UX:

• ctrl+scroll to increase decrease the vectorscope/histogram (as is and also implemented for several graph areas in modules)
• shift+scroll to scale (it's shift double-finger-drag on my trackpad; i'm pretty sure other os have similar gestures for scrolling) maybe also shift+drag
• click+drag to move center (quite common, dragging around mask forms is possible this way with a trackpad on my macbook)
• doubleclick to rest to centered full range

[TurboGit]

Ok, so let's merge this now and continue improving all this in separate PR. I'm sure field testing will bring some additional ideas.

[dtorop]

Ok, so let's merge this now and continue improving all this in separate PR.

Fabulous! I'll work on interim documentation.

I'm sure field testing will bring some additional ideas.

Do tag me @dtorop or link to this PR if (when...) there's a bug or feature request, as I may miss that otherwise.

• ctrl+scroll to increase decrease the vectorscope/histogram (as is and also implemented for several graph areas in modules)
• shift+scroll to scale (it's shift double-finger-drag on my trackpad; i'm pretty sure other os have similar gestures for scrolling) maybe also shift+drag
• click+drag to move center (quite common, dragging around mask forms is possible this way with a trackpad on my macbook)
• doubleclick to rest to centered full range

This all makes sense. I'll give this a moment to see if there's a way to make "auto" is sufficient, but it should be trivial to bring the manual code back in.