Implementing a contrast mask refinement

[jenshannoschwalm]

After closing #8501 this is a better approach ... pinging @aurelienpierre and @AxelG-DE

What is this about?
For some use cases we want to discriminate image areas with lots of detail from those without. So it is not the desire about border or shape detection but something that should be understood as looking for pixelwise local contrast.

This idea is used already in darktable's dual demosaicing code (to discriminate areas with high frequency content from those without) and some algorithms in rawtherapee (microcontrast, sharpening, capture sharpening ...) so we know it works really good.

The user interface is pretty simple, for every mask offering a refinement box there is an additional slider defining a threshold for the contrast mask. (details later) Positive slider values select for contrasty, negative values for flat regions.

What are the use cases of this contrast mask refinement?
We would like to use modules like sharpening, local contrast, denoising to work only on image areas most suitable. Or we can brighten up some areas with lots of details.
Also the contrast mask will be the basis of yet to be implemented modules, i will certainly port rt's capture sharpening to dt.

About the first pr discussion
As @aurelienpierre pointed out in the discussion about #8501, the underlying algorithm is a bit more difficult to implement with darktable's roi concept as the calculated contrast value is not reliable after scaling, rotation or even simple things as exposure corrections.
I tried his suggestions but the original algorithm seems to be better suited for the purpose described above.

So for calculating the first pixel contrast i kept the "weird thing" and had to find a way keeping "contrast masks" working exactly as in the original implementation. The "weird thing" code has been taken from rt, the originating code is from the microcontrast code there.

To get you into faster understanding the code, some implementation details.

Whenever we set this slider to a value != 0 the demosaic or rawprepare modules are forced to write a preliminary contrast mask, it always has the size of the roi_out of that module, it is not scaled but only clipped so it still contains pure pixel data.
This preliminary mask is calculated only once whenever the writing module is processed and is later used for every mask using a "local contrast" refining step.

Whenever a mask in any module wants a "local contrast" refinement it
a) takes the preliminary mask and calculates a 'mask overlay' using the threshold to calculate a sigmoid value 0->1 for every pixel
b) the just calculated mask overlay is now transformed through the pipeline using the modules 'distort' functions, the resulting "warped" contrast mask is now combined with the defined drawn or parametric mask.

• opencl code has been impemented
• some shared code with rcd and dual demosaicing has been de-duplicated
• supports all files using rawprepare or/and demosaic.

[aurelienpierre]

What exactly didn't work in my proposal ? I'm always suspicious when correct maths work worse than hacky algos without clear theoritical grounds.

[jenshannoschwalm]

What exactly didn't work in my proposal ?

I carefully read your comments and suggestions especially about the gradients ... absolutely correct. Of course they improved the mask output. But the generated mask still was very dependent on the scale in darkroom view (also we have scaling artefacts here depending on the used algo). It detected edges much much better (and your paper-in-pipe explained this is detail) but:

the question here is: how do we detect a small region (lets say 5x5 pixels) in the original image having local pixel differences / local contrast? As you pointed out the "weird thing" added gradients in a "hacky" way. Isn't this exactly why the used algo can detect such areas as is equally weighs those distances?

BTW i also looked at the focus peaking code as that seemed to be cosely related ... maybe i misunderstand the code but also here i see the peak-overlay depending on the image size.

So i think, the preliminary mask is good as it's a stable base for further processing so any modules switchend on/off before doing the contrast mask won't affect it.

[aurelienpierre]

BTW i also looked at the focus peaking code as that seemed to be cosely related ... maybe i misunderstand the code but also here i see the peak-overlay depending on the image size.

The focus-peaking is a vanilla second order gradient (laplacian), it's not exactly the same (gradients represent transitions, laplacians represents curvature).

the question here is: how do we detect a small region (lets say 5x5 pixels) in the original image having local pixel differences / local contrast? As you pointed out the "weird thing" added gradients in a "hacky" way. Isn't this exactly why the used algo can detect such areas as is equally weighs those distances?

OOOOOkay, I missed that. So the weird thing you compute is a patch-wise variance measure : sqrt(sum((average - patch samples)^2)). Makes sense. Variance is, indeed, much more immune to scaling than gradient (since downscaling is equivalent to a low-pass filter, it's only logical that gradients, being an high-pass filter, yield a different result). But you know what else thresholds variance to detect edges ? Guided filters… They are designed to avoid edges while blurring, but we could probably reverse that to tape them to edges. Also, they have very good scale-independent properties.

It would have saved us a lot of time if some background equations were provided with the code, or at least the variables in the code were properly named by what they actually represent.

[jenshannoschwalm]

It would have saved us a lot of time if some background equations were provided with the code

I will add such info. At least for me this discussion has not been wasted time :-)

[jenshannoschwalm]

CI failing is not related to this pr btw.

[AxelG-DE]

Just a quicky, as I am very busy these days/weeks/months... :-)

It is easy to handle. A combined drawn+parametric mask, where the drawn mask getting inverted, results in something unexpected (TBH I saw that before, so it is 99.99% nothing about this PR).

• what happens is, the local contrast threshold is getting applied inside the original mask, where I expect it to be applied to the inverted part.

But first things first:
This is a real use case for that local contrast mask from an Olympus EM5 MKII (contrastEQ focused but still off)

lct-mask:

Faked smaller DOF:

w/o lct-mask:

So there is an effect, which I am very grateful about (did not spend real time to attenuate)

And finally the inverted drawn mask together with the parametric one, wehere I expect the effect outside the circle and not inside, as it is shown here:

[jenshannoschwalm]

Just to leave a note, everything works just right now and i will re-open another pr after the rcd maintenance update has been merged for an easier review and testing in case the feature freeze for 3.6 comes too early.