New shadows/highlights implementation

[agriggio]

The current shadows/highlights tool is really not my cup of tea. I never managed to get good results with it. So, I started experimenting with an alternative implementation. I kept the same UI (except for the "sharp mask" checkbox, which is now useless so it's gone) but I changed completely the implementation. I actually have no idea how the old code worked, I developed the new tool completely "from scratch" (but it's really simple, so probably it's nothing new).

This PR is a proposal to get rid of the old implementation and use the new one. However, since this is not backwards compatible (the results are quite different with the new tool), I won't fight for it -- if you prefer to keep the old tool, no problem, I will keep this in my own (private) fork of RT.

Anyway, here is a little demo for the interested people (all images start from the "Auto Matched Tone Curve - ISO low" profile, and I only play with the new S/H tool).

https://filebin.net/jivbz36z4mn8aze6/sh.mp4

[Beep6581]

The results look good, looking forward to trying it.

Could you explain how it works?

[agriggio]

I'll try to post a video of how to do more or less the same with GIMP (using layers and masks, and working in lab mode). the code basically automates the manual way (plus some tweaks here and there to try to keep artifacts under control)

[TooWaBoo]

like it 👍

[agriggio]

Here is an explanation video and a raw to play with:
https://filebin.net/3dzjlx4zqkz1f8jy

[gaaned92]

I was never able to get acceptable result with previous S/H tool. For what I tested this version is better

[aferrero2707]

I have added the new-shadows-highlights to the list of branches for which an AppImage gets automatically built. The new package is here.

Hopefully this will allow more people to test this cool feature!

[heckflosse]

@agriggio I found an inconsistency:

Apply neutral profile: Radius of Sh/Hl is set to 40
press the reset button for the radius: Radius is set to 30.

[agriggio]

@heckflosse thanks for the review! about the reset bug, I thought I fixed it already but perhaps I forgot to port it from my playground to the branch... I'll check again

[Beep6581]

PPVERSION should be bumped.

The last three dates in ppversion.h use the wrong format, @agriggio could you correct that? If I correct that in dev, there will be a merge conflict. Should be YYYY-MM-DD.

Looking forward to the merge!

[heckflosse]

@agriggio I prepared a speedup for the generation of the LUTf f(32768). Currently it takes ~6 ms to build it ( * 2 if shadows and highlights sliders are != 0), which doesn't sound very much, but for thumbs generation and panning in editor it's quite much. I managed to get it down from ~12 (2*6) ms to ~2.3 ms for the case multiThread == false (which is the case for thumbs). For multiThread == true it's less than a ms now, but that's not so important.

[heckflosse]

@Beep6581 @agriggio @Hombre57 @Desmis @TooWaBoo @iliasg @Floessie @gaaned92 @...

As the new SH/HL tools breaks backwards compatibility, I think we need an agreement.

My vote is to replace the old SH/HL tool with the new one.

[heckflosse]

@Beep6581

The last three dates in ppversion.h use the wrong format

Sorry, my fault as well...

[agriggio]

@heckflosse Ingo, feel free to push your fixes and speedups directly 👍

[Desmis]

No problem for the new tool :)

[Floessie]

I didn't have time to test, so I'll abstain.

[heckflosse]

@agriggio Done! Also merged dev into this branch.

[Beep6581]

Would it be possible, and would there be any interest, in adding a way of controlling what is considered a shadow and what a highlight?

For example, I want to see inside the engine, but not touch the red paint which is already near clipping:

http://rawtherapee.com/shared/test_images/tracteur.dng
tracteur.dng.pp3.txt

This is only a question/discussion, not a feature request yet, because HDR Tone Mapping lets me do just that:

[agriggio]

@Beep6581 you can try playing with the "tonal width" sliders to restrict the effect (pull down for shadows, or push up for highlights). However, this picture is particularly tricky as the shadows have very sharp edges, and it's very easy to get halos with the gaussian blur that is currently used.
Maybe we could consider using a bilateral filter, though I'd have to play with it first to see if I can understand how to use it... :-)

[kazah7]

I don't know if it's helpful or not, but just in case i'm uploading a result from lightroom + clip in the link provided. If you want me to process some more photos this way (for references purpose) let me know.

clip: https://filebin.net/ovk1xywbbw67ek9y

[Beep6581]

Thank you @kazah7 , nice and smooth.

[agriggio]

thanks @kazah7, I think the hdr tone mapping tool could help in getting close. I'll try later if I find some time...

[agriggio]

Here's my attempt at a lightroom-like result. As anticipated, I used the HDR tone mapping tool, not the S/H tool here:

[gaaned92]

And my attempt
It's a Massey-Harris. Cannot read the model. It seems to be a 820D that was in my grandfather farm.

tracteur.dng.pp3.txt

Using HDR tone mapping with additionnal contrast.

[agriggio]

Hi all, is there anything missing before merging this? Is the current quality/functionality good enough? One thing that comes to my mind is how to deal with pp3's that were using the old version of the tool. Given the large differences between the two outputs, my proposal would be to simply disable the new S/H tool for old pp3's. What do you think?

[Beep6581]

No foreseeable requests from me, looking forward to the merge.
Agreed about disabling the tool when loading old PP3s which use the old S/H.

[heckflosse]

Should we add some default settings for sh/hl? Currently, when a user enables the tool, nothing happens, because the settings are zero. Just an idea... I know it was the same with the old sh/hl tool

[agriggio]

I think the defaults for radius and tonal ranges are reasonable. regarding the amounts, I can't think of anything sensible that will give good results across different pics -- it's just too dependent on the contents of each pic IMHO

[heckflosse]

ok

[okoncevoy]

I have a question on those openmp usage. Is is, that with current implementation, each call for omp parallel for creates a bunch of new threads without any limitations? As I remember, openmp on *nix does not use any thread pool for execution. Might be that this is not a problem, but it can be solved with thread teams.

[heckflosse]

@lvreclp

each call for omp parallel for creates a bunch of new threads without any limitations?

Usually the number of threads is limited by the number of cores on your machine.

Can you be a bit more specific about the problem?

[okoncevoy]

@heckflosse
I had an issue, that I needed to debug software that uses openmp for parallel processing of image, and it created more than 12k threads, like one thread per for loop iteration. So as I know you can limit thread count by calling omp_set_num_threads() and available thread count can be get by std::thread::hardware_concurrency(), if using c++.

Or you can limit thread count by using next construction: #pragma omp parallel for num_threads(4)

[heckflosse]

@lvreclp

maybe OMP_NUM_THREADS environment variable was set wrongly?

https://gcc.gnu.org/onlinedocs/libgomp/omp_005fget_005fnum_005fthreads.html