Random lilac blown highlights

[nugecom]

Dear darktable developers,

I couldn't find this issue in previous reports so here goes:-

• Sony A7iv camera
• Uncompressed RAW files (.arw)
• darktable: 4.0.0
• Ubuntu linux 20.04

Some of my photographs are showing blown highlights as a 'lilac' colour, rather than the white I would have expected.

This can be corrected with a mask but often not easily. I cannot find a common factor that links the photographs where this happens - it seems to happen randomly. The camera's JPG thumbnails do not show this issue - they have white blown highlights - so I assume the camera is not causing the issue.

I have tested the same images in RawTherapee and the blow highlights are white, as expected - so I think the issue seems to be with darktable. I have also converted the ARW files to DNG and the same issue happens in darktable.

Here is an example photograph - where you can clearly see the issue on the left arm of the middle walker:

To be clear, most of my blown highlights are being shown white in darktable. The issue only effects 10-20% of the photographs that have blown highlights.

Is this a bug that can be fixed?

Thanks for all you do to make darktable such an amazing tool!

Hopefully,

John

PS. You may advise that I should not blow my highlights... but that is a separate issue! ;)

[kmilos]

Possibly related to changing white levels w/ Sony cameras: darktable-org/rawspeed#349

If so, no need for masking, just reduce the white level a bit in the rawprepare module.

See also many recent topics about highlights reconstruction on https://discuss.pixls.us/

[nugecom]

Thanks kmilos.

I don't understand the discussion on changing white levels you linked to - and couldn't find anything more comprehendible elsewhere.

I can't find a rawprepare module in the modules or settings. However, my searching lead me to discover that it is the 'filmic rgb' module which is creating the lilac blown highlights. If I click the module below filmic in the history stack (i.e. filmic is not applied to the image) the lilac blown highlights become white, as expected.

Can I make a permanent change to the filmic rgb module that would stop the lilac on all future blown highlights?

If not, what should I adjust in filmic to safely fix the issue on a picture-by-picture basis?

Thanks for your help!

[gi-man]

The module you most likely need to tweak is Raw White/Black point. One of the first things to process an image is to define the white and black levels of the raw data. It appears that for your camera the white point is too high, so darktable thinks it is not blown but it really is.

Darktable uses rawspeed to read the white point and it seems that it is incorrect for your camera/iso. Rawtherapee uses a JSON file with the values to use, hence the difference.

[MStraeten]

However, my searching lead me to discover that it is the 'filmic rgb' module which is creating the lilac blown highlights.

filmic isn't creating the blown highlights it just makes them appear since it maps the input tonal range into the much more limited output tonal range without desaturating extreme highlights.
have a look at https://www.youtube.com/watch?v=z-zVMHKTBBY for a goo explanation and how to cope with that effect

[ralfbrown]

Also note that, if you are using the "modern" chromatic adaptation workflow, this violates an assumption made by the older code in the highlight reconstruction module, which is that the final white balance has been set on the data it receives, so that 1.0/1.0/1.0 is pure white. This affects in particular the "clip" method and to a lesser extent "LCh". Since in the modern workflow, we WB for D65 before Highlight Reconstruction and then adjust for the correct WB in Color Calibration, the clipped 1/1/1 usually gets a cyan cast since most images have a color temperature lower than 6500K.

[jenshannoschwalm]

Also note that, if you are using the "modern" chromatic adaptation workflow ....

Absolutely. Unfortunately we currently have tons of wild speculations about this on pixels.us

[PeterWem]

Any difference if you set the same white point as the white level found in its Exif?

[jenshannoschwalm]

Yes - if set to "as shot" it helps for highlights reconstruction - unfortunately that's bad for the rest of the pipeline so not a good option

[PeterWem]

Not what I meant, but that was also good to know.

[nugecom]

Thanks all.

Playing with the 'raw black/white point' had no impact on the issue.

However, reducing the 'filmic rgb / reconstruct / highlights clipping / threshold' suddenly fixed the issue at a value of -1 EV versus the default +3 EV. This change was very similar (i.e. -4 EV from the default) on all the images with the issue.

I then watched the video link (thanks) which confirmed this solution and explained the colour cast.

However, I am still unsure why my camera's images are only sometimes having this problem. I see no commonality in the histograms or ISO.

I took a look at: https://rawspeed.org/CameraSupport.html and it seems my camera (Sony ILCE-7M4) is not supported yet despite launching in Dec 2021. I guess this issue might be related to that? I'd be curious to understand what determines their timeline for support.

Incidentally, Darktable's use of rawspeed also, I think, sheds light on why my camera's 'lossless compressed' file format still isn't supported in Darktable - my first question to this forum.

I hadn't considered the post-production workflow support when buying the camera. I wonder what other deficiencies my processing might have given the lack of support? I have noticed a tendency for colour casting that effects the whole image e.g. with green vegetation and green skin (not reflection). Also a tendency for greyish skin tones for no obvious reason. I moved from a Nikon D750 and thought these might be Sony's colour science at play.

Thanks again for everyone's help on this. I now know how to implement a quick fix - and look forward to my camera being fully supported (I hope) in due course!

[jenshannoschwalm]

The color casting issue related to highlights reconstruction & white balance is my main concern and topic of further work for 4.2 - stay tuned!

[PeterWem]

7M4 is supported in Rawspeed.

[nugecom]

If the Sony A7M4 is supported in rawspeed, does that mean it's still on the backlog for full darktable support? If so, is there any idea of timeline? What does 'support' in darktable even mean?? I'm keen for ARW lossless compression files to work - what else should I be aware of that doesn't work in the meantime?

[gi-man]

If the Sony A7M4 is supported in rawspeed, does that mean it's still on the backlog for full darktable support? If so, is there any idea of timeline? What does 'support' in darktable even mean?? I'm keen for ARW lossless compression files to work - what else should I be aware of that doesn't work in the meantime?

It was already linked above that there is an issue with the support of sony raw files and white levels. There is a suggested fix for it. But that other project rawspeed (not darktable), needs to incorporate the fix. The timeline is just whenever the maintainer gets to it.

[github-actions]

This issue did not get any activity in the past 60 days and will be closed in 365 days if no update occurs. Please check if the master branch has fixed it and report again or close the issue.

[Darkdriver84]

I figured out that i may help to set in the WB module "as in camera" instead of the reference point and then reconstruct the hightlights in LCh. Anyway...very annoing issue. I am also having the Sony A7IV.

[nugecom]

Rawspeed maintainer lebedevRI claims the highlights issue has been fixed but not yet incorporated into darktable. Lossless compressed format support fix is also in his backlog.
darktable-org/rawspeed#405 (comment)

[Darkdriver84]

Rawspeed maintainer lebedevRI claims the highlights issue has been fixed but not yet incorporated into darktable. Lossless compressed format support fix is also in his backlog. darktable-org/rawspeed#405 (comment)

Hope it will be in the next darktable update, which should come around christmas.

[starsforeveryone]

Hope it will be in the next darktable update, which should come around christmas.

Yes, that would be awesome!

Support for ILCE-7M4 (Sony A7 IV) - including all pending bugfixes & support for lossless compressed RAW format - would be a really nice christmas gift 🥲

If I can help in any way, let me know! I have the camera and can test.

[github-actions]

This issue did not get any activity in the past 60 days and will be closed in 365 days if no update occurs. Please check if the master branch has fixed it and report again or close the issue.

[LebedevRI]

This should surely be fixed by now.