negadoctor invert mistake

[radioproektor]

Hello. There is a formal mistake in negadoctor Invert logic.
Currently invert goes inside working color space and depending of working color space size vs input color space size this cause saturation clipping and hue shifts. (Saturated Yellows shifts to Green, Reds shifts to purple. Larger working color space - stronger color shift)
From my experiments invert should be done in original untouched RGB data before transform to working color space. This helps to avoid color clipping due color spaces mismatch.
Curve may be based on Cineon or on L* gamma and looks like this:
gamma 1.0 -> L* gamma -> Invert -> gamma 1.0

RGB Auto Levels also work better in L* gamma. But i guess in your app they probably operate linear float in some different way.

For example globally workflow may look like this:
Invert LUT -> Input ICC -> ACES L* -> RGB Auto Levels -> WB

There are some examples and discussion about invert problem in this thread https://liftgammagain.com/forum/index.php?threads/home-scanning-negatives-proper-cineon-inversion.13714/page-2#post-150583

[aurelienpierre]

There is no mistake here. "Untouched" RGB data will always be some sensor RGB, either scanner or DSLR, so it needs to be profiled to XYZ and white-balanced at very least. Otherwise you invert sensor RGB, not film RGB. Plus, for cheap scanners, images are saved to 8 bits JPEG with an OETF (what you improperly call gamma), and this needs to be reverted before inversion.

You can choose the RGB working space you want in input profile module, that will then be used after the correction is performed to actually inverse. Some people have developed specialized RGB spaces that you can add in ~/.config/darktable/color/in, reproducing the actual RGB space of film. You can find the details here : https://elstoc.github.io/dtdocs/module-reference/processing-modules/negadoctor/

There is no clipping anywhere, if clipping happens it is because of bad settings. darktable pipeline is 32 bits floating point unbounded.

If you absolutely must invert in input RGB, the module can be moved earlier in the pipeline with Ctrl+Shift+Drag and drop in the GUI list of modules.

[radioproektor]

Just try so invert inside sRGB and you will see the clipping effect. You can try my film scan example https://www.dropbox.com/s/mbjtdt7rpljk62v/900x600-16bit-scan-test.zip?dl=0

More-less ok solution is to invert inside BestRGB color space. BestRGB was designed to perfectly fit all colors produced by film. But even BestRGB have some gaps and can't 100% keep all inverted colors and produce some tiny hue and saturation shifts compare to original.

Invert in scanner color space is not easy task because it produce green color cast after invert (and it is not a part of film negative color, but just a shifted white point in original RGB data color space). Here is an example patches from IT8.7 positive slide film target.
Top row: Input ICC -> BestRGB L* ICC -> Invert (no white point color shift, but depending of working color space some colors may clip)
Bottom row: g1.0-to-L*-invert-to-g1.0 LUT -> Input ICC -> BestRGB L* ICC (white point color shift, but no color clipping because invert done inside original untouched scanner color space)

Currently i have no idea how to deal with that color cast...

[radioproektor]

Maybe it could be some better way to transform color spaces? Some special transformation of input colors for proper invert?

[radioproektor]

[aurelienpierre]

Currently i have no idea how to deal with that color cast...

Correction tab, white balance and color shift, are meant to deal with color casts.

Just try so invert inside sRGB and you will see the clipping effect. You can try my film scan example https://www.dropbox.com/s/mbjtdt7rpljk62v/900x600-16bit-scan-test.zip?dl=0

Again, there is no clipping. So maybe clipping is not the right word for what you see, or Dmax and scan exposure bias are ill-set in film properties tab, but there are several safety jackets to normalize white and black points in order to not clip.

The example you provide is encoded in sRGB, which is a small space, so your gamut is already clipped when the scanner saves the file. Improving the color processing will not restore the gamut you have lost at this destructive step.

This is what I get using negadoctor default workflow:

Your problem seems to be an ill-set Dmin along with an aggressive Dmax. My Dmin is Red = 39%, Green = 41%, blue = 39%, thus medium grey, while yours is bright cyan, and Dmax is set to 2.26 dB while you set it to 1.7 dB.

[radioproektor]

INPUT DS_Elite_II g1.0 K3-E130401 HQ XYZ_cLUT.icc Input profile in archive. It is not sRGB. It is wide gamut. You need manually attach input profile to image in darktable. From your example it seems you use sRGB as input color space by mistake. This is incorrect. Please attach input profile provided in archive.
Maybe it is not "Clipping" but i don't know how to name this thing with another world.
In other graphic editing apps i see this "Clipping" and colors changes in unprocessed image. Simple workflow like: Input ICC -> Working ICC -> Invert
Colors and saturation clipping changing a lot when i change Working ICC profiles.
This is all because Invert require original scanner color space (or at least some color space close to original).
It also depends a lot of color management intent. That's why i was thinking that some special color space transform may help to transform image from Input profile to Working profile in better way and provide less artifacts during invert.

[radioproektor]

Just for reference. In real life those Yellow batteries shown on the photo in the upper left corner have warm yellow color. They look correct when photo processed in BestRGB color space.
In ProPhotoRGB they became yellow-acid green, which is incorrect color.

[radioproektor]

Another example. This is a example DSLR RAW photo of film negative shared by someone on some photoforum. Switching between working color spaces and picking the same frame edge produce different results. Most saturated red color is dimmed in Prophoro and overloaded (clipped) in sRGB. This is all due problem with invert. Download original RAW file here https://drive.google.com/file/d/12Avltcaz9XLeGLfNP2qilopWogwGzb8l/view?usp=sharing

[aurelienpierre]

Switching between working color spaces and picking the same frame edge produce different results.

Works as it should. Changing color spaces changes the coordinates, no problem here. That's vector algebra in 3D spaces.

This picture edits just fine if you don't input a crazy Dmin and use the white balance adjustment in corrections tab :

Clipping means bad users settings. Everything is made in the module to prevent it, the one thing I can't prevent is user doing silly things.

[radioproektor]

Seems i figure the source of strange color clipping during invert inside different color spaces problem!
RGB Exposure adjusted directly inside scanner or WB adjusted in image before input ICC profile produce dramatic changes in input RGB data color space. As result scanner color space don't match to Input ICC profile generated with IT8.7 target. Overall it is like apply Input ICC profile designed for totally different device.
Same goes to RAW sources. WB adjusted in RAW may cause overload or shift colors during invert. So those who use photo cameras to copy negatives should somehow find a neutral WB that will match best to camera input ICC profile and will not produce colors shift and overload during invert.

These workflow chains produce color cast due mismatch between input profile and actual RGB data and probably should not be used:
Scanner RGB exposure adjust (orange mask fix) -> Input ICC -> Working ICC -> Invert -> RGB Auto Levels
Image WB adjust (orange mask fix) -> Input ICC -> Working ICC -> Invert -> RGB Auto Levels

To be safe just keep orange mask during scan untouched and remove it later inside working color space.
Untouched scan -> Input ICC -> Working ICC -> WB adjust (orange mask fix) -> Invert -> RGB Auto Levels

Here is archive with new source scans (with and without orange mask removed in-scanner) so you can experiment with your own settings and see the difference. Note that these scans are from different scanner, so don't forget to apply proper input ICC profile included in this archive.
https://www.dropbox.com/s/v6b1eb8hbfl8lf9/Scan_Negative_TEST-2.zip?dl=0

[radioproektor]