Further improve the negative film inversion with a new UI Widget (new controls)

[laurencelumi]

Is your feature request related to a problem? Please describe.

Darktable includes Negadoctor,

This basically implements a well known transform (actually several)

The key transform is "Derivated from Kodak Cineon system, it aims at inverting a log-encoded (density-scaled) scan, then linearize it (sort-of) for better handling in the scene-linear pipe."

Note the above is lifted from #4113

However fundamentally the key controls to this formula are hidden, and not directly accessible from the UI.

For example the ability to increase the exposure to the log or density values correctly for each channel does not exist. i.e. the ability to mimic printer lights color calibration used for balancing a traditional optical print. (The Cineon system is design to mimic the analog system)

While the currently implementation is usable, its essentially unique because the controls don't match how the cineon system is supposed to work.

Describe the solution you'd like

I started refactoring making the changes to negadoctor.The changes will result in new values being stored in DTs xmp files. So intend to clone this module and give it a new name, with the required changed.

The new module will still use the same basic algorithm but with new sliders controlling the variables.

Alternatives

I think the issues are fundamental yet easily fixed.

Additional context

In addition, a full workflow involves additional steps, namely:

1. An adjustment to scan, prior to the inversion to better match the scanning device response to a notional printing density. see TSMPTE ST 2065-2 - Academy Printing Density (APD)
2. After the inversion a shaper matrix or LUT needs to be applied to match response of the print film.

These steps are difficult to implement, or even attempt as negadoctor bundles everything in one step.

[gi-man]

The new module will still use

I think it might be best to implement the new controls in the same module via versioning instead of creating a new module.

[laurencelumi]

I think it might be best to implement the new controls in the same module via versioning instead of creating a new module.

I have split it out for the time being, probably easier with my programming skills for the moment :-) Plus I don't wont be constrained just yet.

Once I have something firm to show, definitely will revisit.

[Jiyone]

the ability to increase the exposure to the log or density values correctly for each channel does not exist. i.e. the ability to mimic printer lights color calibration used for balancing a traditional optical print

I don't understand what is your issue.
This is done with wb_high.

wb_high is used to control RGB channels exposure (since it's a coef) of your RGB scan in log space.

An adjustment to scan, prior to the inversion to better match the scanning device response to a notional printing density. see TSMPTE ST 2065-2 - Academy Printing Density (APD)

As I understand it, this is to convert spectral data to normalized density point.
As far as I know, it is not possible to deal with spectral data in darktable, or I am missing something?
How would that help anyway?

After the inversion a shaper matrix or LUT needs to be applied to match response of the print film.

Can't you do that already if you use the LUT module after Negadoctor?

I would be pleased to read your answer, I'm a big negadoctor user :)

[laurencelumi]

I don't understand what is your issue.
This is done with wb_high.
wb_high is used to control RGB channels exposure (since it's a coef) of your RGB scan in log space.

wb_high is in effect a gamma adjustment, it is not a density adjustment. Yes its a coeef, but its applied to the density so it becomes a power or gamma adjustment, density adjustments should be addition or subtraction when working in log. wb_low are the actual printer lights, but you can't control them properly.

`As I understand it, this is to convert spectral data to normalized density point.
As far as I know, it is not possible to deal with spectral data in darktable, or I am missing something?
How would that help anyway?

Converting to Academy Printing Density from the spectral sensitivity of the sensor used, should be possible, this is probably a lot of work. Using a gamma correction gets you a lot of the way, wb_high, so that control is still needed. But need to start somewhere :-)

Can't you do that already if you use the LUT module after Negadoctor?

Yes a LUT module should be used after negadoctor. But should the LUT work on data that is converted to scene linear, or should it work on data that has additional tone curve to reflect the print medium? I want to trial both.

Negadoctor isn't really anything special, in it that the method used can be found in Cineon/Kodak documents from 30 years ago. The conversions can be found Nuke, Davinci Resolve, 3D-Equalizer, Autodesk Lustre (5D Colossus), etc.

What I have tried to do, is go back to the original formula/principles and bring those controls directly into the UI, to see if I have a better process, in an attempt to get closer to the original approach.

Perhaps it won't be better, but I wanted to try. I did some basic prototyping a while ago, I have made my own module, which I may split into two. Which Hopefully will be ready for others to try.

I would be pleased to read your answer, I'm a big negadoctor user :)

I think I will be ready to with a pull request in a about 2 weeks. Would love some feedback!

[Jiyone]

wb_low are the actual printer lights, but you can't control them properly.

Can you develop? what is your expectation on this setting?

Converting to Academy Printing Density from the spectral sensitivity of the sensor used, should be possible, this is probably a lot of work. Using a gamma correction gets you a lot of the way, wb_high, so that control is still needed. But need to start somewhere :-)

Is this worth the effort ... 🤔

[laurencelumi]

Can you develop? what is your expectation on this setting?

wb_low should just be added or subtracted to the density calculation, in negadoctor it is a co-efficent to the offset, (wb_high is also another coefficient of offset) which is added or subtracted to the density.
i.e. d->offset[c] = p->wb_high[c] * p->offset * p->wb_low[c]

How should it work... This explains it well.

as per http://dicomp.arri.de/digital/digital_systems/DIcompanion/ch06.html
under section 6.2. Color Correction, specifically the section Primary Color Correction

Is this worth the effort ... 🤔

For scanning with DSLR with broad illuminant, probably not. But with a scanner/camera that is perhaps already close to status-M, maybe not so hard. But it is done currently, so maybe it can be done in DT.

[Jiyone]

i.e. d->offset[c] = p->wb_high[c] * p->offset * p->wb_low[c]

It's just that wb_low is expressed as a factor of the offset, and because offset also correct the scanner exposition, it's also multiplied with wb_high factor
Offset could be separate to the scanner exposure correction, which wouldn't be a bad idea actually.
Anyway you can achieve the same result, it is just a different scale and unit.

[laurencelumi]

It's just that wb_low is expressed as a factor of the offset, and because offset also correct the scanner exposition, it's also multiplied with wb_high factor
Offset could be separate to the scanner exposure correction, which wouldn't be a bad idea actually.
Anyway you can achieve the same result, it is just a different scale and unit.

Yes, I understand that, and initially I started down that route, but I realised a few things. 1. The concept of scanner exposure correction is just plane wrong. (It is really a global offset for film camera exposure) 2. How to do this is in the public domain, with excellent implementations Foundy, Resolve etc. 3. apply wb_high as factor of offset is also wrong.

Perhaps in a nutshell this is the problem, important concepts are packed all in the one place, where they don't really belong. For example wb_high which is acting as an exponent of the relative scene luminance (gamma correction), is used as the main method of color correction. Ideally wb_high would not be needed at all, in_the_inversion_process, it is really a hack to account for the mismatch, between the scanner spectral sensitivity and print mediums spectral sensitivity. Once the data is inverted sure you may want to adjust the gamma of individual channels, but there are several modules in DT that will do that. i.e. let the rest of the excellent DT do its job!

Or at least this is what I thought... So I started hacking to see if I went back the original source material, to see if can get a better result. Lets see :-)

I will post some screen shots of the gui shortly.

[Jiyone]

wb_high can also acts as paper's tint.