False colors under tungsten illumination

[Beep6581]

Originally reported on Google Code with ID 2747

After Revision: bf5be3ed66b8 - Issue 2691: No chromatic adaptation in DCP code => wrong
colors

there is a serious regression of color rendering on scenes under tungsten illumination
.. at least when we use single dcp profiles built with x-rite's color passport utility.
These profiles are usually built using a single illuminant (daylight).

I had only tested the patch with samples illuminated by close to daylight illumination
and the issue2691 patch was an improvement there. Unfortunately with samples under
tungsten we have wrong colors.

A useful data set is Dpreview's studio samples which are shot under both "daylight"
and tungsten (push the lamp icon at the top for tungsten) http://www.dpreview.com/reviews/image-comparison/fullscreen?attr18=daylight&attr13_0=sony_a7&attr13_1=canon_eos5dmkiii&attr13_2=nikon_df&attr13_3=sony_a7s&attr15_0=raw&attr15_1=raw&attr15_2=raw&attr15_3=raw&attr16_0=100&attr16_1=100&attr16_2=100&attr16_3=100&normalization=full&widget=1&x=-0.042789977043272844&y=0.664669889676727

As RT comes bundled with 5DIII dcp I used the Dpreview 5DIII raw samples from the above
link .. find screenshots 
https://drive.google.com/folderview?id=0B0NqktTgc54sfkJ1aGlXRTdGbEVBQ3pNU0dENnp1ODBmaUJTNUN4Nzl6TWVvaHN2OGdPR3c&usp=sharing

Reported by iliasgiarimis on 2015-04-21 09:42:43

[Beep6581]

Most easily detected are redish colors ..

Reported by iliasgiarimis on 2015-04-21 09:43:51

[Beep6581]

Reported by iliasgiarimis on 2015-04-21 09:44:20

• Status changed: New

[Beep6581]

Reported by iliasgiarimis on 2015-04-21 09:45:05

[Beep6581]

Good find! This explains why I kept getting pink toys in a photo I knew had red ones
  in a color-managed workflow... 

Reported by entertheyoni on 2015-04-21 13:17:21

• Status changed: Accepted

[Beep6581]

DrSlony, do you have any cc24 shots under tungsten ? it would be useful for some measures
.. Dpreview's tungsten shots are not lighted properly for this (light fall off right
to left).

Reported by iliasgiarimis on 2015-04-21 15:22:24

[Beep6581]

I will make one tonight. Does it have to be a clear tungsten bulb? Can it be a white
tungsten bulb? Can it be a candle?

Btw. I just learned how to make ICC profiles using ArgyllCMS. Ah I'll paste it here
for reference:
argyll-scanin -v -p -G 1.0 -dipn /tmp/old_wb.tif /usr/share/argyllcms/ref/ColorChecker.cht
/usr/share/argyllcms/ref/ColorChecker.cie /tmp/old_wb_ColorChecker_diag.tif
then                                                                              


argyll-colprof -v -A "Pentax" -M "K10D" -D "Pentax K10D overcast matrix" -qh -am -nc
-u old_wb && mv -v old_wb{,_matrix}.icc                                           
argyll-colprof -v -A "Pentax" -M "K10D" -D "Pentax K10D overcast Lab cLUT" -qh -al
-nc -u old_wb && mv -v old_wb{,_Lab_cLUT}.icc                                     

argyll-colprof -v -A "Pentax" -M "K10D" -D "Pentax K10D overcast gamma+matrix" -qh
-ag -nc -u old_wb && mv -v old_wb{,_gamma_matrix}.icc                             

argyll-colprof -v -A "Pentax" -M "K10D" -D "Pentax K10D overcast singlegamma+matrix"
-qh -aG -nc -u old_wb && mv -v old_wb{,_singlegamma_matrix}.icc                   
argyll-colprof -v -A "Pentax" -M "K10D" -D "Pentax K10D overcast shaper+matrix" -qh
-as -nc -u old_wb && mv -v old_wb{,_shaper_matrix}.icc                            


-a is algorythm type. Choices are:
l = Lab cLUT (def.), x = XYZ cLUT, X = display XYZ cLUT + matrix
g = gamma+matrix, s = shaper+matrix, m = matrix only,
G = single gamma+matrix, S = single shaper+matrix

The resulting Lab cLUT ICC leads to poor results as predicted because my chart only
has 24 patches. The rest all look good, and don't exhibit this issue when applied to
this 2507 Kelvin photo: http://rawtherapee.com/shared/test_images/pentax_k10d_09_20.pef

I will upload the tungsten cc24 shot here too, tonight or in the morning: http://filebin.net/rt2747xdcp

Reported by entertheyoni on 2015-04-21 15:33:17

[Beep6581]

I still have to learn how to make DCP profiles using ArgyllCMS, so we can eliminate
the X-Rite program as being the cause.

Reported by entertheyoni on 2015-04-21 15:35:04

[Beep6581]

# 6, anything that gives full spectrum 2500-3000K light. I think the white is better
due to diffused light.

nice with agryll, up to now I relied on frontends (coca or makeinputicc) for this :).
The first step exports *.ti3 .. yes ?. 
BTW I am in search for an "Old_colorchecker.cht" to be compatible with Ted's cc24 target
(see issue_2729) .. 

#7, Anders could give a hand I suppose :)
I think it's not x-rite's utility at fault, but RT's dcp code, building an icc and
converting to dcp will again have the same issue as #0, something goes wrong with the
bradford adaptation .. it was OK before.

Reported by iliasgiarimis on 2015-04-21 19:00:48

[Beep6581]

I'm working intensively with a camera profiling software which I hope to release open
source in the coming weeks. It can do DCP profiles, both from targets or direct by
using spectral sensitivity functions, it will be useful for making profiles for RT.

Until release I won't be doing much context switching, so I can't look into this issue
now.

I'd suggest to test the problematic profiles in Lightroom as a reference. It can be
the problem of the profile rather than the DCP handling.

Reported by torger@ludd.ltu.se on 2015-04-21 20:03:56

[Beep6581]

Correct, it created the .ti3 I uploaded.
I will try to help with his profile.

Reported by entertheyoni on 2015-04-21 20:24:44

[Beep6581]

Sorry for the delay. Real tungsten clear bulb ("milky white" would change the spectrum).
http://rawtherapee.com/shared/test_images/pentax_k10d_2015-04-20_passport_daylight.dng
http://rawtherapee.com/shared/test_images/pentax_k10d_2015-04-22_passport_tungsten.dng

I will compare this to candlelight tonight, I'm thinking four candles some distance
away at each side to minimize falloff.

Reported by entertheyoni on 2015-04-24 15:49:44

[Beep6581]

Off topic but related; I have now made my first release of my open-source digital camera
profiling tool, you can make DCPs with this (you need Argyll too):

http://www.ludd.ltu.se/~torger/dcamprof.html

Reported by torger@ludd.ltu.se on 2015-04-30 17:28:14

[Beep6581]

Anders, Congratulations !!. 

I wish a win32 complilation will be available to test it's results.

BTW are there any profiles built by Dcamprof available for testing ..  

Reported by iliasgiarimis on 2015-04-30 22:36:08

[Beep6581]

Thanks :-).

You really should get a Linux box running, everything is just soooo easy to compile
there, development environment is builtin. A virtual box with ubuntu in it running
inside your windows perhaps?

Well, maybe I'll look into building a windows version. I have windows 8 running in
a virtual box on my Linux box :-).

I haven't actually had the time yet to play around that much with DCamProf other than
testing during development to see that it works. I'll start making profiles soon.

One cool thing with it is that you can make profiles directly from the camera's spectral
sensitivity functions, that its color filter responses. Unfortunately it's not so easy
to come by that kind of data, and it requires costly gear to measure yourself. For
my Canon 5Dmk2 there is SSFs to get though, but for my Hasselblad I'll have to make
profiles the traditional way with a target.

Reported by torger@ludd.ltu.se on 2015-05-01 07:20:00

[Beep6581]

Here are two profiles generated with the default settings of DCamProf using the images
you attached in #11

Workflow:
1. rotate crop and export for profiling *without white balance*
2. argyll: scanin -v -dipn rawfile.tif ColorChecker.cht cc24_ref.cie
   (ColorChecker.cht comes with Argyll, cc24_ref.cie comes with DCamProf)
3. dcamprof make-profile -i D50 rawfile.ti3 profile.json
   Replace -i D50 with -i StdA for the tungsten
4. dcamprof make-dcp -c "Pentax K10D" profile.json profile.dcp

Let me know what you think.

Reported by torger@ludd.ltu.se on 2015-05-01 13:43:57

---

- _Attachment: [k10d-tungsten.dcp](https://storage.googleapis.com/google-code-attachments/rawtherapee/issue-2747/comment-15/k10d-tungsten.dcp)_ - _Attachment: [k10d-daylight.dcp](https://storage.googleapis.com/google-code-attachments/rawtherapee/issue-2747/comment-15/k10d-daylight.dcp)_

[Beep6581]

...and here's a DCP with both DCPs from #15 merged into one dual-illuminant profile.

Workflow:
1. make two separate profiles as described in #15
2. convert both profiles to JSON, "dcamprof dcp2json profile.dcp dcp-profile.json"
3. merge both profiles using a text editor, the lower temp profile gets index 1, the
higher index 2.
4. convert back to DCP, "dcamprof json2dcp edited.json dual.dcp"

Reported by torger@ludd.ltu.se on 2015-05-01 13:54:53

---

- _Attachment: [k10d-dual.dcp](https://storage.googleapis.com/google-code-attachments/rawtherapee/issue-2747/comment-16/k10d-dual.dcp)_

[Beep6581]

Great work, Anders!
If you (or someone else) wants to do some comparisons or quality checks, I uploaded
all the CC24 Raw files that were used for the creation of the current DCP profiles
in RT for the following cameras: Pentax K-5, Pentax K-5II, Canon Powershot S110.

The DCP profiles were created with the X-Rite software and had their quality checked
with the colorchartcheck.exe utility. For comparison I also include an alternative
profile created with Adobe DNG Profile Editor for the K-5II. 
http://filebin.net/u6jmuy4dio 

Reported by stefan.ittner on 2015-05-01 19:19:35

[Beep6581]

I just released a version 0.5.1 of my software which fixes some issues, so the profiles
in #15 and #16 should be regenerated, but it would make only a tiny (probably invisible)
change.

Yes it would be nice if someone could do some qualitive testing, I haven't had much
time for that. Note that DCamProf does not only take precision into account, there's
no problem to match 100% each patch when you have a LUT, but it can lead to sharp bends
in the LUT which leads to bad gradients. DCamProf makes averaging automatically if
patches are too close in chromaticity which happens for a few patches in the cc24.
Therefore it may seem like the profile is "worse" if the only evaluation is DE on patch
matching.

DCamProf also makes a larger LUT than needed to only match the patches; as DCP HSM
LUT uses linear interpolation and DCamProf has spline-based native LUT it creates the
DCP HSM LUT by sampling the spline-based LUT to get smoother interpolation and thus
better gradients. This is also an aspect that is not easily seen when just evaluating
patch matching.

Reported by torger@ludd.ltu.se on 2015-05-01 20:58:22

[Beep6581]

Anders great! Can't wait to test it, but its 2AM now so I'll have to sleep on it :)
I tried the DCPs from #15 and by eyeball the patch colors look more as they should
than when I use DCPs made using X-Rite or ICCs made using ArgyllCMS. More specifically,
the red looks more red and less pink, green is more green, magenta is not oversaturated,
and orange is more vibrant. Not sure about the blue and purple.

How can we quantify the accuracy?

I read that creating LUT profiles from CC24 targets leads to bad results because there
are too few sample points, but it sounds like your spline approach works around this.

What is HSM?

Reported by entertheyoni on 2015-05-02 00:07:39

[Beep6581]

With DCamProf you can through its SSF support test how good or bad a specific target
is when comparing them to others. I compared it to 1600 patch Munsell and the difference
is not that large, it's hard to say which one's best.

The thing is that the profiling problem is "impossible", each vision color can be represented
by an infinite amount of spectra, and I've seen in my SSF simulations that contradicting
results are not uncommon, that is a green leaf may pull the LUT in one direction and
a green printed color of the similar vision color may pull it in a different, so you
can't correct for both.

That said it would not hurt if the cc24 had more saturated colors and some more purple.
I'm less worried about cc24 profiles now than I was before I ran through my simulations
though.

Of course a profile will be good at fulfilling accuracy of the test target patches
though, so testing if a cc24 target matches cc24 patches is more of a sanity check,
good to do though. DCamProf won't match all 100% as it makes a 2.5D LUT and therefore
groups together the neutral row for example into one group and make and average correction
for those.

HSM = HueSatMap, the DNG Profile LUT.

Spline probably doesn't improve accuracy, but it improves smoothness, ie transitions
from one color to another. This paper does show some advantages of thin plate spline
though (which is the what DCamProf is using):
http://www.ncbi.nlm.nih.gov/pmc/articles/PMC3435966/

Reported by torger@ludd.ltu.se on 2015-05-02 06:25:12

[Beep6581]

I created a dual-illuminant DCP from those tungsten and daylight shots using your latest
version 0.5.2.

For reference:
argyll-scanin -v -dipn daylight.tif /usr/share/argyllcms/ref/ColorChecker.cht ~/programs/dcamprof-0.5.2/data-examples/cc24_ref.cie
argyll-scanin -v -dipn tungsten.tif /usr/share/argyllcms/ref/ColorChecker.cht ~/programs/dcamprof-0.5.2/data-examples/cc24_ref.cie
~/programs/dcamprof-0.5.2/bin/dcamprof make-profile -i D50 daylight.ti3 daylight.json
~/programs/dcamprof-0.5.2/bin/dcamprof make-profile -i StdA tungsten.ti3 tungsten.json
~/programs/dcamprof-0.5.2/bin/dcamprof make-dcp -c "Pentax K10D" daylight.json daylight.dcp
~/programs/dcamprof-0.5.2/bin/dcamprof make-dcp -c "Pentax K10D" tungsten.json tungsten.dcp
~/programs/dcamprof-0.5.2/bin/dcamprof dcp2json tungsten.dcp dcp-tungsten.json
~/programs/dcamprof-0.5.2/bin/dcamprof dcp2json daylight.dcp dcp-daylight.json
cat dcp-tungsten.json dcp-daylight.json > dual.json
~/programs/dcamprof-0.5.2/bin/dcamprof json2dcp dual.json "Pentax K10D 2015-04-22 daylight
tungsten dcamprof.dcp"

Reported by entertheyoni on 2015-05-02 11:33:06

[Beep6581]

I attached the DCP along with one made using Adobe's DNG Profile Editor, another from
X-Rite's Passport program, and two ICC profiles from ArgyllCMS, all made using the
same shots, just in case anyone wants to compare.

Anders I sometimes shoot in light from LED stage cans, if I was to shoot the passport
under white LED light can you advise what options to pass to DCamProf, how to specify
the right illuminant (there is no LED standard illuminant I'm aware of) and whether
I could still use the cc24_ref.cie file?

Reported by entertheyoni on 2015-05-02 11:42:11

---

- _Attachment: [Pentax K10D 2015-04-22 daylight tungsten DPE.dcp](https://storage.googleapis.com/google-code-attachments/rawtherapee/issue-2747/comment-22/Pentax K10D 2015-04-22 daylight tungsten DPE.dcp)_ - _Attachment: [Pentax K10D 2015-04-22 daylight tungsten dcamprof.dcp](https://storage.googleapis.com/google-code-attachments/rawtherapee/issue-2747/comment-22/Pentax K10D 2015-04-22 daylight tungsten dcamprof.dcp)_ - _Attachment: [Pentax K10D 2015-04-22 daylight tungsten xrite.dcp](https://storage.googleapis.com/google-code-attachments/rawtherapee/issue-2747/comment-22/Pentax K10D 2015-04-22 daylight tungsten xrite.dcp)_ - _Attachment: [PentaxK10D 2015-04-20 passport daylight.icc](https://storage.googleapis.com/google-code-attachments/rawtherapee/issue-2747/comment-22/PentaxK10D 2015-04-20 passport daylight.icc)_ - _Attachment: [PentaxK10D 2015-04-22 passport tungsten.icc](https://storage.googleapis.com/google-code-attachments/rawtherapee/issue-2747/comment-22/PentaxK10D 2015-04-22 passport tungsten.icc)_

[Beep6581]

Anders RawTherapee's "DCP Illuminant" combobox is grayed-out when I load this DCamProf
DCP. Did I merge them incorrectly?

Reported by entertheyoni on 2015-05-02 12:19:17

---

- _Attachment: [dcp-tungsten.json](https://storage.googleapis.com/google-code-attachments/rawtherapee/issue-2747/comment-23/dcp-tungsten.json)_ - _Attachment: [dcp-daylight.json](https://storage.googleapis.com/google-code-attachments/rawtherapee/issue-2747/comment-23/dcp-daylight.json)_

[Beep6581]

You can use the cc24_ref.cie for any illuminant as it contains reflectance spectra,
that is a specification on how each patch reflects light. So DCamProf can calculate
exactly what light in the full visible spectrum that reflects back to the observer
(eye).

I don't know what the spectrum from your led stage lights are. Probably the spectrum
is a little bit peaky just like fluorescents and not really ideal for profiling. Halogen
lamps are still better. However if you really want to make a profile to be used under
that led light, it's best to use that as light source of course.

DCamProf uses illuminant spectrum for two things, 1) to calculate XYZ reference values
for the color matrix and 2) to assign calibration illuminant to the DCP. Actually it
does not matter that much if it's correct. The color correction takes place with the
forward matrix which is always related to D50. Specifying the calibration illuminant
is more for informational use. Calibration illuminant matters when calculating the
color matrix and it's only used to figure out which color temperature we're shooting
at (white balance), and has some value when making a dual illuminant matrix too as
it will decide how to mix the two matrices and LUTs.

With DCamProf you can specify illuminant StdA, D50 and D65, plus any blackbody. One
suitable illuminant for your led could be to find out what it's correlated color temperature
is and use that as illuminant.

If the led's correlated color temperature is reasonably close to StdA (~2850K) or D50
(~5000K) or D65 (6500K) I'd specify that as illuminant to make-profile. Otherwise I'd
specify with a blackbody say "-i 3700K" if the CCT would be 3700K. The illuminant name
in the DCP will then be the boring "Other", but the profile should work as it should.
Haven't really tested how RT reacts to "Other" illuminant, as the dual-illuminant files
are nearly always StdA + D65.

Another way is to simply see if the light is redder than daylight and then pretend
it's StdA. If it's a lot wrong it will only be a potential slight problem in a dual
illuminant profile.

Note that RT never get its white balance temperature from the DCP matrices, it gets
it from the DCraw/Camconst D65 matrix. In a longer term we should change this so RT
uses the DCP.

Reported by torger@ludd.ltu.se on 2015-05-02 14:33:36

[Beep6581]

I tried your Pentax K10D 2015-04-22 daylight tungsten DPE.dcp in #22 and it is dual
illuminant and it works, combo box is not greyed out.

Reported by torger@ludd.ltu.se on 2015-05-02 14:37:30

[Beep6581]

Yes, I want to use the LED profile only for those shots, as I shoot there quite frequently.

The profile you tried with "DPE" in the filename comes from DNG Profile Editor. It
is the one named "Pentax K10D 2015-04-22 daylight tungsten dcamprof.dcp" which has
the combobox grayed out. I see now that concatenating doesn't work, I did it like this
now and it works http://i.imgur.com/gfl74Bw.png
Fixed dual-illuminant DCamProf DCP attached.

Reported by entertheyoni on 2015-05-02 15:55:33

---

- _Attachment: [Pentax K10D 2015-04-22 daylight tungsten dcamprof.dcp](https://storage.googleapis.com/google-code-attachments/rawtherapee/issue-2747/comment-26/Pentax K10D 2015-04-22 daylight tungsten dcamprof.dcp)_

[Beep6581]

If you have access to a spectrometer you can read the LED light's spectra using Argyll's
illumread. I've just added ability to read Argyll's SPECT format directly when specifying
illuminant, coming in the next patch release. In the current one have to reformat it
to JSON to read it.

But as said it's not that important that it's right, if you're making a single-illuminant
profile and using it with RT it's not used at all as all color correction is done with
the ForwardMatrix which is locked to D50, and white balance derivation is done from
the camconst/dcraw color matrix.

Reported by torger@ludd.ltu.se on 2015-05-02 16:11:57

[Beep6581]

Released anew version 0.5.3 where you don't need to merge dual illuminant profiles manually.

dcamprof make-dcp profile1.json profile2.json output.dcp

One can also get a full list with all exif lightsources and their temperatures, useful
for choosing calibration illuminant.

Reported by torger@ludd.ltu.se on 2015-05-03 17:32:34

[Beep6581]

Do these DCP profiles support the perceptual rendering intent?

Reported by entertheyoni on 2015-05-03 21:40:14

[Beep6581]

#29, uh I don't think I understand. What does it mean to support perceptual rendering
intent? As far as I know DCP profiles is not supposed to do any gamut mapping.

I've seen custom profiles that actually do gamut mapping (that is move in saturated
colors outside sRGB so they get inside) made by people that is not satisfied how Lightroom
does gamut mapping.

However this means you get a static mapping so that even if no colors are outside sRGB
you still get compression. It's better to let the raw converter color pipeline handle
gamut mapping, which usually just means pure clipping and then you as a user use the
color adjustment tools to adjust to your liking. I prefer that way to work.

Reported by torger@ludd.ltu.se on 2015-05-04 06:34:07

[Beep6581]

Why have we to hide this excellent open source utility (DcamProf) in a different issue
??. It deserves it's own topic I think ..

The current issue is about the wrong colors under warm illumination after issue 2691
commit (bradford adaptation in dcp code). RT was behaving better before 2691 patch
for warm illumination .. Although for daylight/cloudy there was an improvement by 2691
.. !!.

I 'll make some measures on DrSlony's sample in the next hours, but even without measures
looks like either something goes wrong with Bradford Adaptation in RT's dcp code when
the target is warmer that the dcp's illuminant or it is incompatible with RT's color
management flow .. (double adaptation ??) ..  

Reported by iliasgiarimis on 2015-05-04 14:19:31

[Beep6581]

Any modern DCP should be unaffected by Bradford adaptation as they contain a forward
matrix, except in the case of dual illuminant when it should slightly affect the mix
between FM1 and FM2.

We can't have an own interpretation of how DNG Profiles should work in RT, it must
follow the standard.

One cannot expect that a D50 profile should produce accurate colors for Tungsten. On
average using Bradford adaptation should make it less bad, that's why it's there in
the standard, but some profiles could probably become worse by coincidence.

It could be a bug somewhere of course, but to verify I'd first run the same profile
in a different DNG-compatible software such as Lightroom or Iridient. If the same problem
is seen there the problem is in the profile not RT's DCP code.

Reported by torger@ludd.ltu.se on 2015-05-05 06:02:03

[Beep6581]

What's the status of this one?

Reported by torger@ludd.ltu.se on 2015-07-07 14:20:51

[iliasg]

Bump up ..
The problem is still valid and because RT distributs some single illuminant dcp profiles this issue is serious. The colors under tungsten are much worse than before the dcp_bradford adaptation commit.

The fact that the single illuminant dcps was working correctly before, makes me suspect that an adaptation code is somewhere in RT workflow and now with the bradford adaptation in dcp two adaptation are applied one upon another.

We can either

• disable the bradford adaptation in dcp code if a single illuminant is detected (or disable the old adaptation in later stage if it exists)
• convert the existing single illuminant dcp profiles to double illuminant (I think Dcamprof can do it)
• remove single illuminant dcps from RT distribution

[Beep6581]

@atorger ping

@iliasg I am updating all DCPs we have color target shots for, for the 5.0 release. Busy with that right now in fact.

[iliasg]

@Beep6581 I am not familiar with DcamProf .. is it true that it can build dual illuminant dcps even from a single illuminant raw sample ?.

[atorger]

You can force dual illuminant profiles yes, but it's basically just used for copying Adobe Lightroom's color matrices to avoid white balance shift when using DCamProf profiles in Adobe's products.

It can't make a "sane" dual illuminant profile from a single raw sample though, it's not possible as the profiler needs the information of how the camera responds under both illuminants to make a dual-illuminant profile with matrices and LUTs that are adapted for those lights. I guess you could make some educated guess on how the camera would respond under a different light with some statistical modeling etc, but DCamProf doesn't have that.

[Beep6581]

@iliasg

The colors under tungsten are much worse than before the dcp_bradford adaptation commit.

Could you point to the commit and the color measurements which show this?

[iliasg]

@Beep6581 it was 4.2.101 I think .. see the first post of the current issue .. there is a link with samples there ..

No measures (I have to build again the measuring structure ..) but the result is obvious with naked eyes ..

[iliasg]

#2673

[atorger]

Have you compared to ACR/Lightroom/DNG reference code behavior? I don't think it makes sense to have an own DNG Profile interpretation in RT. The adapatation patch was made to put the conversion in line with the "standard". Profile makers that make DNG profiles must be able to rely on that the DNG-using software is actually following Adobe's document on how to render.

I could be some problem with the profiles themselves. Try loading the same profile into some Adobe product and see if the result differs. If Adobe renders colors "right" with the same profile and RT "wrong", then there is some bug somewhere, but if Adobe renders the colors equally "wrong", I don't think RT should be "fixed", the profile should be fixed instead -- as I don't think it's a good to break the documented DNG standard so the situation eventually becomes as bad as with ICC...

Step 1: verify if RT's rendering is following the DNG standard

When we know the answer to that we can decide which action to take. It's not unthinkable to me that Adobe's Bradford adaptation is a bad idea, but it's standard, so if we don't like it it's better to redesign the profiles to counteract that conversion, which I guess would be to render a forward matrix from the color matrix.

[iliasg]

@atorger @Beep6581
I don't have any adobe product available so I cannot check ..

But I checked with RT the latest dcps from #3269 (comment) .. they look almost fine (well .. red component is still too much) on tungsten .. unlike with Canon 5D3 dcp (first post, build by x-rite's utility ?) which is terrible !!

Anders, my point is that all profiles worked fine (for all illumination temps) before your patch so a kind of adaptation must exist in RT .. after your patch for Bradford adaptation on dcps (where I see no mention of changes in other places) some dcp profiles stopped working fine .. could we relate this with possibly double adaptation ?

[Beep6581]

@iliasg or anyone, could you provide a link to a raw file + DCP where this happens?

[atorger]

Some kind of double adaptation sounds like a plausible theory. The thing is that I verified the patch by comparing to Lightroom and found no error back then, it was an A equals B test, that is no subjectivity. It was a long time ago though so I could have made some mistake in the testing.

I think you could use Adobe DNG Profile Editor to get a reference rendering with DCPs, it's free.

[atorger]

Without having time to retest my theory is that RT is doing it right, but maybe the DCP way of converting the colormatrix is not so good. Not all aspects of the DNG standard is good, Adobe themselves have changed their profiles designs over the years. No modern profile use colormatrix only (it was abandoned like 10 years ago), but use forwardmatrix for the actual color and colormatrix only for color temperature estimations (for which RT has its own model by the way). The DNG document doesn't state it clearly but between the lines I read forwardmatrix was added because colormatrix model didn't really work well. Should be said that ICC matrix profiles has always worked the "forward matrix way"

A profile with only a single forward matrix will not get its matrix Bradford-transformed but just whitebalanced - as the DNG document says, and thus work as RT did before the patch, and the same way as a matrix-only ICC profile is handled.

If my theory is right the right way is to fix the profiles (convert from colormatrix to forwardmatrix profile), and the "easy" way break standard behavior and restore as before, but I wouldn't like that.

[iliasg]

@Beep6581

@iliasg or anyone, could you provide a link to a raw file + DCP where this happens?

It's a Canon 5DIII file linked below and the dcp is the one that RT distributes and gets autoselected by default. It;s a profile build by x-rite's profiler ..
Canon EOS 5D Mark III.zip
https://www.dpreview.com/reviews/image-comparison/download-image?s3Key=0bfcf93a05cb4efb926f37fd85eb1369.cr2

[atorger]

Quoting the DNG spec page 81, about the forward matrix:

"The use of the forward matrix tags is recommended for two reasons. First, it allows the camera
profile creator to control the chromatic adaptation algorithm used to convert between the
calibration illuminant and D50. Second, it causes the white balance adjustment (if the user
white balance does not match the calibration illuminant) to be done by scaling the camera
coordinates rather than by adapting the resulting XYZ values, which has been found to work
better in extreme cases."

That is, it sounds to me that even Adobe found out that the adapting via bradford probably ain't that good, if D50 to StdA is one of those "extreme cases".

1. check if RT is doing like the standard, use an Adobe product as reference.
2. if RT is doing as the standard says, and the bad color is due to bradford, then convert the old colormatrix profiles to forwardmatrix profiles, or reprofile them with DCamProf to get a proper profile with forwardmatrix. If X-Rite really only makes colormatrix profiles it shouldn't be used I think.

There's a special place in hell for programmers that make their own interpretations of standards and thus fragment them into meaninglessness. DNG is not a true standards body standard but it is at least an attempt to bring some order into the industry when it comes to camera profiles, and I think we should support that, rather than contributing to fail it. We shouldn't have a special RT DCP pipeline, then it becomes as broken like with ICC where different raw converters have different ways to interpret them. So I oppose removing bradford adaptation even if it makes better colors for old DCPs.

[iudnikov]

Exactly the same issue: my StdA target was shoot under 2600K tungsten lamp. But when I create dual illuminated profile and specify -i StdA in make-dcp red colours became pink at ~2800K white balance. But when I keep only -I D55 then everything works well. I already tried make-profile with both -i 2600K (which makes target shot neutral) and -i StdA... the same result. Warm red becomes pink, but at 5000K white balance red colours are okay. Then I've tried to compose two sources of light to achieve 2900K while target shooting... and the same!