Canon EOS D60 files are extremely green

[garrett]

To Reproduce

1. Open a raw file from a Canon EOS D60 (not to be confused with a Canon EOS 60D, which came quite a bit later).
2. Observe wrong colors (it's usually extremely green).

A Canon D60 is a very old camera, but I used it a several years and would love all of my photo library to work in darktable. 😉

Expected behavior
Images should have natural colors. Opening in other raw editors, such as RawTherapee, gives expected results.

It's not a color balance issue, as greens persist in the dark parts of an image even after a color balance attempt to fix.

Also, I should note that I have had a handful of cameras since, both from Canon (5DmkII) and Fuji cameras, and colors are fine from default. And I also know not to expect raws to look like out of camera JPEGs and that different raw processors handle things differently, but darktable with a D60 is something else (clearly a bug somewhere).

Screenshots
darktable, with modern defaults (Filmic & new color module for white balance):

RawTherapee with defaults:

Another one; darktable:

RawTherapee defaults:

Platform

• darktable version : 3.5.0~git2563.319d682233-6084.1, but this happens in 3.4 as well.
• OS : Linux - kernel 5.12.11-300.fc34.x86_64
• Linux - Distro : Fedora 34
• Memory : 16 G
• Graphics card : Intel 620
• Graphics driver : i915
• OpenCL installed : yes, Intel Neo: intel-opencl-21.23.20043-1.fc34.x86_64
• OpenCL activated : yes
• Xorg : XWayland
• Desktop : GNOME
• GTK+ : gtk3-3.24.29-1.fc34.x86_64
• gcc : 11.1.1 20210531 (Red Hat 11.1.1-3)

Additional context

Canon D60 raw files had sidecar JPEG previews when raws were taken. Here's the raw and the ".thm" sidecar:
CRW_1456.zip

And the .thm renamed to a JPEG so GitHub accepts it:

And the same for the bird photo:
CRW_2869.zip

• Can you reproduce with another darktable version(s)? Yes, 3.4. I am pretty sure it worked fine a while ago, but didn't work in 3.4 and possibly not 3.2... but I think it worked before that?

• Can you reproduce with a RAW or Jpeg or both? Most D60 raws have this issue to one degree or another.

• Are the steps above reproducible with a fresh edit (i.e. after discarding history)? Yes

• If the issue is with the output image, attach an XMP file if (you'll have to change the extension to .txt)

• Is the issue still present using an empty/new config-dir (e.g. start darktable with --configdir "/tmp")? Yes

  

• Do you use lua scripts? Yes, but it happens with a fresh config, with no Lua scripts enabled.

Changing the color balance does help somewhat, especially with some photos, but it never fixes the problem. And some photos are worse than others, but underexposed photos or those with details at the dark end are especially affected (even after white balance correction).

---

Additionally, I am happy to license these raw files as CC0 and would cheerfully submit more raw files that were taken under different conditions, if it helps. Unfortunately, I gave this camera to my sister (who used it to shoot in JPEG) many years ago, so I only have my existing collection.

[garrett]

darktable 3.0.1 (on Fedora 32, tested from toolbox... the furthest I can go back w/o installing and using a VM) had the same issue:

(Thumbs look correct, but I didn't edit the photos. Still, they don't have their sidecar JPEG-based THM files, so I wonder if there's an additional JPEG embedded inside the CRW files or if this version of darktable skips the bug when processing thumbnails.)

Difference in rendering (while still wrong) is probably due to a basecurve and old white balance versus filmic rgb and the new color calibration module.

[garrett]

OK, perhaps darktable never handled Canon EOS D60 files properly. I remembered I could run Debian in toolboxes in Fedora, so here's darktable 2.2.1 from Debian 9:

[johnny-bit]

I've checked with sample file available on raw.pixls.us - https://raw.pixls.us/getfile.php/2010/nice/Canon%20-%20EOS%20D60%20-%20RAW%20(3:2).CRW and it doesn't exhibit same problems as your files.

for some reason your files read "as shot" wb coefficients as 1,1,1 which is kinda bonkers. sample file on raw.pixls.us has 2.624,1,1.098 wb coeffs for "as shot"...

I wonder what's more different about your shots than the sample on raw.pixls.us...

Enabling camera reference (d65) mode + color calibration also causes same problem: green everything. but if you choose wb out of say white walls on the house it figures out correct. So there IS a workaround in darktable itself...

[LebedevRI]

I guess a wrong whitebalance field is being read for some reason.

[garrett]

but if you choose wb out of say white walls on the house it figures out correct

Perhaps it might in that photo, but it doesn't really. The dark tones are still heavily shifted green, even after adjusting the white balance. I should look for another photo which shows this more prominently.

I've been setting white balance and adjusting things with color balance to try to overcome it, but it doesn't really work.

An interesting thing is that other raw converters handle the files from my old D60 correctly. (But I don't want to use other raw convertors... I ❤️ darktable and want to use it for all my cameras.)

[johnny-bit]

OK, then we need couple CC0 samples from D60 that exhibit the issue and figure out why the sample from RPU doesn't exhibit the problem :)

[garrett]

I looked quickly for an underexposed raw file and a better exposed one and found these two. Try changing the white balance on the underexposed one. It's still sorely green, and a raw file that's underexposed like that should be salvageable. The other, more properly exposed file is mostly OK, especially after a white balance adjustment. (It still has some issues, but it's subtle.)
d60-castle.zip

License: CC0 for these raw files as well as the ones above.

Here's what the underexposed file looks like with defaults:

I did use the D60 to take a picture of my ColorChecker Passport when I first got one, but I didn't know what I was doing with it so much and also converted it as a DNG (and don't have the CRW, at least easily available... if it still exsits), so it's probably not useful.

I'll look for more examples.

[garrett]

This penguins photo is in daylight and largely monochromatic. While penguins aren't pure black and white, they're close enough. It's exposed about right and adjusting the balance gets it 90% of the way there in this photo. But it's still not right.

darktable default:

darktable white balance only (look at the waveform in the right on the bottom and how the red channel isn't aligned properly):

darktable, with color balance tweaked, with highlights, dark tones, mid tones all selected from the penguins (again, look at the waveform in the top right and compare to the previous waveform):

RawTherapee defaults for comparison (it's a bit darker than darktable, but the color is right from the beginning):

THM sidecar JPEG:

Penguins raw file with XMP and THM sidecars. Two XMP files are included; one for the default, the other with the modification, including the tweaked color balance module:
d60-penguins.zip

[garrett]

Here a few more examples, including one underexposed with blown highlights (and the silver should be mainly neutral, which it isn't after a white balance), one with a grey card in the shade in a forest, and two flowers where the one with a bee is wrong, but somehow, the one by itself (CRW_7289) seems fine with the defaults. Same camera for all.

Preview:

Files:

d60-additional.zip

Again, I'm licensing all of these under CC0. Whichever ones you think are most useful, we can upload to the raw db.

[garrett]

I ran dcraw -i -v on the files to inspect the information and see if there's anything different. The only thing that came back as different are the multipliers, so I filtered for that.

I'm assuming that the example in the DB (the first one in this list) is in daylight. Most of mine are in normal daylight too, although some are in the shade. Shouldn't the multipliers be similar? Could it be that different D60 cameras reported different multipliers?

Is it possible that the multipliers are user-set? And perhaps darktable is or isn't taking these into account, but other raw editors are... or aren't?

Here's the list (daylight is the same for all, but the camera multipliers are different):

Filename: Canon - EOS D60 - RAW (3 2).CRW
Daylight multipliers: 2.799998 0.919642 1.004551
Camera multipliers: 1989.000000 758.000000 832.000000 756.000000

Filename: CRW_1456.CRW
Daylight multipliers: 2.799998 0.919642 1.004551
Camera multipliers: 2695.000000 824.000000 977.000000 817.000000

Filename: CRW_2869.CRW
Daylight multipliers: 2.799998 0.919642 1.004551
Camera multipliers: 2603.000000 821.000000 901.000000 815.000000

Filename: d60-additional/CRW_1508.CRW
Daylight multipliers: 2.799998 0.919642 1.004551
Camera multipliers: 2070.000000 830.000000 1026.000000 823.000000

Filename: d60-additional/CRW_1551.CRW
Daylight multipliers: 2.799998 0.919642 1.004551
Camera multipliers: 2178.000000 830.000000 1118.000000 823.000000

Filename: d60-additional/CRW_7289.CRW
Daylight multipliers: 2.799998 0.919642 1.004551
Camera multipliers: 2051.000000 833.000000 1307.000000 826.000000

Filename: d60-penguins/CRW_4919.CRW
Daylight multipliers: 2.799998 0.919642 1.004551
Camera multipliers: 2287.000000 827.000000 1002.000000 820.000000

Filename: d60-castle/crw_6154.crw
Daylight multipliers: 2.799998 0.919642 1.004551
Camera multipliers: 2654.000000 822.000000 915.000000 815.000000

Filename: d60-castle/crw_6155.crw
Daylight multipliers: 2.799998 0.919642 1.004551
Camera multipliers: 2766.000000 821.000000 911.000000 815.000000

[jenshannoschwalm]

I think we already have the underlying problem, as @johnny-bit mentioned, the coefficients should not be 1,1,1 !

So either the decoder reads wrong numbers or the file is wrong. Could it be, you used other software on those files, that might write a modified raw file?

On the phone so I can't test myself but does exiftool give any hint here?

[johnny-bit]

Exiftool output differs - the sample from raw.pixls.us HAS WB RGGB Levels field while @garrett's files don't.

for example here's one of garett's images:

$ exiftool CRW_1456.CRW | grep WB
WB RGGB Levels Auto             : 2178 830 823 1063
WB RGGB Levels Daylight         : 2224 828 821 1012
WB RGGB Levels Shade            : 2433 825 818 935
WB RGGB Levels Cloudy           : 1626 959 951 1829
WB RGGB Levels Tungsten         : 1848 871 864 1424
WB RGGB Levels Fluorescent      : 2520 822 815 909
WB RGGB Levels Flash            : 2695 824 817 977

and here's RPU sample:

$ exiftool Canon\ -\ EOS\ D60\ -\ RAW\ \(3\ 2\).CRW | grep WB
WB RGGB Levels Auto             : 1989 758 756 832
WB RGGB Levels Daylight         : 1856 784 781 945
WB RGGB Levels Shade            : 1976 759 757 848
WB RGGB Levels Cloudy           : 1378 937 933 1774
WB RGGB Levels Tungsten         : 1635 867 864 1439
WB RGGB Levels Fluorescent      : 2004 741 738 804
WB RGGB Levels Flash            : 1742 828 825 1273
WB RGGB Levels                  : 1989 758 756 832

[jenshannoschwalm]

So I guess we know why it's greenish. Maybe there was a camera firmware change later that includes that data?

Or exiftool or other software had been used wrongfully writing crap? I did that a long time ago on some images.

We could ensure data via adobe-coeffs I guess. You can assign me for 3.8 as there is more stuff for that with dng files pending if you want.