DNG with "Embed Fast Load Data" not rendered the same as plain DNG or NEF

[jannefoo]

Version: 5.4
Branch: 5.4
Commit: a5e8eb9
Commit date: 2018-03-20
Compiler: gcc 7.3.0
Processor: generic x86
System: Windows
Bit depth: 64 bits
Gtkmm: V3.22.0
Lensfun: V0.3.2.0
Build type: release
Build flags: -m64 -mwin32 -mthreads -Wno-aggressive-loop-optimizations -std=c++11 -mtune=generic -Werror=unused-label -fopenmp -Werror=unknown-pragmas -Wall -Wno-unused-result -Wno-deprecated-declarations -DNDEBUG -O3
Link flags: -m64 -mthreads -static-libgcc -mtune=generic -s -O3 -fno-use-linker-plugin
OpenMP support: ON
MMAP support: ON

Files: https://filebin.net/mfvbadtonhy3s0d8

Below is preview of what I'm seeing. Two dng files made with dng converter 10.4.0.976 have vastly different output when saved with identical raw therapee settings. the original .nef files are saved correctly, also photoshop exports the dng files correctly, but not raw therapee.

[Beep6581]

https://filebin.net/b556ghmhwf2mxky3

When I converted your NEF to DNG using Adobe DNG Converter 10.4 without using "embed fast load data", it renders identically as the NEF in RawTherapee-5.4-520-gd73e542.

When "embed fast load data" is enabled, there is a difference.

[jannefoo]

strange, because out of batch of 18 images, only two were affected.

[Beep6581]

Setting all black levels to -200 in RT for the "embed fast load data" file leads to identical results.

There was some work ~recently done to match DNG to non-DNG. Maybe the embedded preview is messing with the logic? Ping @heckflosse @agriggio @iliasg

Metadata diff: http://www.mergely.com/QG6ARgwg/

[afontenot]

Hi, I'm coming here from my issue, which was closed as a duplicate of this one. I'd very much like to be able to use RawTherapee as an alternative to Lightroom, but this issue is definitely a blocker for me. (In my case, the effect of enabling fast load data is even more disastrous than here.) Just for starters, I would have to recreate the DNGs for all my files, which would require digging out my raw files from backups, assuming I still have them all. (The whole point of converting to DNG is that it's supposed to be an "archival" format.)

Changing the black level on my images does not solve the problem.

But I'm not just here to complain, I'd like to look into the issue and see if I can write a fix for it. If anyone has any pointers on why the existence of fast load data might cause this problem, please let me know.

[afontenot]

A little bit of documentation. I'm sure the developers of RawTherapee know all this, but it might help someone else who drops by and ends up working on the problem.

"Fast Load Data" is a lossy-jpeg compressed full size image preview that can be embedded in a compliant DNG file as an IFD. (DNGs are based on TIFF.) The point of embedding such a preview is so you can show the user a full sized representation of the image without having to process the raw data. (Lightroom might currently be the only product that makes use of such previews.)

This is completely within spec. DNG 1.5 says:

The highest-resolution and quality IFD should use NewSubFileType equal to 0. Reduced resolution (or quality) thumbnails or previews, if any, should use NewSubFileType equal to 1 (for a primary preview) or 10001.H (for an alternate preview). DNG recommends, but does not require, that the first IFD contain a low-resolution thumbnail, as described in the TIFF-EP specification.

So there's no limit to the number of preview images that could be included in an in-spec DNG file. Recentish versions of Adobe's raw conversion tools add a full size jpeg to the file as a preview by default.

A quick skim suggests that Darktable handles this by immediately excluding any subsampled IFDs from their DNG parsing function: https://github.com/darktable-org/rawspeed/blob/develop/src/librawspeed/decoders/DngDecoder.cpp#L81

My hypothesis is that something in dcraw is getting confused; there are two full size images in the file, both probably jpeg encoded (lossless jpeg is used to compress the raw data). So it's pulling some critical bit of data from the preview IFD and that is fouling up the results. I couldn't tell with a quick look how this might be happening. RawTherapee contains a solid 20k+ lines of borderline undocumented dcraw code, and I may or may not have time to dig through it all.

(Incidentally, I do see the same decoding problem with the original dcraw.)

[agriggio]

you can try this and see if it helps:

diff --git a/rtengine/dcraw.cc b/rtengine/dcraw.cc
--- a/rtengine/dcraw.cc
+++ b/rtengine/dcraw.cc
@@ -6548,6 +6548,38 @@ guess_cfa_pc:
 	cblack[4] = cblack[5] = MIN(sqrt(len),64);
       case 50714:			/* BlackLevel */
                 RT_blacklevel_from_constant = ThreeValBool::F;
+//-----------------------------------------------------------------------------
+// taken from LibRaw.
+/* 
+  Copyright 2008-2019 LibRaw LLC (info@libraw.org)
+
+LibRaw is free software; you can redistribute it and/or modify
+it under the terms of the one of two licenses as you choose:
+
+1. GNU LESSER GENERAL PUBLIC LICENSE version 2.1
+   (See file LICENSE.LGPL provided in LibRaw distribution archive for details).
+
+2. COMMON DEVELOPMENT AND DISTRIBUTION LICENSE (CDDL) Version 1.0
+   (See file LICENSE.CDDL provided in LibRaw distribution archive for details).
+
+   This file is generated from Dave Coffin's dcraw.c
+   dcraw.c -- Dave Coffin's raw photo decoder
+   Copyright 1997-2010 by Dave Coffin, dcoffin a cybercom o net
+
+   Look into dcraw homepage (probably http://cybercom.net/~dcoffin/dcraw/)
+   for more information
+*/
+                if (tiff_ifd[ifd].samples > 1 && tiff_ifd[ifd].samples == len) // LinearDNG, per-channel black
+                {
+                    for (i = 0; i < 4 && i < len; i++)
+                    {
+                        double b = getreal(type);
+                        cblack[i] = b+0.5;
+                    }
+
+                    black = 0;
+                } else
+//-----------------------------------------------------------------------------
 		if(cblack[4] * cblack[5] == 0) {
 			int dblack[] = { 0,0,0,0 };
 			black = getreal(type);

[Beep6581]

Tested, compared Sony ILCE-7M2 ARW converted to DNG using Adobe DNG Converter 11.3, one with "fast load data" and one without. Both opened identically and correctly.

[Beep6581]

Putting this on the 5.7 milestone as it seems safe enough.

[Beep6581]

Samples for testing:
https://filebin.net/shaw3hhc1j0wsd8m