Panasonic S5: Underexposed images green/Black levels not read

[jdneumeyer77]

Describe the bug/issue

Underexposed images are very green for the Panasonic S5 raw files (rw2). For instance, I was capturing the Milky Way by way of stacking. So I set up the proper exposure for iso3200 with a test shot, but switched it to iso640 before taking the exposures for the stack to further reduce noise (640 is the dual-gain point for this sensor).

While investigating why my files were so green I saw this ticket: #9285 which seems similar. Based on that I used exiftools to find the Black levels and that seems to have fixed.

Expected behavior
The expected behavior is shown in the screenshots without the green cast and much darker exposure overall. The exposure was pushed 5 stops on each image.

Dark and overly green image

Correct Behavior after setting the black levels

Looking at rawspeed has support in cameras.xml for it and looks to set the default to 509. Glancing at the rawspeed decoder, it attempts to read the values from the exif properly, but must fail for some reason.

Here's the exif snippet:

exiftool P1010852.RW2 | grep -i Black
Black Level Red                : 512
Black Level Green             : 512
Black Level Blue                : 512

I've attached a test image and I license it under under CC0 for testing purposes.
samples.zip

To reproduce bump the exposure to +6ev and to correct use 509 or 512 as the black levels in raw black/white levels module. Happy to help and dig further once I have more time.

Platform

• darktable version : 3.7.0.r699.g081b924ed5-1
• OS : Linux 5.10.53
• Linux - Distro : Manjaro
• Memory : 64gb
• Graphics card : Nvidia Ti2060
• Graphics driver : Nvidia
• OpenCL installed : yes
• OpenCL activated : not available / probably a primus issue...
• Xorg : 1.20.13
• Desktop : XFCE
• GTK+ : 4.2
• gcc : 11.1
• cflags : unknown/defaults
• CMAKE_BUILD_TYPE : unknown/defaults

[johnny-bit]

Is the issue present on samples on raw.pixls.us?

What's weird is that on your green shot it looks like Black Levels are set to 527, the defaults are as you say 509 while exiftool says 512.

@LebedevRI - what do you think?

[jdneumeyer77]

The sample file exif says 510 for Black Levels, but when I open it in DT it says 525 (after disabling my auto apply workaround). It almost looks like something is adding 15.

I've attached a diff between the exif output from one of my files and the sample.
samplefile.exif.txt
myfile.exif.txt
exif.diff.txt

[jenshannoschwalm]

exiftool might do some magic before display, the relevant data should be tested to be correct via exiv2 .

[jdneumeyer77]

Using exiv2 it doesn't seem to have those entries Black Level {Red, Green, Blue}. The exiv2 doesn't seem to have raw documentation for the tags or their website is just confusing. I see the one for Maker Notes, but the tag id doesn't match the output file.

sample.exiv2.txt

Exiftools has some documents on the raw. It matches the tag Ids being read by rawspeed.

Based on the Exiftools docs though I think the problem is that it's reading the entry values as 32 bit unsigned ints instead of 16 bit unsigned ints.

    const auto getBlack = [&raw](TiffTag t) -> int {
      const auto val = raw->getEntry(t)->getU32();
      int out;
      if (__builtin_sadd_overflow(val, 15, &out))
        ThrowRDE("Integer overflow when calculating black level");
      return out;
    };

That seems to be where the 15 is coming from. 512 + 15 = 527. So it looks like it's overflowing either because it should be reading a 16 bit entry (not sure how getEntry(t)->getU32() handles reading more then it's supposed to). I'm rusty with C/C++ (I'm a Scala developer...), but also I'm not sure how __builtin_sadd_overflow handles being given a unsigned value when it handles signed values (gcc docs look poor). Because it does seem to be reading the right value yet still overflowing/adding to it.

Edit: actually I misunderstood the built in function. Why is it always adding 15 to each black level? The other points are still valid.

[jenshannoschwalm]

Isn't the getU32 here just wrong?

[jdneumeyer77]

@jenshannoschwalm I agree. I also don't understand why it's adding 15 to them which seems to causing problem with underexposed images. The affect on normally exposed images is very subtle. Similarly exposed as above, but at higher ISOs the green is subdued yet still there.

Should I just make an PR to switch to getU16 and remove the addition/overflow check? If there is any value checks that traditionally happen could you point me to an example?

[jenshannoschwalm]

I thought about it again, i think the U16 is ok and safe, also it's just as the docs suggest. Also i don't understand what an overflow check is here for. We might want to check for plausible values (must be >=0 and < whitze point).

In any case it's a rawspeed issue and @LebedevRI is not always reading here. So rise the issue there and why not a pr? If he thinks it's bad because of any reason we can't think of now - he will let you know :-)

[jdneumeyer77]

@jenshannoschwalm Fixed upstream in rawspeed. Any plans to refresh/update the rawspeed git submodule soon or should I put in a PR?

[LebedevRI]

Not yet