darktable-gen-noiseprofile incorrectly reads extended ISO settings from Z 6ii

[TekCastPork]

Introduction

When trying to generate a noise profile for the Nikon Z 6ii, the darktable-gen-noiseprofile script uses incorrect ISO values for any images taken using the Lo and Hi extended ISO ranges. Everything else seems to work though, just the ISO is not the expected value for each image.

To Reproduce

1. Extract input images into a folder/directory
2. Run the darktable-gen-noiseprofile script on input images
3. Wait for script completion
4. Check noise results pdf and presets JSON file (the console output can also be used to check for reproduction when it detects the images)

Input Images.zip (970MB of raw .NEF images)

Expected behavior

ISO values for expected behaviour come from darktable itself as well as EXIF metadata when exiv2 is manually executed:

• DSC_5122 should have an ISO of 50 (Lo 1.0)
• DSC_5123 should have an ISO of 63 (Lo 0.7)
• DSC_5124 should have an ISO of 79 (Lo 0.3)
• DSC_5153 should have an ISO of 64508 (Hi 0.3)
• DSC_5154 should have an ISO of 81275 (Hi 0.7)
• DSC_5156 should have an ISO of 102400 (Hi 1.0)
• DSC_5157 should have an ISO of 204800 (Hi 2.0)

Actual behavior

Script PDF, presets.json, and console output report the following:

• DSC_5122 has an ISO of 48
• DSC_5123 has an ISO of 52
• DSC_5124 has an ISO of 56
• DSC_5153 has an ISO of 172
• DSC_5154 has an ISO of 176
• DSC_5156 has an ISO of 180
• DSC_5157 has an ISO of 192

Platform

• darktable version : 4.0.0 (built using latest stable release source, no additional cmake flags)
• OS : Linux - kernel 5.13.0-35-generic
• Linux - Distro : Ubuntu 20.04 LTS
• Processor : Intel Core i7-4500U
• Memory : 7.5GiB
• Graphics card : Intel HD Graphics 4400
• Graphics driver : Unknown

Additional Notes

I did have to apply the fix to the script as described here to get it to run, but otherwise it is the darktable-gen-noiseprofile script that gets built from the source.

Also, I am a big novice when it comes to Linux, as I am a primarily Windows user. My "on-the-go" laptop runs linux though and that is where I am trying to generate this noise data from as it seems the Windows installation does not come with the necessary scripts.

[github-actions]

This issue did not get any activity in the past 60 days and will be closed in 365 days if no update occurs. Please check if the master branch has fixed it and report again or close the issue.

[miroslavfikar]

I tried to work on noise profiles of Z 6_2 yesterday and noticed the same behaviour, DT 4.2.1, ubuntu 22.04, compiled from sources. Actually, I also got segfault when calling darktable-cli during the run and the script did not finish correctly. I can provide more details if needed.

[kmilos]

Please run exiv2 -pa -u foo.NEF and exiftool -a -u -s -g1 foo.NEF and check if there are any tags/data that are a better match for Lo/Hi ISO?

This logic might need to be reverse engineered and updated for the newer Z firmware:

darktable/tools/noise/subr.sh

Lines 289 to 308 in 9627a68

	[Nn][Ii][Kk][Oo][Nn]*)
	# Read "Exif.Nikon3.*" before "Exif.NikonIi.*":
	# 1. "Exif.NikonIi.*" are bytes, not even
	# shorts, so they're smaller than other
	# keys.
	# 2. That looks like versioned nodes:
	# "Nikon2" vs. "Nikon3".
	iso=$(get_exif_key "$file" Exif.Nikon3.ISOSpeed)
	if [ -z "$iso" -o "$iso" = "0" ]; then
	iso=$(get_exif_key "$file" Exif.Nikon3.ISOSettings)
	fi
	if [ -z "$iso" -o "$iso" = "0" ]; then
	iso=$(get_exif_key "$file" Exif.NikonIi.ISO)
	# read hi/low iso setting
	ciso=$(echo $iso | cut -d' ' -f2)
	if [ "$ciso" = "Hi" -o "$ciso" = "Lo" ]; then
	iso=$(echo $iso | cut -d' ' -f1 )
	fi
	fi
	;;

[kmilos]

Actually, I also got segfault when calling darktable-cli during the run and the script did not finish correctly.

Probably a separate issue?

[miroslavfikar]

The segfault was resolved - profiles were treated using development version of DT but dt-cli was called from subr.sh using the production version. Once $PATH was modified, it run smoothly.

[miroslavfikar]

It seems to me that the wrong read of unusual ISO numbers lies in the command switches of exiv2 in subr.h at line 260:

darktable/tools/noise/subr.sh

Line 260 in 9627a68

exiv2 -K "$key" -Pv -b "$file" 2>/dev/null | sed 's/ *$//g' || :

I had better luck with translated print: exiv2 -K "$key" -Pt -b "$file" 2>/dev/null | sed 's/ *$//g' || :

This reads correct ISO values. But I did not check the whole procedure, something broke with my development DT.

If needed, I can upload the NEF files somewhere.

[miroslavfikar]

I finished testing with changed subr.sh and switch -Pt. Everything works as expected. Please consider testing it with other camera models.

[github-actions]

This issue has been marked as stale due to inactivity for the last 60 days. It will be automatically closed in 300 days if no update occurs. Please check if the master branch has fixed it and report again or close the issue.

[github-actions]

This issue was closed because it has been inactive for 300 days since being marked as stale. Please check if the newest release or nightly build has it fixed. Please, create a new issue if the issue is not fixed.