DNG GainMap opcode support

[paolodepetrillo]

See this Darktable bug for background info and references.

I'm trying to figure out how to add this to Darktable, for vignetting correction for RAW DNGs from the Google Camera app on my Pixel 4a, and I'm not sure if it should be done in rawspeed or in the darktable pipeline.

I have an implementation of the opcode in this branch that seems to work with the Pixel 4a dngs. I added the GainMap opcode to the existing function that implements the OpcodeList2 tag for lossy DNGs. So that I could test it in darktable, I temporarily modified DngDecoder to apply the OpcodeList2 processing to all DNGs, not only lossy DNGs, and also moved the cropping to after the opcodes are applied - DNG spec says opcodes "should be applied to the raw image, as read directly from the file".

I'm not sure how it would be possible to make this change without breaking any existing darktable edits. Without this change, a RAW DNG is output to darktable as the original pixels, black level, and white level. But with this change, it isn't really the actual RAW anymore, it's just a linear 16 bit integer image with black level 0 and white level 65535. This is fine for a newly imported image, but any existing image would already have the original black and white levels set in the rawprepare module which would no longer be valid. Maybe there could be an option that determines whether or not OpcodeList2 should be applied to a raw dng?

[LebedevRI]

Best answer: just profile the camera for lensfun.

This indeed won't really fit. Indeed, we don't want to always apply OpcodeList2.
I guess you'd need to create a new rawspeed::RawImage from the image at that stage of dt's pipeline,
and somehow call RS's dng opcode handling code to apply GainMap only.
But can we even apply just that one, or will you need to apply all of the OpcodeList2?

darktable used to intentionally ignore all this oh so special but proprietary DNG/DCP/LCP stuff.
Did that change?

[jenshannoschwalm]

Yeah, a lot of DNG stuff is proprietary and lot's of generated files are just buggy. BUT:

Couldn't rawspeed espose a struct holding all data relevant for gainmap? The dt rawspeed imageio would be able to get that struct and set a pointer in a new dt_image_t element. That would available for all modules at runtime and there would be no need to keep it somehow.

rawprepare would be the dt module to go. (Just define a new version and you would keep old edits). As all data defining the cropping from sensor and white/blacklevels are handled there you could use the gainmap with minor x/y transformation maths.

I personally don't care about this feature at all but there are other opcodes / data that might be used if exposed properly, hotpixels, lens ... So if you want it up-and-running a lensfun profile would be 2hours work :-)

[paolodepetrillo]

The GainMap is more general than the lensfun vignetting model, but the lensfun model might be close enough that the extra complexity of the GainMap isn't worth implementing. The GainMap does not have to be radially symmetric, can be off center, and is applied to each color channel separately. Don't know exactly how the phone is generating these, but I'm guessing it's probably based on a combination of the focus distance and lens shifting by the optical image stabilizer. Here's a few examples of what they look like, from the Pixel 4a. I plotted the map for each channel, and the values on a diagonal line across the map:

In these dngs the GainMap is the only opcode in OpcodeList2, and there is a lens distortion correction in OpcodeList3 (after demosaic). Don't know if other devices / apps would use other opcodes. The ART implementation just looks for the GainMap and ignores any other opcodes.

I had started working on another implementation that was entirely in darktable without requiring any changes to rawspeed. It would just get the raw OpcodeList2 contents at the same time as it was reading all of the other tags and put it in dt_image_t, and do its own parsing of the opcodes in C without trying to reuse the existing rawspeed code. It seemed like it might be easier to do it in rawspeed because the code for parsing the opcode and iterating over all of the pixels was already there and working, and the correction would be done once at image loading time rather than being done within the DT pipeline. But now I'm thinking it would be better to just add it as an option to the rawprepare module.

It looks like there is a crop based on DefaultCropOrigin and DefaultCropSize tags that happens in rawspeed before the image gets to darktable rawprepare, and the GainMap should be applied to the image before that crop. But it should be be easy enough to read the crop tags in DT and compensate for it.

[jenshannoschwalm]

If you do it in rawprepare you would have to take care of the roi (region of intereset) but cropping info is in dt_image_t