How can I incorporate HDR gain map from JPEG / HEIC?

[AttilaTheFun]

The iPhone for many years has had a "Smart HDR" feature which created "fake" HDR images by storing a HDR gain map in the image metadata of HEIC and JPEG images.

There is some discussion here about how to read / write these gain maps on iOS here:
https://gist.github.com/kiding/fa4876ab4ddc797e3f18c71b3c2eeb3a

I would like to read out this gain map and incorporate it with the image data using vips to create a true HDR image, and then save it out to a format that natively supports it like AVIF.

It sounds pretty straightforward to extract the gain map as a bitmap, but I'm not sure the best way to apply this to the vips image.

From that thread users suggested using a YCbCr color space and then multiplying the Y components by the bitmap values. I'm not sure how I would do that with vips though.

[jcupitt]

Hi @AttilaTheFun,

I think libvips can read gain maps out of HEIC images -- just set the page param. Do you have a sample image somewhere?

[AttilaTheFun]

Hi @jcupitt ! This is one of the sample images provided with iOS simulators:
https://drive.google.com/file/d/1CpgVmp_lkhULvokPfT9v5v6RUzRoFK5A/view?usp=share_link
(I uploaded it to Google Drive because I wasn't sure if Github would convert it.)

As you can see in this screenshot, iOS recognizes it as an HDR image even though it's only 8-bit.

[jcupitt]

I'm not sure that's got a gain map. I see:

$ heif-convert heic-hdr.heic x.png
File contains 1 images
Written to x.png
Depth image written to x-depth.png

So there's depth (from the LIDAR), but not gain. It looks like a regular 8 bit image with a P3 profile.

strukturag/libheif#479 (comment)

Says images with a gain map should look like this on decode:

$ heif-convert input.heic out.jpg
File contains 1 images
Written to out.jpg
Auxiliary image written to out-urn:com:apple:photo:2020:aux:hdrgainmap.jpg

[jcupitt]

I tried with git master libheif as well, and there's no gain map reported.

[AttilaTheFun]

Hmm weird. Maybe the HDR tag apple is showing just indicates that it was produced by combining multiple exposures into one regular image?

Let me see if I can find one of these images thar actually includes the gain map. It's possible it needs to be one of the "Smart HDR" images produced by the iPhone XS and later.

https://www.trustedreviews.com/explainer/what-is-ios-smart-hdr-4264112

[gregbenz]

Initial support for HDR JPG with gain maps in libvips: #4645
This includes support for JPG gain maps (a future update to libultrahdr will expand support to AVIF / HEIF)