Add support for HDR gain maps

[gregbenz]

Is your feature request related to a problem? Please describe.

HDR (high dynamic range) is a significant leap forward in image quality on supporting monitors.

One of the challenges is to create an image which shows as an ideal HDR where possible, but safely falls back to showing SDR (standard dynamic range) where not supported. A new "gain map" standard (https://helpx.adobe.com/camera-raw/using/gain-map.html) offers this and is already supported by Chrome Canary v116 (including both MacOS / Windows desktop clients and the Android 14 beta). Enable the dev flag for gain maps via chrome://flags/#gainmap-hdr-images.

When images are reprocessed (such as by ImageMagick for posting on a Wordpress site via the media library), it is important that the gain map meta data and auxiliary image are retained. And if the image is resized/cropped, the secondary image should be altered in a consistent way.

When a JPG with a gain map is processed with ImageMagick now (ImageMagick 6.9.12-86 Q16 x86_64 17814), the primary image metadata (XMP-hdrgm) is retained, but the secondary image is discarded.

Describe the solution you'd like

• Metadata and color profile of the base image are retained
• Auxiliary image is retained, including its metadata
• Auxiliary image is rescaled as needed to match any alteration of the primary image. This should be the same percent reduction (as the gain map itself may be at 1/2 or 1/4 resolution of the original to save space in the file).
• Solution should at a minimum support JPG gain maps encoded with an SDR base image (as this offers full backwards compatibility). It should ideally support AVIF gain maps (as this offers much smaller files with AVIF). Gain map support for JXL may be helpful, though the format lacks critical support to be important in the foreseeable future.

Describe alternatives you've considered

Direct posting of the image without support for derivative assets like thumbnails or use of the media library. This could broadly affect a wide range of sites managing such images, so lack of support is very limiting and would require exporting multiple versions of images (if the platform even supports bypassing asset management which may reprocess the file).

Additional context

See https://gregbenzphotography.com/hdr/ for examples, monitor tests, and more info on HDR images.

See https://helpx.adobe.com/camera-raw/using/gain-map.html for sample gain map images in JPG, AVIF, and JXL formats.

[snibgo]

I can't find any image files with gain maps on the Greg Benz site, or elsewhere. Can you point to some?

When an image has an associated gain map (GM), I can see that cropping and resizing the GM in parallel to the main image would be useful. For ImageMagick (IM), one difficulty is that the architecture is designed to process individual images, or lists of images. IM doesn't have a concept of "associated" images.

Another difficulty is: why limit parallel operations to cropping and resizing? How about general -distort operations? Sharpening? And many others?

Perhaps as a first step, IM could include an option to extract a gain map from a file, adding the gain map to its usual list of images. It could have another option to insert a gain map into a file. It would also read/write the GM metadata. The next step is the fairly simple image processing: given 2 out of 3 images (SDR, HDR, GM), make the third.

[gregbenz]

@snibgo There are some gain map examples under the Adobe link (direct download is https://www.adobe.com/go/gainmap_sample_photos).

Your question on other operations is a good one. I would say the support I outlined above would probably cover most needs (certainly the most critical ones for sharing images on the web). However, those additional operations would likely be useful to others (especially sharpening if some service wants to use that to offset softness caused by resizing). Personally, I would probably not invoke such options. My emphasis is simply on derivative versions of a finished image, especially for use with the WordPress media library.

I would imagine that internally extracting the gain map, scaling/cropping, retaining HDR metadata, and inserting back into the main image would work well. The scaling should be proportional rather than absolute pixels, as the specification allows for gain maps to be lower resolution than the primary image (they get scaled by the viewer to match the primary image, which ought to be a very good compromise in the interest of size reduction outside of some edge cases with high frequency detail in the HDR range). If the tool works that way internally, I would say it would be important that the tool automates that step rather than require the external management of that process to (1) simplify implementation, (2) ensure widespread adoption, and (3) avoid mistakes during 3rd-party implementation.

Auxiliary images seem to be showing up in other places, the iPhone appears to use it for at least some HEIC images: https://gist.github.com/kiding/fa4876ab4ddc797e3f18c71b3c2eeb3a. There are also examples of alpha channels like depth masks. To be clear, I don't personally see an obvious use case for HEIC / depth masks those with ImageMagick (as those are probably going to be used for inputs to editing tools like Photoshop), but just want to note there seems to be an increasing list of potential uses for associated images.

[snibgo]

Thanks for the samples.

IM has an option for extracting depth maps, "-define heic:depth-image=true", but it can't write depth maps into files, and doesn't treat then as "associated" images, to process in parallel with main images.

I suspect HDR will increase in popularity. The "gain map" system seems to be a useful bridge between SDR and HDR.

Parallel processing of associated images could be very messy. However, something like this might be feasible (bash syntax):

magick \
  -define hdr:read-gain-map=true \
  in.jpx \
  -crop 2000x1000+100+200 +repage \
  -resize 600x600 \
  -define hdr:write-gain-map=true \
  out.jpx  

This would read images from in.jpx, including GMs. Each GM would be automatically resized to the same size as its parent image. -crop will crop all images (including GMs) to the same dimensions, then -resize will resize all images to fit in a 600x600 box. Finally, each image with its GM is written to out.jpx.

When IM reads the GM, it records any GM metadata as image properties. IM would also record the scale of the GM (eg 25%). When writing the GM, IM resizes the DM by the same scale, and writes the associated metadata.

Under this scheme, -crop and -resize etc would not need any modification. We would need changes to the coder for jpx, and coders for any filetype that might contain GMs.

How does that sound?

[fmw42]

If the gain map is pixel data (same size as input), then could it not be put into a meta channel?

[gregbenz]

If the gain map is pixel data (same size as input), then could it not be put into a meta channel?

What's a beta channel?

One thing to note with the spec is that the child metadata varies by format, at least in the Adobe samples. In some cases (JPG if I recall correctly), critical gain map metadata is in the auxiliary image, so that probably needs to remain (not sure if the spec allows flexibility between that and the parent image metadata). There is always at least the gain map version in the parent (for quick parsing), but that may be it for some images.

[fmw42]

meta channel (not beta channel) is an extra channel apart from RGBA for example. TIFF supports extra channels that it uses for example as masks. But they can be used for anything. A depth channel could be stored in Imagemagick as a meta channel, for example. I am not sure how it is stored now in HEIC images.

[snibgo]

fmw42 wrote:

If the gain map is pixel data (same size as input), then could it not be put into a meta channel?

Ah, yes, thanks, I think that's a better idea. Using meta channels would mean the GM would have to be stored in memory at full scale, ie the same size as the main image.

The "Gain Map Specification (1.0 draft 12, May 2023)" https://helpx.adobe.com/content/dam/help/en/camera-raw/using/gain-map/jcr_content/root/content/flex/items/position/position-par/table/row-3u03dx0-column-4a63daf/download_section/download-1/Gain_Map_1_0d12.pdf says the GM will have either 1 or 3 channels. Hence there would be 1 or 3 meta channels.

IMagick is an interface between PHP and ImageMagick. (It isn't developed or supported by the same people as ImageMagick.) When ImageMagick has the capability to read and write GMs, IMagick could inherit that capability.

I don't have access to an HDR display. If I do the development work, I would rely on other people for testing.

[gregbenz]

@snibgo I would be happy to help test any gain maps you convert. You can reach me directly via gregbenzphotography.com/contact/

The Adobe AVIF images would be a also good option to test on an SDR system. They are built with an HDR base image, so they will show as tone mapped if gain maps are not supported and show the proper SDR image if the gain map is used. So if the converted AVIF matches the source AVIF on your screen, the gain map should be ok (the JPGs are built with an SDR base image and therefore would show the same result on your SDR display whether the gain map was valid or not).

The Adobe materials include an image browser to view gain maps, which is a helpful reference.

[gregbenz]

meta channel (not beta channel) is an extra channel apart from RGBA for example. TIFF supports extra channels that it uses for example as masks. But they can be used for anything. A depth channel could be stored in Imagemagick as a meta channel, for example. I am not sure how it is stored now in HEIC images.

Ah, makes sense. I'm used to referring to it as "alpha" channel (or "layer mask"), per Photoshop lingo.

[snibgo]

Update on this:

The supplied sample JPEG files with gainmaps are stored in JPEG MPF (Multi-Picture Format) files. (Similarly JPEGs with depth maps.) Exiftool can extract the multiple images, but can't write them.

So a subproblem is to read and write JPEG MPF files. These have multiple instances of SOI/EOI markers.

It would be useful if IM had a feature that could read all the images in the file. Similarly, when IM has a list of images, it should be able to write all these into a single JPEG MPF file.

A complication is that within the first instance of SOI/EOI, there is (or should be) an APP2 segment labelled "MPF", with pointers to the images.

If we are reading a JPEG MPF and splitting it into separate images, we shouldn't save that APP2 MPF segment. Similarly, when we are writing multiple images to a JPEG MPF, we need to create a APP2 MPF segment and write it to the appropriate place in the file.

As far as I can see, jpeglib etc has no facility for correctly handling MPF. It simply reads the first image (in the first SOI/EOI markers), including the APP2 MPF segment, and ignores the other images. I am happy to be corrected on this.

The best solution would be for jpeglib etc to handle MPF. Failing that, IM could have a fairly simple coder to read and write JPG MPF files.

Thoughts, anyone?

[madmanchan]

Relevant CIPA document on Multi-Picture Format: https://www.cipa.jp/std/documents/e/DC-X007_E.pdf
Note that while gain maps are an exciting direction for HDR imaging and formats, this is a swiftly moving area with many changes happening. There is an effort to standardize the approach, but that will take time. MPF is likely to be the mechanism used for JPEG (mostly for lack of a better option), but details like metadata and the math may change.

[gregbenz]

@snibgo Does jpeglib or the info from madmanchan cover what would be needed for IM to add support for JPG gain maps? (I can't offer much on your question on jpeglib, outside my expertise).

Anything I can do to help test?

[gregbenz]

JPG gain maps (aka "Ultra HDR JPG") are now supported in 4 browsers:

• mainline Chrome v116 (enable via chrome://flags/#gainmap-hdr-images).
• Brave v1.57.47 (via brave://flags/#gainmap-hdr-images)
• Opera v101 (via opera://flags/#gainmap-hdr-images)
• MS Edge Canary v117 (no dev flag needed)

They look amazing on all these browsers under a variety of SDR and HDR viewing conditions (and still look great as SDR on other older browsers that lack support since this format is automatically backwards compatible, the gain map data for HDR is just ignored when not understood or not applicable).

Sample gain maps: https://www.adobe.com/go/gainmap_sample_photos

[fmw42]

Gain maps are being researched but have not been implemented in IM. The first step of supporting MPO JPGs has been implemented. That is as far as I know has been done. It is not an easy implementation for Gain Maps.