No support for HDR gain maps (transcoded to SDR-only)

[gregbenz]

Describe the bug
When a JPG image with an HDR ("high dynamic range") gain map is transcoded, the gain map is stripped. This results in significant degradation of the image (loss of support for enhanced HDR display).

Modern displays support brighter and more vibrant HDR images. This includes most smart phones, Apple computers, and a general trend towards widespread support on the web. More info at https://gregbenzphotography.com/hdr and https://gregbenzphotography.com/hdr-photos/jpg-hdr-gain-maps-in-adobe-camera-raw/. Instagram also supports HDR gain maps and sample images may be easily viewed with the iOS or Android app on most smart phones with comparison images at https://www.instagram.com/gregbenzphotography/

To Reproduce
Upload an image with a gain map to WordPress (using GD) and the resulting image in the media library will no longer no longer have a gain map.

Expected behavior
The image should retain a proper gain map, even after any resizing / compressing / cropping.

Actual results
The transcoded JPG contains only the SDR ("standard dynamic range") base image, not the required auxiliary image / metadata to support HDR display.

Environment (please complete the following information):
Any version of GD will fail.

Additional context
The Google libultrahdr library fully supports proper transcoding of these images: https://gregbenzphotography.com/hdr-gain-map-gallery/

[cmb69]

Somewhat related to #136.

[gregbenz]

@cmb69 Certainly keeping an ICC profile is beneficial to any image (including JPG gain maps which typically have a profile in the base SDR but not typically in the auxiliary gain map image), but beyond that I'm not clear what connection you see on these issues.

[ Note that longer term (later in 2025), AVIF images with a gain map should become a key format for sharing HDR as well. With AVIF, we can use CICP/NCLX to represent the color profile for web use. This is beneficial as CICP is 4 bytes, rather than 500-1000ish for a similar ICC profile. It also allows specifying an HDR EOTF like PQ or HLG (ICC profiles have no standard at this time for HDR). The CICP stuff is separate from the main consideration here for JPG gain maps, I'm just noting that we aren't far from a point where this more compact alternative to ICC will be a useful consideration for some file formats (a CICP tag can be added to an ICC profile as well for various reasons, but that approach would not reduce file size). ]

[cmb69]

The connection is that both are meta-data, and libgd ignores almost all meta-data so far (heck, even Exif Orientation is ignored). That means that developers using libgd need to cater to that themselves (until libgd supports the relevant meta-data; given that the linked issue has been reported more than ten years ago, I wouldn't hold my breath).

Upload an image with a gain map to WordPress (using GD) and the resulting image in the media library will no longer no longer have a gain map.

I suggest you report that to the WordPress developers; they might be able add support sooner than we can.

[gregbenz]

@cmb69 There are open requests here for GB, and for ImageMagick. As far as I know there is no path for WP media library support without at least one of them supporting gain maps - and ideally both (given both get a lot of use for WP).

Good to know on the metadata support, thank you.

[cmb69]

I haven't looked up where HDR gain maps are stored in JFIF files, but ICC color profiles can be handled in userland with plain PHP: get APP2 segment via getimagesize(), and write back using binary string manipulation. A bit fiddly, but is doable. And yes, that only caters to JFIF; but that is due to a limitation of getimagesize() which only reads JFIF APP markers; you could also read these (and other metadata) directly. And doing this is PHP instead of C shouldn't be a particular performance issue (contrary to image pixel manipulation).

Thinking about that, it might be best to not add support for (most) metadata to libgd, since that would drastically increase the API scope, and we're already barely focussing on image manipulation, due to the need to support yet another image format every other year.

Wrt HDR gain maps: these are to add support to image formats which don't allow to store HDR images directly. Now libgd supports AVIF and HEIF, and these formats are HDR ready; however, libgd doesn't support that, because we only support up to 8bits per color channel. If we ever extend that, it would make sense to read HDR gain maps, and apply these; and then figure out what to do with that additional information when writing back to a desired image format. However, as is it might not make much sense. Consider thumbnail creation: read JFIF including HDR gain map, resize in LDR, reapply the old HDR gain map when storing – would that even make much sense? The resizing already lost information. And if there is going to be more drastic image manipulation than resizing, keeping the old HDR gain map may even make no sense at all. This is, however, nothing libgd knows about up-front; only clients know what they are going to do with the image.

[gregbenz]

JPG gain maps store the auxiliary gain map per MPF. The Android XMP spec adds a GContainer header not required in the Adobe XMP spec. The more important ISO spec is at a draft international stage and therefore not available for review. However, it is not only supported by multiple providers, but the files are dual encoded with both Android, Adobe, or the undocumented Apple XMP - and ISO. So the ISO spec would be fully compatible with the published Adobe spec as far as the encoding of the image itself (the metadata for ISO moves from XMP to the image codestream).

[gregbenz]

The Google-backed open source C++ library libultrahdr contains all the code necessary to take an existing gain map and transcode it to another, with support for recompress, rescale, crop (as well as rotate/mirror).

Simply integrated it into libgd and using it when the source image has a supported gain map should be all that's needed (I don't if libgd supports a wider scope outside these gain maps and handling only gain map inputs with this path is probably ideal to avoid any risk of edge cases). This library supports both the widely used Android XMP standard, as well as the critical ISO standard (which is not yet official, but already supported by Android, Chrome, Adobe LR/ ACR, iOS, MacOS, etc). The mainline is feature complete and just needs a minor fix (which is complete/validated) to be merged to mainline. Within a few days, the mainline release should be fully ready for this use.

google/libultrahdr#254

ImageMagick has nearly completed integration of this library (a related PHP wrapper likely needs to be updated for this to be used by WP, but even when complete support in libgd will be ideal to support a wider range of WP hosting environments): ImageMagick/ImageMagick#6377

[gregbenz]

libultra does not yet support AVIF / HEIF, but the team has noted that it is on their roadmap. I would think that could ultimately be managed via libultrahdr as well.

[pierrejoye]

I am planning APIs to extract/insert meta data, raw, buffer. I am not keen at add to do our own parsing/handling but providing APIs to get the meta data from any image formats, the caller can then do whatever is needed with it, reapply then when required.

Most if not all libraries specialized in such meta data accepts the data as buffer, exif, iptc, etc. So that could be helpful already and trivial then.

Some formats have been smarter and have moved some of the meta data as part of their own codecs, as they affected the loading/saving/compression/etc of their own format. That should be the way :). The rest is a nightmare if not standardized.

I understand some could be used by gd, like orientation, but where do we stop? Especially as many are not standardized at all. Open for discussions for standard cases or cases where not reading and handling them will actually affect what or how GD should process an image. Please let me know if any cases like this exists and what we could do about it. And if possible without having to implement a full meta data parser in GD, which is something I would avoid :)

[pierrejoye]

See #1002

Initial metadata extraction/insertion. No parsing, but ability to pass it to a specialized library/sdk as needed. Feedback more than welcome.

About the specific case:
"The transcoded JPG contains only the SDR ("standard dynamic range") base image, not the required auxiliary image / metadata to support HDR display."

I never used HDR before, so I am not sure if that HDR auxiliary image needs changes or not based on what the image will be modified before saving and adding the HRD meta back. F.e. resizing or similar. But I suppose that's something that could be handled of the image.

If that HDR image is also a JPEG, it may work to load it once extracted as such, or? Pointers/refs welcome if needed

[pierrejoye]

Did a quick research but enough to bring some directions.

There is little to no chance that libgd will ever support HDR natively. It is entirely a display issue. Also, logically, resizing, cropping or similar will ruin the existing HDR map. It is not resilient even if low res.

Other graphic libs, including libjpeg, completely ignore HDR gain map

Google created libultrahdr, that wraps itself with libjpeg-turbo. This is something I could add separately, if enough projects or people need it.

But having the current JPEG codec handle this will not be done, that's opening a can we cannot handle.

A possible solution, for a case where user do operations on the image, would to apply these ops on both the HDR gain map jpeg low res and the SDR image, then use libultrahdr to combine them back and do all the maths.

If only resize, libultra has builtin primitives for basic ops.

[gregbenz]

@pierrejoye Thank you for looking into this. I'm not sure I understand some of your comments. What do you mean by it being a "display" issue?

Resizing, cropping, compressing, rotating, and mirroring are already very well supported for JPG gain maps with libraries using libultrahdr. This includes wasm-vips, Sharp, and libvips (ImageMagick also includes it under an optional flag when compiling). You can see a wasm-vips demo here: https://gregbenzphotography.com/test/wasm-vips-hdr-demo/. This WASM library will be the basis of support for client-side transcoding in WordPress v7.1, so that takes some pressure off the need for GD to support, but it likely does not address all transcoding needs (even after media library support is added, which is not currently in the v7.1 beta, just in the editor).

I have a local demo of support via Sharp as well if that is helpful, but it is not available online as I'm not using Node JS on my server. The results there are excellent like all the others, but performance is about 2x wasm-vips due to current limitations of libjpeg-turbo in that context, so I would assume similar performance increases with GD relative to my WASM demo.

libultrahdr is probably an ideal way to add support. Not only is it very well tested at this point, the road map will include support for AVIF gain maps (much smaller files / higher quality) and HEIF gain maps (primarily important for supporting the native gain maps created with iPhone and you'd likely transcode to JPG/AVIF gain maps with such a source).

Routing gain maps through libultrahdr has been the approach used in these other libraries (while other JPGs go through existing code paths, which. ensures maximum support for JPG overall). There is a probe feature in libultrahdr and there are other ways you could test the source.

[pierrejoye]

@gregbenz the "display issue", I should have formulated it as "task" instead. For what I see it is mostly done by the gui layers (browser, or other apps). The hdr map is a floating point rgb image (the gray level to boost/reduce areas). The hdr data is applied most of the times by the display layer, using gpu f.e. when available, as gpu are extremely effective for such floating ops.

gd supports only 8bit per channel.. That's the biggest challenge. Vips support various floating point or custom format data as image buffer. That allows it to do hdr map operations.

Specifically for GD, adding floating point rgb(a) to the current gdImagePtr is technically possible but suboptimal. GD 3 could have it and use the more advanced surfaces and ops to do resizing, composition or similar operations. Not a small task but totally possible and cleaner than doing it in 2.x.

Alternatively, a separate set of apis using liibultrahdr would be possible too. I am not sure how watermarks or other "destructive" operations could affect the hdr composition tho'. It would allow to do whatever is needed on the sdr, then use the ultrahdr to api to match the size/flip/etc but the altered parts of the sdr will have display disruption.