Add libultrahdr - #4645
Conversation
half to float seems to be failing?
|
It seems to work -- I see: Then: 
I can adjust the sliders to see detail in the exhaust: 
Woo! |
|
Cool that it already seems work (good enough for my use case), should scaling from UltraHDR -> UltraHDR already work? If so I'd like to give it a try if it works for previews and scaled images in Nextcloud although I'm sure how soon I can test it, given how I'll probably have to build up the stack from source and get it into docker somehow (IIRC it's libvips -> bimg -> imaginary -> docker image). |
and attach the gainmap as metadata
|
@cromefire I think we'll probably need Yes, building it will be a pain, you'll have to compile libultrahdr from source I think. |
|
Also, this branch is just for testing, I doubt it'll get merged to libvips master. Right now at least, the memory behaviour is pretty bad. |
So that means the upstream libraries would also have to build explicit support for scaling ultrahdr, right? The building I can handle, just have to have a few hours time for it... |
|
FYI - it appears that a significant refactor of libultrahdr is on horizont, as Android 16 supports gainmaps also in HEIC files, and current library is pretty hardcoded to JPG. I haven’t seen any public mention of this yet, but it’ll probably be expanded work that you can see here: https://github.com/ittiam-systems/libultrahdr/tree/experimental |
and shrink-on-load
|
Tiny python test prog: #!/usr/bin/env python
import sys
import pyvips
image = pyvips.Image.uhdrload(sys.argv[1])
gainmap_buffer = image.get('gainmap')
print(f"{len(gainmap_buffer)} bytes of gainmap data")
gainmap = pyvips.Image.jpegload_buffer(gainmap_buffer)
print(f"gainmap = {gainmap}")
profile = image.get('icc-profile-data')
print(f"{len(profile)} bytes of ICC profile data")
with open("profile.icc", "wb") as f:
f.write(profile)
print(f"profile written to profile.icc")
for name in ["gainmap-max-content-boost",
"gainmap-min-content-boost",
"gainmap-gamma",
"gainmap-offset-sdr",
"gainmap-offset-hdr",
"gainmap-hdr-capacity-min",
"gainmap-hdr-capacity-max",
"gainmap-use-base-cg"]:
field = image.get(name)
print(f"{name} = {field}")I see: So it's correctly reading the SDR image, attaching the gainmap as a compressed JPG, and the gainmap metadata as a set of tags. It's able to stream SDR decode, so memory performance is OK in this case anyway. You can set two options: |
compiles, but doesn't work
though it still doesn't work haha
no gainmap is being generated though, how odd
and more logging
|
|
||
| object_class->nickname = "uhdrload_buffer"; | ||
| object_class->build = vips_foreign_load_uhdr_buffer_build; | ||
|
|
There was a problem hiding this comment.
I think we still need to declare ->is_a_buffer here that always returns FALSE. For example:
/* Compile with:
* gcc -g -Wall test.c `pkg-config vips --cflags --libs`
*/
#include <vips/vips.h>
int
main(int argc, char *argv[])
{
if (VIPS_INIT(argv[0]))
vips_error_exit(NULL);
VipsSource *source;
if (!(source = vips_source_new_from_file("/dev/null")))
return 0;
vips_foreign_find_load_source(source);
vips_foreign_find_load_buffer(NULL, 0);
return 0;
}$ ./a.out
(process:1903495): VIPS-WARNING **: 12:46:12.704: loader uhdrload_source has no is_a_source method
(process:1903495): VIPS-WARNING **: 12:46:12.705: loader uhdrload_buffer has no is_a_buffer methodThere was a problem hiding this comment.
Oh, good point, I'll fix this tonight.
|
|
||
| object_class->nickname = "uhdrload_source"; | ||
| object_class->build = vips_foreign_load_uhdr_source_build; | ||
|
|
There was a problem hiding this comment.
Ditto, we should probably declare ->is_a_source here that always returns FALSE. Similar to csvload_source:
libvips/libvips/foreign/csvload.c
Lines 625 to 633 in 42cf313
|
|
||
| option('uhdr', | ||
| type: 'feature', | ||
| value: 'auto', |
There was a problem hiding this comment.
Note for downstream packagers: libultrahdr is not yet widely available on the various distros.
If you are building with -Dauto_features=enabled (see e.g. rule 3 from https://blogs.gnome.org/mcatanzaro/2022/07/15/best-practices-for-build-options/), you must either:
- explicitly disable the feature by passing
-Duhdr=disabled, or - ensure that libultrahdr is properly packaged.
There was a problem hiding this comment.
I'll add a note to the README "dependencies" section for libuhdr.
Co-authored-by: Kleis Auke Wolthuizen <github@kleisauke.nl>
|
Thank you for the review, @kleisauke ! |
Co-authored-by: Kleis Auke Wolthuizen <github@kleisauke.nl>
|
Thanks! |
Support for AVIF and HEIF is on the roadmap for libultra. Support for AVIF in libvips will be extremely useful, as this format is widely supported by all modern browsers, offers higher image quality, and files which are about half the size as JPG. Support for decoding HEIF will be important to support transcoding from images captured on iPhone, though encoding to HEIF is more of a niche use case as support is primarily only in Safari / Apple software. |
|
@jcupitt I'd be happy to help test. I'm very familiar with the gain map format and libultrahdr, but not libvips. Could you share a couple quick test commands to use via the CLI (or point to examples)? I believe the following would be very helpful to ensure core transcoding operations work as expected:
All of these should be JPG gain map being transcoded to another JPG gain map with as much fidelity to the original as practical for a given transform and quality parameters. |
|
Hi @gregbenz, thanks for the offer of help.
libvips has had HEIC and AVIF support since 2018, including high bit depth support. HDR-in-AVIF and -HEIC should just work if you have the right profile (I think). Gainmaps in HEIC and AVIF are not supported -- it sounds like the new libultrahdr release will help!
There are testing notes up the thread ^^^ #4645 (comment) with some sample python code. Right now it's not an automatic process, you'll need to manipulate the gain map yourself, and explicitly load via the We're going to improve the integration with the libvips load and save system in a follow-up PR, so you don't need to call This is now merged to master (our dev branch), so if you find problems (you probably will!), please open a new issue. |
|
@jcupitt Great to hear standalone HDR AVIF is already supported! Gain map AVIF will be very helpful and is the ideal way to share HDR (unless file size is a much higher priority than image quality). Gain maps are often misunderstood as a bit of a "hack" to enable HDR in the 8-bit JPG format. While they do that, the more important benefit is high-quality adaptation for displays lacking full headroom (anything requiring more than 1 stop of tone mapping tends to be low quality compared to a gain map created with artistic input). A gain map AVIF will offer much higher quality than AVIF without it (and avoids rendering issues in FireFox which shows HDR AVIF very dark, while the base SDR AVIF would render safely). Here's a direct comparison of AVIF with and without a gain map: https://gregbenzphotography.com/hdr-photos/gain-maps-vs-tone-mapping-hdr/ Are there CLI commands which might be used for testing? I'm sure I could figure it out, but don't currently develop in Python. I should be able to build the binary if that's included in the output for the build process described above. I'm eager for the potential that WP media experiments should be able to leverage this for client-side WASM support to upload HDR gain maps to the WordPress media library, and though it would be ideal to do some testing at this level before the WASM is available in case there are some bugs. libultrahdr itself works very well, so for me it's just a question of confirming the new interaction between it and libvips. |
Yes, they are in that comment above, under "To test". vipsdisp is very handy for testing, I'd try that too. https://github.com/jcupitt/vipsdisp Linux, mac and win binaries. If you load a uhdr image to .v format you can view it with vipsdisp and see individual pixel values, metadata. etc. |
|
@jcupitt I see that round trip test, but am wondering about the best CLI test to validate resizing, compression, and cropping? Something that simulates deriving a thumbnail for an upload to WordPress would make for an ideal test. My goal here is to transcode the JPG, I've got good tools to evaluate the output (the Adobe Gain Map Demo app is an excellent one to confirm a valid output and most map-specific metadata: https://helpx.adobe.com/camera-raw/using/gain-map.html#resources). |
|
You could just about do a resize or crop at the CLI, but it'll be really awkward. I'd use python, it'd be much simpler. You can test load and save at the CLI, both SDR and HDR, and compression settings (use the The nice thing about vipsdisp is that you can see the intermediate image, so you can test uhdrload and uhdrsave separately. |
|
@jcupitt Good to know, thanks, will take a look at that route. |
|
Currently loading a jpeg normally still discards the gainmap and you have to know it's an UltraHDR file and specifically load it, right? Once it's supported in normal JPEG load and save, I'd be interested to maybe see whether I can get UltraHDR support in 1-2 downstream libraries that affect me (if I'm able to do C lib stuff...). Mostly for Nextcloud, not sure whether WordPress also uses libvips in it's default config, not sure anymore whether it's was in PHP's image stack. |
|
That's right, I'm planning to add a thing to the regular libvips jpeg loader that spots an UltraHDR image and transfers control to the uhdr loader instead. Same for save. |
|
@jcupitt FYI that libultra has a probe to validate a gain map in unit tests (google/libultrahdr#201) and CLI (google/libultrahdr#368) which may help point to existing capabilities to leverage. I haven't dug in to review the C++ here. If the library does not already have something you would consider sufficiently performant to determine when libvips should use libultrahdr vs another code path, may be good to raise a request in libultrahdr as I would expect similar concerns elsewhere (ImageMagick similarly segregates libultrahdr from other JPG processing to ensure no impact to existing workflows, edge cases, etc). |
|
@cromefire WordPress has a few potential image pipelines for HDR support in the media library:
Limited workaround you can use now: share HDR gain maps via WordPress currently by using the "full" size in the media library, as this is just your original image. You won't have any derived sizes (thumbnails, etc) and you can't allow any optimization plugins that might alter the image. So it is not ideal, but there is some support as long as you do a unique upload for each size you need. Several of these may require PHP 8.0+ in the hosting environment (Media Experiments does, and I believe AVIF support does too). |
|
You can also use intervention: https://github.com/Intervention/image It's a general image handling module for php with backends for GD, imagick and php-vips. I think there are several wordpress things that let you use intervention for image handling. There are some benchmarks of php-vips, imagick and gd here: https://github.com/libvips/libvips/wiki/Speed-and-memory-use (intervention will need some work to support libvips uhdr, but probably not too much) |
|
@jcupitt One thought on leveraging libultrahdr. It can be compiled to output gain maps using the ISO standard (binary data in the aux image codestream), XMP metadata (per the Google/Android spec) or dual-encoded to use both. I recommend that libvips use ISO encoding, as XMP encoding is deprecated in favor of ISO - and libultrahdr will decode images with either regardless of which compiler options you choose for encoding new images. The ISO format is now well supported (all major browsers, Adobe software, Android, iOS, etc), and skipping XMP encoding reduces the file size by ~2K. I don't recall the current defaults if you don't set the flags, but you can use something like: |
Results from my test UltraHDR test harness (libvips + libultrahdr via pyvips) - ContainerizedHi all — I set up a small, reproducible test harness to exercise
What I tested
Key observations
Why I’m postingGiven how the API is exposed today (gain map as a compressed buffer in metadata), this seems by design: libvips won’t automatically recompute or recrop the map, and implementers doing geometry changes (crop, non-uniform resize, rotate, etc.) must keep base + gain map aligned. That matches comments above about leaving most gain-map manipulation to higher-level libraries. I wanted to share concrete, reproducible confirms + artifacts so downstream implementers (e.g., web thumbnailers, CMS pipelines) don’t accidentally ship mismatched crops. Snippet: proportional gain-map crop (pyvips)import pyvips
in_path = "in.jpg"
out_path = "out.jpg"
img = pyvips.Image.uhdrload(in_path)
gm = pyvips.Image.jpegload_buffer(img.get("gainmap"))
# crop box in base pixels
X, Y, W, H = 220, 220, 640, 640
img2 = img.crop(X, Y, W, H)
# map base->gainmap coordinates
Wb, Hb = img.width, img.height
Wg, Hg = gm.width, gm.height
xg = round(X * Wg / Wb); yg = round(Y * Hg / Hb)
wg = max(1, round(W * Wg / Wb)); hg = max(1, round(H * Hg / Hb))
gm2 = gm.crop(xg, yg, wg, hg)
# (If you also resize the base, apply the same scaling to gm2 here.)
# re-embed and save
gm_bytes = gm2.jpegsave_buffer(Q=85)
img2.set("gainmap", gm_bytes)
pyvips.Operation.call("uhdrsave", img2, out_path, Q=85)(For resize+crop, apply the same scale to the cropped gain map region before re-embedding.)
|
6 participants
Add
uhdrloadandudrsave.libuhdr
Build with eg.:
This PR
uhdr_dec_get_gainmap_image()etc. to do our own image decodeuhdrsaveoperationQFollow-up PR:
half -> floatfunction in libultrahdr, we should use itsdr + gainmap -> hdruhdr -> radconversion