Repository navigation
Indicating (technical) rendering hints for HDR content to viewers #2312
Description
Abstract
This proposal is about indicating technical rendering hints (for HDR images in the UltraHDR format) to Image API Clients.
Background
As a proof of concept I've created a static Image API endpoint delivering HDR enable content, you can read more about it and have a look in my blog.
Since it's only a PoC it shows a 70s wallpaper, since it's from my domain, there are other use cases for HDR scientific images, like:
- Image analysis
- Fluorescence photography
The tiles aren't just JPEGs, but UltraHDR images. This format allows to add a so called gainmap to the image. To be backwards compatible the format is still JPEG with an embedded image, as specified in MPO.
Keep in mind that you'll only see a SDR image, if the requirements (see below) aren't meet. This is actually a feature since it allows transparent fallback if those aren't provided (this is also a reason for this proposal: a Viewer needs to know whether it should hinting those requirements).
Displaying UltraHDR images needs to have the following requirements to be fulfilled:
- The device needs to have a HDR capable screen.
- The browser need to have HDR support - currently only recent Chromium-based browsers
Currently the combination of a Apple Silicon notebook or desktop and Google Chrome is the most likely working combination. Windows devices usually need to have HDR enabled explicitly. Have a look at the page created by photographer HDR evangelist Greg Benz.
Since the HDR support of the browser (currently) needs a specific initialisation of the viewer, some sort of indicator / flag is needed to:
- Init the client element (
canvas) - Check the requirements and indicate missing support to the user, maybe giving hints on browser and display settings.
- Expectation management of user experience
Possible solutions
Current implementation: Using supports of Image API 2.0
Since the current implementation is using the Python module iiif with a custom tiler backend the example uses the Image API 2.0. This allows to use supports with the URI https://christianmahnke.de/iiif-hdr/.
"supports": [
"https://christianmahnke.de/iiif-hdr/"
]Image API 3.0
In Image API 3.0 the best field is certainly extraFeatures.
Summary
- HDR on the web is already there - support will broaden over the next years
- Implementors might want to use this possibility to present either "spiced up" versions of their contents or the "real" (OpenEXR comes to mind) images.
- Currently it's not possible to hint the requirements for display devices in a standardised way.
Further questions / discussion
- Is there enough interest to define an extension?
- Since every UltraHDR image is a valid JPEG the example above didn't define contents as a
format(orextraFormats), this is of cause open for discussion. - Since every UltraHDR image is (technically, not in a viewer) a MPO is a JPEG - might it be worthwhile to consider adding support for both formats?
- Might it be worthwhile to abstract from the format and think about general HDR indication (in the Presentation API) which could also be used for 3D objects?
- Might it be worthwhile to abstract from the format and think about general indication of colorspace intents?
Activity
azaroth42 commentedon Aug 28, 2024
Thanks for such a well researched, well explained and well implemented suggestion!
Understanding that it's backwards compatible with jpg, and indeed doesn't have a separate media type or file extension (which I think is an odd choice, but not ours to make), I feel that in the IIIF view of the world, it's a different quality.
My rationale:
extraFeaturesare not features of the image delivered, but of the service that delivers the images. Hence the features are things likerotationArbitraryandsizeByW... functionality that the server has, independent of the image formats.ultrahdrquality.If the above is accurate and convincing, it would mean
{ "extraQualities": ["ultrahdr"] }Secondly, in thinking about the viewer requirements, it seems like this would need to be reflected into the Presentation API somehow, so that the viewer knows to do the UltraHDR tests and setup. I think this is fine without change to the API, as the functionality is tied to the image, and the annotation can embed information from the image service's info.json. Thus the above would be part of the ImageService block in the Manifest.
(e.g. per https://iiif.io/api/presentation/3.0/#56-annotation )
Thoughts?
robcast commentedon Aug 28, 2024
I like @azaroth42 's idea of treating HDR as a quality rather than a feature.
I would really like to extend this to colorspaces outside of sRGB, as @cmahnke also hinted to. I recently implemented support for bigger colorspaces in digilib where it is currently only accessible through the native API: https://dh-tech.github.io/blog/2023/01/16/java-color-profiles/ (also viewers seem to have even less control about how the browser and the OS handle this)
I also don't know if "more-than-srgb" colorspace would be a separate quality from "ultrahdr" and what to do about about combinations of both...
cmahnke commentedon Aug 28, 2024
@azaroth42: My rationale to use
supportswas thinking the other way around: The current approach can be considered to be "server push": The client requests JPEG but get's "more" by default. This way there is no need for additional content negotiation steps (and thus end user interaction). This has been a nuisance of mine when I've "tinkered" withqualitiesin the past.But I don't have a strong opinion on this, since the reasons you gave are convincing.
Another remark would be that
ultrahdrmight be the wrong name, since other formats also support HDR and UltraHDR itself is to certain degree a marketing name (by Google).Technically related names are Gain Map by Adobe or Apple HDR. Although sometimes other container formats are used, such as HEIF. But as almost always the official "name" or better identifier is rather hard to remember: ISO/CD 21496-1.
On the other end there are HDR initiatives for other formats, like PNG as well. Another, although made up, example would be TIFF with high bit depth (in Rec. 2020).
To be able to combine
formatand the proposedqualityit might be onlyhdror something similar.As to the question regarding "upscaling" or how I called it "spicing images up": This is definitely possible, either programmatically or via a UltraHDR capable graphic program.
ruven commentedon Sep 3, 2024
The gain map approach to SDR <-> HDR conversion looks really interesting. I'll look into adding support for this in IIPImage, especially as there will presumably be published specs for including gain maps into other formats once the ISO standardization process for gain maps has completed. Adding support will be non-trivial, though, as when you modify the image through resizing or rotation etc, you also need to modify the gain map image (which doesn't necessarily have the same resolution) in the same way.
For IIIF,
hdras a quality rather than a format makes sense as a number of formats can support both SDR and HDR either through gain maps (JPEG) or natively (AVIF, TIFF, EXR etc). However, there are some ambiguities as HDR is not just about the color space, but also about bit depth.So, if we introduce a
hdrquality, if the user does not specifyhdr, by default you should presumably only get back an SDR version even if your source image is HDR. So, for JPEG, for example, this would be a standard JPEG (with no gain map as these take up quite a bit of space) and, say, for AVIF an 8 bit SDR version.If you now request
hdrquality: for JPEG output, the server has only one option, which is to include a gain map. However, for AVIF and other higher bit-depth formats, the server has, in theory, 2 options:I presume the user would prefer the latter if requesting an AVIF image. Perhaps, in practice, it will depend on what the source image consists of. If it's natively in 8 bit + gain map form, it will be easier for the server to output in the same form. Whereas if the source image is in native HDR form, the server will output to native HDR if the format supports this, or will have to transcode to 8 bit + gain map if it doesn't.
This raises the issue of bit depths in general, though. This is a separate discussion, but if the IIIF
qualityis concerned with color space, maybe there also needs to be a way to also specify output bit depth. JPEG is 8 bit only, but other formats that have native web browser support such as WebP, AVIF and PNG can all handle higher bit depths. Currently, there's no way to request, say, a 10 bit AVIF or 16bit PNG through IIIF. Perhaps this could be enabled by allowingavif10orpng16as output formats? Or by including this somehow in thequalityfield:hdr10orcolor16(or with a separatorcolor-16)? The native bit depth is also not currently included in the Image API info.json metadata, so it might be useful to add something like adepthfield to this.In fact, on the image server side, I can see the use of gain maps as being something useful for improving the handling of high bit-rate images in general. IIPImage, for example, simply scales 16 or 32 bit source images down linearly to 8 bits per channel so that they can be output as JPEG. If the image included a gain map (which can also be used to go from HDR to SDR), this would allow a more sophisticated tone-mapping to be done when going down to 8 bits.
The approach IIPImage takes here is simply to avoid unnecessary color space conversion: if the source image has a DCI-P3 ICC profile, just use the raw pixel values and copy the profile across (sure, ideally all image processing should be carried out in linear space, but this adds a lot of overhead). All current web browsers nowadays fully support ICC profiles, so images should be rendered correctly (in theory!).
If, however, we want IIIF servers to perform color space conversion, then, yes, we'll need a way specify this as a
qualityand we'll need a syntax to specify multiplequalities. Having said that, the only use-cases I can see for this would be if you want the server to downgrade a wider-gamut image to sRGB for a device that doesn't support ICC profiles. Or if your native image is in something that is not widely supported, such as CIELAB color space and you want to convert to a color space of you choice for output as JPEG or WebP etc.cmahnke commentedon Sep 8, 2024
Just a small update: Everyone knowing to have a working HDR setup but didn't get the example to work: I was under the misconception, that Chrome enabled this feature by default in June - this isn't the case.
One needs to enable the
enable-experimental-web-platform-featuresflag in Chrome, I've updated the page.cmahnke commentedon Nov 15, 2024
Yesterday I had the opportunity to give a lightning talk essentially on this topic at the 2024 online meeting.
I focused heavily on use cases (most of them currently not implemented using IIIF yet) in order to promote this afford. You can find my slides here. I'm waiting for feedback by the participants and report here if I get any.
cmahnke commentedon Dec 27, 2024
Jus to let anyone know who isn't reading the Slack Channel on a regular basis, @caitlinPerry posted this:
I'll try to summarize this discussion there as well and be'll happy if the participants can also attend the meeting.
cmahnke commentedon Jan 16, 2025
Hi everyone,
the notes of the call are available as a Google Doc, my slides are available here. Long story short:
The decision reached is a solution without any API change, by providing use case tailored profiles and to revisit the issue if this approach isn't sufficient. This would, of course, require that there be a certain number of users.
@ruven @robcast @azaroth42 @glenrobson : Everyone who was present is invited to add to the minutes if I have forgotten, misunderstood or abbreviated something.
azaroth42 commentedon Mar 17, 2025
Flagging as defer as there's no change needed for further experimentation and implementation, per the call. When there are implementations we can review and potentially revise the Image API spec if needed.