How to generate a wide gamut surface ?

[kulve]

Hi,

This is not explicitly about the avif-coder but maybe somebody here has solved this problem already.

I've managed to create wide gamut bitmaps with this library, but how can I actually render them in wide gamut?

I'm able to create a FP16 SurfaceView but how can I render the bitmap into it? I tried using the Holder.lockHardwareCanvas + canvas.drawBitmap() but the end result is an RGBA8888 SurfaceView.

And if I create GLSurfaceView, all the EGLConfigs are only 8 bit even though the device supports wide gamut surfaces (at least for native code).

I initially thought that using something like Image widget from the Jetpack Compose would automatically give me wide gamut surfaces but that doesn't seem to be the case.

[awxkee]

Hi!
Thanks for you question.

Library itself make a cheat because android directly do not support HDR bitmap library does tone mapping using Logarithmic tone mapper or Rec2408 at your choice. That means after tone mapping curve your image already SDR not HDR.

Straight to your question:
It's not possible to render to SurfaceView/TextureView with canvas api. You must use OpenGL/Vulkan to achieve that.

I don't know very well OpenGL api however I know familiar with Vulkan/Metal if you wish to render HDR, yes, you have to use wide gamut surface.

1. So you have to at fisrt create surface with reliable type for render PQ/HLG.
2. Create your command buffers, textures and encoder, or render pass and shaders if needed ( as I mentioned I do not know OpenGL very well however concept of GPU is same so I do not expect any significant difference from Vulkan/Metal and OpenGL )
3. Copy your bitmap to texture and render your texture to the surface using blit or render pass
4. As I mentioned, library already returns SDR images instead of HDR so you'll have to at final disable HDR tone mapping in library by creating your own copy if that you needed
5. Create your own PQ/HLG shaders to apply curve to your image

Hope, that helps :)

[kulve]

Thanks for the explanation.

I'm using an Android TV device and my final goal is to get real HDR images to a HDR10 capable TV. HDR movies has been common for years but I didn't find any apps that would show photos in HDR.

Is there a way to avoid that "cheat" you mentioned?

[awxkee]

At the moment it is possibly only by recompiling library by yourself. The most of users do not expect that it will work this way :)

You have to open JniDecoder.cpp and at lines 178-180 you'll se this code

else if (hasNCLX && nclx &&
      nclx->transfer_characteristics != heif_transfer_characteristic_unspecified &&
      nclx->color_primaries != heif_color_primaries_unspecified) {

You have to disable entering this block when transfer function is PQ and HLG

nclx->transfer_characteristics == heif_transfer_characteristic_ITU_R_BT_2100_0_PQ

nclx->transfer_characteristics == heif_transfer_characteristic_ITU_R_BT_2100_0_HLG

By disabling this it is expected that you're responsible to display image in proper color profile using valid transfer curve. Both of them in most cases expected as per ITU-R standarts works in BT.2020 or BT.2100 color profile however I seen images in sRGB with PQ, so probably you'll have to return to nclx->color_primaries in some way also.

Actually android do support rendering wide gamut out of box for api 30+ that you may also consider after you block tone mapping. However some manufacterers may do not implement it correctly or completely drop this and this API works to me even on good device highly unpredictable with totally unexpected crashes and totally lacks any documentation from android team.

bitmap.colorSpace = ColorSpace.getFromDataSpace(DataSpace.DATASPACE_BT2020_PQ)

[awxkee]

There are a lot of intrinsic code. Please, consider publish on jetpack for tests, or use prebuilt release because you'll see performance degradation with factor x10-x1000 in debug versions.

[awxkee]

There also ready to go shaders for Vulkan in assets. It will be easy to adopt them to OpenGL if you still do not have one

[awxkee]

Also if you familiar with ndk you may use a ANativeWindow API to send you buffer into a surface. However after it applying shaders and transformations can be a tricky one:)

[kulve]

Some progress. I'm now using EGL/GL more directly instead of trying to use GLSurfaceView etc.

I'm able to use FP16 buffers and I can additionally force BT2020_PQ gamut and get the TV in the HDR mode! I'm still trying to figure out if I'm able to set the HDR metadata so that it's actually passed through the stacks. Hopefully at some point I get those from the avif-coder :)

Btw, I'm using PreferredColorConfig.RGBA_F16 with HeifCoder().decodeSampled() and that works but with PreferredColorConfig.HARDWARE I get just black.

What does the HARDWARE config do in practice?

[awxkee]

Glad to hear that you have a progress!

PreferredColorConfig.HARDWARE does just exactly what it means that means your image will be wrapped into a HardwareBuffer and Bitmap is not real Bitmap anymore ( consider any android bitmap just a class that contains a reference to c++ native buffer) instead you've got a Bitmap backed by HardwareBuffer thay may be accessed from 31+ API bitmap.getHardwareBuffer(). .

Hardware buffer is a service layer between GPU, CPU and system. At the very low level you may consider Hardware Buffer just a wrapper of a Vulkan allocated buffer. If you're working on the latest OS versions you probably may find some API to convert underlying Hardware Buffer to a GPU texture. There are some convenient API if I may call it so for Vulkan to pass directly a Hardware Buffer to a texture. Perhaps the same is done for OpenGL.

Then, when you bitmap is HARDWARE if you're trying to access direct pixels of bitmap it values are non-sense, because they are not exists in real. I think you do so, and get just black pixels :)

[awxkee]

Also there are not guarantees that F16 surface will be used in the library when you're requesting a HARDWARE buffer.
Library making a decision based on image bit depth to set a HB buffer format. For depth > 8 it will be AHARDWAREBUFFER_FORMAT_R16G16B16A16_FLOAT otherwise AHARDWAREBUFFER_FORMAT_R8G8B8A8_UNORM.

[awxkee]

Probably at final point it may be easier for you do some reformat in library to always return Java ByteBuffer that wraps a native buffer always contains f16 data. This is a little overcomplicated expect bitmap to acquire raw data

[kulve]

I noticed that bitmap.getHardwareBuffer was introduced in API 31 while I have API 30. But I think the RGBA_F16 does everything I need on my device.

I'm creating my BT2020_PQ AVIF photos from Canon CR2 with Darktable. I don't have any HDR devices to verify that they look correct but at least importing it to GIMP produces somewhat correct looking image.

But if I open them with my avif-coder code on my Android TV device they are overly bright (only the dark colors are visible, everything else is white). Attaching one file here. Any idea of the file is "broken" or if there's perhaps something not supported by the avif-coder?
img_0199_avif.zip

[awxkee]

There is an ICC profile inside and cICP profile that signal that photo is BT.2020 PQ. The library at the moment make cICP profile priority and ignores ICC. Probably I'll do changes later.

The over bright photo that you see is a real PQ curve applied photo. If darktable ships photo with PQ curve with ICC profile - it is not possible to exactly represent PQ in ICC.

Btw, probably something wrong in darktable I don't see on any device HDR content. Of course it is not possible to share HDR by screenshot but photo doesn't feels like HDR.

Perhaps, the photo is intented to be like this one.

I manually set parameters in attempts to restore HDR content. MacOS + Chrome and my app cannot detect standard HDR PQ content. Encoding in darktable is wrong. All apps expect ICC profile be omitted to display HDR ( because as I mentioned PQ cannot be represented in ICC ).

Finally, you are probably using logarthmic tone mapper ( that set by default ), change it to rec2408, results will be better however still not original.

So, there a few problems:
1 - I'll change lib to chrome and MacOS behaviour to honor ICC over NCLX.
2 - Point 1 means that darktable encoding wrong and no software will detect by default your HDR content.
3 - Logarithmic tone mapper is bad for this case.

Here is restored photo without ICC. Looks good. Rec2408 in app restores compromisable results in SDR.
IMG_0199_rr.avif.zip

[awxkee]

If darktable honor NCLX over cICP as my library at the moment it won't be easy for you to share photos.

Many software ( like I am ) takes a Chrome and WebKit on MacOS as a reference software and they definitely do honor ICC over cICP that mean any HDR content will be never recognized as HDR.

[awxkee]

Even when you do not have a HDR device you may open your image in chrome. SDR tonemapping also done in chrome and work when needed.

[kulve]

I'm new to Darktable so it's definitely possible that I'm doing something wrong there.

Chrome shows the AVIF in the same way I see it in Darktable. So somewhat correctly, even though it's darker that the attached JPG. But maybe that's just because of my Darktable settings.

Here's the JPG from the camera for the same image (it stores both JPG and CR2).
Uploading img_0199_jpg.zip…

[awxkee]

Your link is broken :)

Might be Darktable settings mixed up also. Restored photo in Chrome looks exactly as HDR bright and good and I see on my HDR display brightness increasing.

Of course I understand how stupid is sending HDR photos by photo from phone :) However, you may see the difference in brightness levels even from phone.

There is ICC based photo.

There is HDR photo.

[kulve]

Trying to upload the JPG again. From some reason it still shows as "uploading" to me.

img_0199_jpg.zip

Maybe I need to wait for the upload to be complete before sending.

And yes, I don't really know what to expect from "HDR photos" :) Currently I feel that the traditional 8 bit SDR images are lacking in "colors" in some cases and I'm hoping that with more bits, wider gamut and also with PQ they might look "better" in general. So now I'm experimenting on what that might be.

[kulve]

On this Android TV I need to provide both the SMPTE 2086 and CTA 861 metadata to enable the HDR mode. I.e. these:

• EGL_SMPTE2086_DISPLAY_PRIMARY_RX_EXT
• EGL_SMPTE2086_DISPLAY_PRIMARY_RY_EXT
• EGL_SMPTE2086_DISPLAY_PRIMARY_GX_EXT
• EGL_SMPTE2086_DISPLAY_PRIMARY_GY_EXT
• EGL_SMPTE2086_DISPLAY_PRIMARY_BX_EXT
• EGL_SMPTE2086_DISPLAY_PRIMARY_BY_EXT
• EGL_SMPTE2086_WHITE_POINT_X_EXT
• EGL_SMPTE2086_WHITE_POINT_Y_EXT
• EGL_SMPTE2086_MAX_LUMINANCE_EXT
• EGL_SMPTE2086_MIN_LUMINANCE_EXT
• EGL_CTA861_3_MAX_CONTENT_LIGHT_LEVEL_EXT
• EGL_CTA861_3_MAX_FRAME_AVERAGE_LEVEL_EXT

Would perhaps the first 8 be something that could be retrieved from the photo's metadata? What about the last 4?

[awxkee]

Yes, primaries and white point will be derived from the photo. EGL_SMPTE2086_MAX_LUMINANCE_EXT and EGL_SMPTE2086_MIN_LUMINANCE_EXT has to be a display max/min nit levels. You have to set it on your own, check by some TV api max level exists, or have hardcoded value taken from your TV spec:). Max content light level supposed to be set in photo metadata however nowadays this property still doesn't exists and all photos often considers static values like 203 cd/cm^2 for SDR and 1000 cd/cm^2 for HDR. Do not know anything about the last one.

[awxkee]

While you're testing you may consider to set BT.2020 always since you're interested in PQ.
static const float Rec2020Primaries[3][2] = {{0.708, 0.292}, {0.170, 0.797}, {0.131, 0.046}};
Order is RX, RY, GX,GY, BX, BY
And White point (x, y) = 0.3127, 0.3290

[kulve]

I'm actually using those primaries and white point already. But it's unclear to me if those should be somehow coming from the camera? Or are they always exactly those after because Darktable outputs BT.2020 PQ?

IMG_0199_rr.avif looks on Chrome much brighter than my original IMG_0199.avif.

I modified your code for the "transfer_characteristics == heif_transfer_characteristic_ITU_R_BT_2100_0_PQ" check as you explained earlier and now I get much better image on my Android TV but it's actually much more brighter than your IMG_0199_rr.avif in my Chrome. But I can see some shades of white there in the sky and in the mountains so it's not completely white.

And both the IMG_0199_rr.avif and the IMG_0199.avif looks exactly the same in HDR through my Android TV.

Tuning the following values don't seem to make any difference:

• EGL_SMPTE2086_MAX_LUMINANCE_EXT
• EGL_SMPTE2086_MIN_LUMINANCE_EXT
• EGL_CTA861_3_MAX_CONTENT_LIGHT_LEVEL_EXT
• EGL_CTA861_3_MAX_FRAME_AVERAGE_LEVEL_EXT

Although I did have first the mix/max reverted so maybe the TVs got confused or something.

[awxkee]

IMG_0199_rr.avif looks on Chrome much brighter than my original IMG_0199.avif.

Yes, that's because now Chrome consider it as HDR photo not an ICC based.

To properly display an image using your approach, please, ensure that code in gamut transfer block is not executed, for testing you may just comment this out ( lanes 174 - 261) and check it on the phone, or your TV that bitmap from avif-coder is dark and not an overbright. You should probably see it a VERY dark.

If you already did so, and the library do not applying tone mapping nor ICC and you see a dark photo as an output then it is misconfiguration on your behalf

There is a photo that you have to see before sending it to your PQ surface

[awxkee]

If you see this photo as it should be already then this is the time to re-check your surface configuration

[kulve]

Did you mean to comment out lines 174-289 instead of lines 174 - 261? I've made sure those lines are not executed.

Above you say I should see it dark and as in the screenshot but that screenshot is quite bright?

[awxkee]

You're right I made not exactly correct statement. Properly call this photo not gamma corrected image, or image in linear colorspace.

However, you still have to see it before you sending to PQ surface/PQ shaders this one

[awxkee]

Also you're right about line numbers. I made a mistake with 174-261

[kulve]

Thanks a lot for your help. I think I'm now getting PQ through avif-coder and I'll focus next on how to get a good CR2 to AVIF conversion in Darktable.

Any chance to get some extra API/configuration to a future avif-coder release to get that PQ out without modifying the source code? :)

Btw, from https://www.darktable.org/2023/12/darktable-4.6.0-released/ :

Exported AVIF files now no longer embed a superfluous ICC profile if the color profile can be encoded as CICP.

I was originally using 4.2.1 version and have now moved to that 4.6.0. But I didn't notice any difference in practice.

[awxkee]

I'm glad to hear that information helps :)

Exported AVIF files now no longer embed a superfluous ICC profile if the color profile can be encoded as CICP.

I was originally using 4.2.1 version and have now moved to that 4.6.0. But I didn't notice any difference in practice.

I think you have to ensure that ICC is not really embedded. The photo should have difference may be more or may be less than the previous one in Chrome because as I mentioned before it is not possible to exactly represent PQ in ICC profile.

Any chance to get some extra API/configuration to a future avif-coder release to get that PQ out without modifying the source code? :)

There are very many edge cases to implementing this and not the all are easy to handle in current API implementation.

And some of them requires serious think how does should it work?

For ex. if your application met the photo as a previous one from Darktable with ICC and CICP and you're ready to render in PQ surface why ICC should be turned off? It is currently bug that library skips ICC and applies CICP. ICC ideally must not be skipped.

There are a lot of non-standard HDR photos that ITU-R expected to exists. It is easy to face a photo with PQ and not BT.2020 color profile.

For ex. Apple seems nothing wrong to apply PQ/HLG to Display P3. And even I myself met an images with CICP sRGB and PQ. There are should be also way which colorspaces and transfer functions consumer may and want handle itself and return determined values to consumer. At the moment library handles all this out the box.

All that makes me think that returning Bitmap is not enough and a lot of parameters should be added that means current implementation should be seriously reworked and to do so it is might be need to add one more function that consumes all parameters from consumer and returning determined CICP and bitmap.

All that makes me a little sceptical that I really want to do so : ))
However, I'm glad to review a PR :)