[Android] Adjust GUI SRD peak luminance when display is in HDR PQ mode

[thexai]

Description

[Android] Adjust GUI SRD peak luminance when display is in HDR PQ mode

This the equivalent functionality of #18984 and #21973 for Android.

Motivation and context

This is requested a lot on the forums....

https://forum.kodi.tv/showthread.php?tid=376360
https://forum.kodi.tv/showthread.php?tid=357205
https://www.reddit.com/r/kodi/comments/1awb6h3/osd_hdr_too_bright_during_playback/

With this is not necessary set subtitles color dark grey or different color for SDR / HDR and also is adjusted OSD and all GUI elements.

Android already does basic tone mapping - there is no need for full tone mapping from BT.709 to BT.2020 or gamma to PQ transfer. Only is need to set the SDR peak to 100 or 200 nits because by default it seems to be set to 1000 nits (at least in Shiled) and this is why it looks much brighter than normal SDR.

When SDR peak is set to 100 - 200 nits then GUI has same appearance than in SDR and GUI white is regular SDR white.

The default setting is 40% and matches what we already have in Windows. That is, the result is consistent with Windows using the same setting value. Adjusting GUI setting to 100% gives you the maximum brightness (~1000 nits), which is the same as before this PR.

How has this been tested?

Runtime Shield Pro 2019

Forum users:

• Fire TV Stick 4K Max 2nd gen (https://forum.kodi.tv/showthread.php?tid=376360&pid=3186614#pid3186614)
• Fire TV Stick 4K Max 1st gen (https://forum.kodi.tv/showthread.php?tid=376360&pid=3186768#pid3186768)

What is the effect on users?

No more tricks to adjust brightness of subtitles / OSD / GUI when playing HDR10 or Dolby Vision contents.

Screenshots (if appropriate):

Types of change

• Bug fix (non-breaking change which fixes an issue)
• Clean up (non-breaking change which removes non-working, unmaintained functionality)
• Improvement (non-breaking change which improves existing functionality)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that will cause existing functionality to change)
• Cosmetic change (non-breaking change that doesn't touch code)
• Student submission (PR was done for educational purposes and will be treated as such)
• None of the above (please explain below)

Checklist:

• My code follows the Code Guidelines of this project
• My change requires a change to the documentation, either Doxygen or wiki
• I have updated the documentation accordingly
• I have read the Contributing document
• I have added tests to cover my change
• All new and existing tests passed

[sarbes]

I can't say I'm too happy about this approach. It will result in heavy banding, having just ~7 bits (?) per color channel. Also, it introduces colour inaccuracies. It might also lengthen the execution time of the shaders, depending on the platform.

Pretty much every SOC with HDR support comes with its dedicated mapping hardware, so that such hacks are not needed.

For devices with lacking drivers, a slightly more reasonable approach would be a proper resolve pass at the end, preferably using pixel local storage or a framebuffer fetch, I think.

[Hitcher]

Run some tests and it's not working for some Dolby Vision videos.

Working:

Video
ID                                       : 1
Format                                   : HEVC
Format/Info                              : High Efficiency Video Coding
Format profile                           : Main 10@L5.1@Main
HDR format                               : Dolby Vision, Version 1.0, Profile 8.1, dvhe.08.06, BL+RPU, HDR10 compatible / SMPTE ST 2086, Version HDR10, HDR10 compatible
Codec ID                                 : V_MPEGH/ISO/HEVC
Duration                                 : 1 h 15 min
Bit rate                                 : 21.3 Mb/s
Width                                    : 3 840 pixels
Height                                   : 1 920 pixels
Display aspect ratio                     : 2.000
Frame rate mode                          : Constant
Frame rate                               : 23.976 (24000/1001) FPS
Color space                              : YUV
Chroma subsampling                       : 4:2:0 (Type 0)
Bit depth                                : 10 bits
Bits/(Pixel*Frame)                       : 0.121
Stream size                              : 11.2 GiB (97%)
Writing library                          : x265 3.5+1-f0c1022b6:[Linux][GCC 10.2.1][64 bit] 10bit
Encoding settings                        : cpuid=1176575 / frame-threads=4 / wpp / no-pmode / no-pme / no-psnr / no-ssim / log-level=2 / input-csp=1 / input-res=3840x1920 / interlace=0 / total-frames=0 / level-idc=0 / high-tier=0 / uhd-bd=0 / ref=3 / no-allow-non-conformance / repeat-headers / annexb / aud / no-eob / no-eos / hrd / info / hash=0 / no-temporal-layers / no-open-gop / min-keyint=96 / keyint=96 / gop-lookahead=0 / bframes=4 / b-adapt=2 / b-pyramid / bframe-bias=0 / rc-lookahead=96 / lookahead-slices=8 / scenecut=40 / no-hist-scenecut / radl=0 / no-splice / no-intra-refresh / ctu=64 / min-cu-size=8 / no-rect / no-amp / max-tu-size=32 / tu-inter-depth=1 / tu-intra-depth=1 / limit-tu=0 / rdoq-level=0 / dynamic-rd=0.00 / no-ssim-rd / signhide / no-tskip / nr-intra=0 / nr-inter=0 / no-constrained-intra / strong-intra-smoothing / max-merge=3 / limit-refs=1 / no-limit-modes / me=1 / subme=2 / merange=57 / temporal-mvp / no-frame-dup / no-hme / weightp / weightb / no-analyze-src-pics / deblock=0:0 / no-sao / no-sao-non-deblock / rd=3 / selective-sao=0 / early-skip / rskip / no-fast-intra / no-tskip-fast / no-cu-lossless / b-intra / no-splitrd-skip / rdpenalty=0 / psy-rd=2.00 / psy-rdoq=0.00 / no-rd-refine / no-lossless / cbqpoffs=0 / crqpoffs=0 / rc=crf / crf=11.0 / qcomp=0.60 / qpstep=4 / stats-write=0 / stats-read=0 / vbv-maxrate=21900 / vbv-bufsize=40000 / vbv-init=0.9 / min-vbv-fullness=50.0 / max-vbv-fullness=80.0 / crf-max=0.0 / crf-min=0.0 / ipratio=1.40 / pbratio=1.30 / aq-mode=2 / aq-strength=1.00 / cutree / zone-count=0 / no-strict-cbr / qg-size=32 / no-rc-grain / qpmax=69 / qpmin=0 / no-const-vbv / sar=1 / overscan=0 / videoformat=5 / range=0 / colorprim=9 / transfer=16 / colormatrix=9 / chromaloc=1 / chromaloc-top=0 / chromaloc-bottom=0 / display-window=0 / master-display=G(13250,34500)B(7500,3000)R(34000,16000)WP(15635,16450)L(10000000,1) / cll=662,211 / min-luma=0 / max-luma=1023 / log2-max-poc-lsb=8 / vui-timing-info / vui-hrd-info / slices=1 / no-opt-qp-pps / no-opt-ref-list-length-pps / no-multi-pass-opt-rps / scenecut-bias=0.05 / hist-threshold=0.03 / no-opt-cu-delta-qp / no-aq-motion / hdr10 / hdr10-opt / no-dhdr10-opt / no-idr-recovery-sei / analysis-reuse-level=0 / analysis-save-reuse-level=0 / analysis-load-reuse-level=0 / scale-factor=0 / refine-intra=0 / refine-inter=0 / refine-mv=1 / refine-ctu-distortion=0 / no-limit-sao / ctu-info=0 / no-lowpass-dct / refine-analysis-type=0 / copy-pic=1 / max-ausize-factor=1.0 / no-dynamic-refine / no-single-sei / no-hevc-aq / no-svt / no-field / qp-adaptation-range=1.00 / scenecut-aware-qp=0conformance-window-offsets / right=0 / bottom=0 / decoder-max-rate=0 / no-vbv-live-multi-pass
Default                                  : Yes
Forced                                   : No
Color range                              : Limited
Color primaries                          : BT.2020
Transfer characteristics                 : PQ
Matrix coefficients                      : BT.2020 non-constant
Mastering display color primaries        : Display P3
Mastering display luminance              : min: 0.0001 cd/m2, max: 1000 cd/m2
Maximum Content Light Level              : 662 cd/m2
Maximum Frame-Average Light Level        : 211 cd/m2

Not working:

Video
ID                                       : 1
Format                                   : HEVC
Format/Info                              : High Efficiency Video Coding
Format profile                           : Main 10@L5@High
HDR format                               : Dolby Vision, Version 1.0, Profile 5, dvhe.05.06, BL+RPU
Codec ID                                 : V_MPEGH/ISO/HEVC
Duration                                 : 40 min 19 s
Bit rate                                 : 24.0 Mb/s
Width                                    : 3 840 pixels
Height                                   : 1 920 pixels
Display aspect ratio                     : 2.000
Frame rate mode                          : Constant
Frame rate                               : 23.976 (24000/1001) FPS
Color space                              : YUV
Chroma subsampling                       : 4:2:0
Bit depth                                : 10 bits
Bits/(Pixel*Frame)                       : 0.136
Stream size                              : 6.75 GiB (95%)
Language                                 : English
Default                                  : Yes
Forced                                   : No

[MauriceW67]

@thexai @Hitcher I also tested some files in Dolby Vision format.

It seems this feature only works for DV files where mediainfo shows that the HDR format is HDR10 or HDR10+ compatible:

HDR format : Dolby Vision, Version 1.0, dvhe.08.06, BL+RPU, HDR10 compatible / SMPTE ST 2094 App 4, Version 1, HDR10+ Profile B compatible

As soon as the file only shows Dolby Vision without HDR10 compatibility, it won't work:

HDR format : Dolby Vision, Version 1.0, dvhe.05.06, BL+RPU

[thexai]

Fixed not working with some Dolby Vision profiles

[thexai]

@sarbes
I know the mapping may not be perfect but at the very least it's better than what's there now. In fact, if there are complaints it is because as it is now is incorrect, on the contrary no one has complained that there is banding in Windows that already implements this.

The most important thing here is to adjust the brightness intensity in OSD. The fact that there may be banding in the subtitles, which are generally solid white, or playback status bar I don't think is a problem.

Keep in mind that when HDR10 playback ends and is returned to the GUI, this is disabled.

[Hitcher]

Fixed not working with some Dolby Vision profiles

Confirmed working, thanks.

[sarbes]

@sarbes I know the mapping may not be perfect but at the very least it's better than what's there now. In fact, if there are complaints it is because as it is now is incorrect, on the contrary no one has complained that there is banding in Windows that already implements this.

Why not implement the mapping correctly in the first place? At least the part we can control is worth to get right, isn't it?

Then we only have the quantization loss.

[thexai]

An GLES shaders improvement can be made in a follow-up PR and also implement same for GL but but I'm not the one to do this and I'm not going to do it.

EDIT:
This PR is just the first step... Windows has needed 4 or 5 PRs to get what it has now:

#18984
#21973
#22756
#22827

[sarbes]

I'm sorry, but I have to object merging the PR in this state, especially since you don't want to work on it in the future.

The current execution is wrong, the texture shaders are not the right place for adjusting the intensity. Gamma correct scaling has to be done in a post processing pass, if you want to have any resemblance of color correctness. Such an implementation would be of similar scale. And the code complexity would be less, as it would be much cleaner overall.

I want to add such a pass in order to have a HDR UI some day. I can offer you to implement a basic version which deals with the full- to limited-range conversion. From there, you could easily append your brightness mapping.

If you like, we can talk about it on our next team call.

[thexai]

Why not implement the mapping correctly in the first place?

Can you be more specific? What is bad? and for you what is "correct mapping" (suggested changes, etc.)

It might also lengthen the execution time of the shaders, depending on the platform.

How a single float multiplication in shader code can slowdown execution?

Then we only have the quantization loss.

8-bit SDR is 256 steps and in shaders is 0.0 to 1.0 range

10-bit HDR is 1024 steps and in shaders is same 0.0 to 1.0 range

Even using only half of scale (0.5) there are 512 steps (double precise that SDR)

With default setting 40% is used 0.0 to 0.58 range

1024 x 0.58 = 593 steps

Where is quantization loss? And why 593 steps are not sufficient to render Kodi GUI that is SDR in origin (256 steps).

I am not a shaders expert and you know much more about this, but I consider these to be very basic concepts although I may be wrong about some things and have misconceptions.

[thexai]

If you like, we can talk about it on our next team call.

I have a normal job of 9 hours a day and night shifts. I can't spend a lot of hours on this.

[ksooo]

Why can't we go with this PR and improve / replace this code once @sarbes (or somebody else) comes up with something real (a PR)? I tend to prefer to have something not 100% now (!) instead of getting something "perfect" in the future. What speaks against an iterative approach? Just my 2ct.

[sarbes]

Because I would need to rip out half of the code. It has no place being there (the texture shaders).

The alternative I'm proposing would be much simpler for @thexai to implement (like 6 LOC).

I don't want to discourage you. I think it is a bit unfortunate that we have not talked about it before you did this PR.

[thexai]

I am not your employee (or anything like that). Here everyone works on what they like/interest.

If you were interested in this (which you are not) you would have already implemented it.

https://forum.kodi.tv/showthread.php?tid=376360&pid=3185207#pid3185207
https://forum.kodi.tv/showthread.php?tid=376360&pid=3185128#pid3185128

[sarbes]

Why not implement the mapping correctly in the first place?

Can you be more specific? What is bad? and for you what is "correct mapping" (suggested changes, etc.)

See above.

It might also lengthen the execution time of the shaders, depending on the platform.

How a single float multiplication in shader code can slowdown execution?

The Mali-400 series can do one MAD per cycle. In the worst case, you would double the amount of cycles needed. Similar for VideoCore GPUs.

In case of your PR, you pay the cycle cost per fragment. In case of a separate pass, you pay the cost only once.

Then we only have the quantization loss.

8-bit SDR is 256 steps and in shaders is 0.0 to 1.0 range

10-bit HDR is 1024 steps and in shaders is same 0.0 to 1.0 range

Even using only half of scale (0.5) there are 512 steps (double precise that SDR)

With default setting 40% is used 0.0 to 0.58 range

1024 x 0.58 = 593 steps

Where is quantization loss? And why 593 steps are not sufficient to render Kodi GUI that is SDR in origin (256 steps).

You need to remember that the framebuffer is just RGBA8.

If you like, we can talk about it on our next team call.

I have a normal job of 9 hours a day and night shifts. I can't spend a lot of hours on this.

Me too, I can relate to that.

[thexai]

Rebased only, no code changes.

[CrystalP]

I think the luminance mapping is good enough because the multiplier is calculated using a linear equation in CWinSystemAndroid::GetGuiSdrPeakLuminance() and the multiplied data is in PQ, so it's perceptually linear.

Unlike Windows where the multiplication is in linear space so the curve of the multiplier is exponential for a similar effect.

Is the fact that the primaries are converted by Android reliable? Even if not I guess not being blinded is better that inaccurate colors. Doing custom conversion to be certain would obviously be much more expensive for the GPU (similar to the Windows shader code).

[thexai]

Is the fact that the primaries are converted by Android reliable? Even if not I guess not being blinded is better that inaccurate colors.

Colors are totally correct (at least on Shield).

[sarbes]

But the fact is that I can't predict the future and I can't implement a thing in "resolve shader" that doesn't exist yet 🙂

I'm pretty vocal about my ideas and changes in our Slack, I would say. I also have quite a bunch of them written down as a project (https://github.com/orgs/xbmc/projects/5). But unfortunately, it seems that I'm mostly alone in both conversations. Aside from my current PRs, I do have quite a bunch of uncommitted code, which is waiting of the approval of the former. I'm always available to talk about changes falling in the GUI/GL/GLES domain.

I don't think Sabres has a DV high Nit TV or he would be prioritising a change that stops our retinas being blinded.

Those are assumptions on your side. My assumption of you is that you don't grasp the base of my objection.

Implement a 'basic' fix in this release that no-one has complained about except Sabre. Mark is as experimental in the Options if you're concerned.

Make no mistake, this is not a fix but a workaround for broken devices/software.

My "complaint" was never about improving the user experience of this release, but the execution. And I don't think that a lot of people are qualified to have remarks about the latter. So naturally, the pool of such people (thus the amount of "complaints") is small compared to our large and rather vocal user base.

Sabres' fix is currently scheduled for Kodo 22... ages away :(

It could be incorporated into 21. But we don't have many reviewers and it might be considered to big of a change for the beta. Personally, I'd be willing to risk it.

Reopened. At least 2 team members have expressed their opinion on Slack in favor of merging as is (and in v21):
...
Maybe it should be put to a vote or just allow it to be merged when it receives 2 approvals.

You have my vote as well, if you would consider working on it once the post processing pass gets merged. I don't have the setup and the time needed to qualify changes. My main objection was (is) mainly about the idea that I would need to take ownership about something I can't test.

[sarbes]

Is the fact that the primaries are converted by Android reliable? Even if not I guess not being blinded is better that inaccurate colors.

Colors are totally correct (at least on Shield).

While I don't have a suitable setup to test, I would expect it to be right (for most systems).

[thexai]

You have my vote as well, if you would consider working on it once the post processing pass gets merged. I don't have the setup and the time needed to qualify changes. My main objection was (is) mainly about the idea that I would need to take ownership about something I can't test.

I will work on it keeping in mind that GL/GLES is not my area and I also have limited time. I will try to do the best I can.

[sarbes]

I will work on it keeping in mind that GL/GLES is not my area and I also have limited time. I will try to do the best I can.

That's a statement I can perfectly live with.

I'll approve the PR from the the GLES side of things. It's probably best to get the green light from an Android guy and the release folks before merging.

[sarbes]

Approving the GLES changes.

[hugbug]

I noticed the brightness adjustment is activated depending on content instead of display mode. That works as long as the Fire TV option "Dynamic range settings" is set to "Adaptive".

With "Dynamic range settings" set to "Always HDR", Kodi UI is very bright until a HDR video is started. After stopping the video and returning to Kodi UI the full brightness is back.

[sarbes]

That's the idea. The adjustment is just for playback.

Kodi has no HDR mode (or similar) for the UI.

[Hitcher]

Yeah, you need to turn off 'Always HDR'.

[thexai]

"Always HDR" seems a feature of Fire TV only:

We could certainly add one more condition to activate "GUI in PQ mode" as long as Fire TV / Android provides an API to detect that this optional mode is active. In any case this does not need to be done in this PR as is "Fire TV" specific setting.

[Hlsgs]

"Always HDR" seems a feature of Fire TV only:

We could certainly add one more condition to activate "GUI in PQ mode" as long as Fire TV / Android provides an API to detect that this optional mode is active. In any case this does not need to be done in this PR as is "Fire TV" specific setting.

That setting is present on the Chromecast as well: "Always DV/HDR"(depending on what another pref is set to) and "Match content". That device skipped Android 11, so I can't confirm whether it was implemented in v11 or v12, but I presume it's on all devices with clean(ish) Android v12+. However, the behavior outlined by @hugbug is par for the course on any and all apps with that setting turned on, so I don't think it needs to be addressed by Kodi, but rather by Android itself, which is a whole nother can of worms.

All that beeing said, I would like to thank you for taking this on and hope that this gets merged in v21 as it's an absolute godsend. I've made a feature request for a separate subtitle color in HDR mode quite a while back, but this is the proper way to deal with all these issus.

[fritsch]

My reasoning was already cited (anonymously ;-))

[MauriceW67]

@thexai Am I correct in assuming that the option does not work for playing DV streams through Input Stream Adaptive, such as with the Disney+ addon?

[thexai]

No idea, but should work if video is currently played as Dolby Vision on TV: i.e. DV metadata is passed to MediaCodecSurface and DV capable display is identified inside Kodi settings information.

[MauriceW67]

No idea, but should work if video is currently played as Dolby Vision on TV: i.e. DV metadata is passed to MediaCodecSurface and DV capable display is identified inside Kodi settings information.

Strange. I played an episode of Secret Invasion through the Disney+ addon and my TV definitely switched to DV mode. But the OSD and subtitles stayed on the SDR brightness level.

Playing local DV content in MP4 or MKV format from my NAS works fine though.

[Hitcher]

Strange. I played an episode of Secret Invasion through the Disney+ addon and my TV definitely switched to DV mode. But the OSD and subtitles stayed on the SDR brightness level.

Playing local DV content in MP4 or MKV format from my NAS works fine though.

I can confirm the same behaviour playing a DV video in the D+ addon.

[thexai]

Add-on thing will be investigated later. May be necessary modify something in InputStreamAddon.cpp and require add-ons API version bump anyway (for v22 only).

The first step to continue working in this is merge this PR now...

[ilikenwf]

@thexai how hard would it be to add this to Linux as well? I use HDR with Kodi on Linux, my own builds, and can see how this would be a nice feature so as not to burn my eyes.

[josh4trunks]

@thexai how hard would it be to add this to Linux as well? I use HDR with Kodi on Linux, my own builds, and can see how this would be a nice feature so as not to burn my eyes.

I would appreciate this coming to Linux on x86_64 as well!
LibreELEC user here, and ever since getting an LG OLED a few months ago I realized GUI/SRT subtitles were an issue with HDR/DV content.