-
-
Notifications
You must be signed in to change notification settings - Fork 3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Tonemapping function relying on OpenCL filter and NVENC HEVC decoder #3442
Conversation
So, hopefully dumbm question but why does this use OpenCL and GPU offloading specifically? I see you briefly mention what I assume is the reason, but is ffmpeg and/or CPU support for tonemapping that subpar? Also, what about AMD GPUs? They can transcode quite a bit as well and are well priced. Typically better for a media server than an Intel iGPU after all. |
There are currently two efficient tonemap methods in ffmpeg, one is To be honest, the current situation of AMD AMF is a bit awkward. Until now, ffmpeg only had its encoder, but could not find the supporting decoder and scaler. This means that some operations cannot be fully executed in the GPU, which reduces the throughput. Their team tried to submit these missing components in 2018 but somehow they were not accepted, and I am trying to push these implements to restart. |
070a185
to
48e3f2b
Compare
What about vaapi decoding on amd? what would amf bring? |
Understood, thanks for the clarification, hopefully the patches can be pushed through this time |
Yay, more conflicts |
Resolved again. |
// The left side of the dot is the platform number, and the right side is the device number on the platform. | ||
OpenclDevice = "0.0"; | ||
EnableTonemapping = false; | ||
TonemappingAlgorithm = "reinhard"; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I personally prefer hable
or mobius
to reinhard
look for most films.
Here were some tests: #415 (comment) (and the rest of that thread)
Also the filters seem to really like if the input is scaled to float first. (And I suspect the GPU doesn't really mind)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This hardware accelerated tonemap_opencl
filter is different from tonemap
. Its average brightness is calculated in real time, which means it will dynamically adjust the brightness according to the threshold
when the scene brightness changes greatly. I also made some attempts on hable
, but failed to get a suitable brightness as reinhard
.
If you also have an nvidia card, you can also try this PR to find better parameters.
Sweet work dude! I'm on the amd gpu side of things as of now, and trying to get real time transcode going. GPU is an rx5600xt on linux (kernel 5.4.4) using Pop!_os 20.04 lts (built on top of ubuntu) As of now, it looks like openCL is completely borked for navi 10 based cards, and a lack of Vaapi_tonemap support means that performance when using vaapi for encode+decode helps a little, but not a lot. in terms of pure cli arguments, I'm running: ffmpeg -vaapi_device /dev/dri/renderD128 -hwaccel_output_format vaapi -i input.mp4 -threads 24 -vf 'zscale=transfer=linear,tonemap=hable,zscale=transfer=bt709,format=yuv420p,hwmap' -c:v hevc_vaapi output.mp4. Running without the gpu net's around 3 fps, running with runs at 5. Any ideas while we wait for proper AMF on linux? |
As per https://www.amd.com/en/support/kb/release-notes/rn-amdgpu-unified-linux-20-30
You can follow the instructions here to install If all things above is installed, you can run #4171 add tonemapping for AMD AMF. I've tried it on Windows but it should works on Linux if AMF and OpenCL is properly installed. |
Seeing an issue initializing the OpenCL device. I have a p2000 passed through. Not quite sure what I should look at.
edit: This is likely an issue with my setup, I am sure it has nothing to do with the feature was just seeing if anyone had an ideas. |
Seems that the ffmpeg cannot load any OpenCL platforms. You may need to install opencl icd loader in the container. Or may be you can follow the guidance from NVIDIA to step up OpenCL runtime in docker. I think OpenCL should works on the host once you have installed their propriety driver and nvidia-opencl. I will give it a try in container and add some docs then. |
It worked! In my case I am using unraid with the linuxserver Nvidia drivers. Their image at the moment does not support OpenCL, I had to use aptitude to install nvidia-opencl-icd but its working great. GOOD JOB! This is really AWESOME!! |
@blazarr can you give some more details on what you did? I'm running the same setup (jellyfin-unstable on unraid with nvidia drivers), I can't get the nvidia-opencl-icd package installed inside the Jellyfin container. Did you have to add the nvidia sources in the container first? |
Run the below commands in the container then restart. Turn off auto updates, an update will require you rerun these commands.
|
That worked. Thanks a ton! Is the plan to have that baked into the image when it moves to stable? |
It's |
This feels like an issue - the way it stands right now, on systems where the nvidia-opencl-icd loader has to be installed inside the container (ie: unRAID), the official image won't be able to tonemap, and will fail at even transcoding HDR content with tonemapping enabled |
The tonemap option is disabled by default until you have configured it well. |
It looks like on my card each stream consumes a flat 1GB of video memory. Does this vary or does is the always a static amount? |
It's controlled by the OpenCL driver for the maximum performance. |
@LP0101 From what I have tested, if you use the official docker image |
@TonyRL |
Ok so I am trying to figure out how many 4k HDR transcodes different system setups can handle. It appears that you get 1 transcode per 1GB of memory on the Nvidia cards (I am sure you need a decent cpu as to not bottleneck for any off gpu work involved ?). As for support for intel hardware, how would you say this does on resources, kind of wondering how many streams a 9th gen Intel CPU could pull off, does it just use 1GB or ram? I know you have been pretty busy with all this stuff so no worries if you can't get to this, I am just trying to better understand the resources consumed. Do you have plans to optimize around the intel 11th gen chips with support for hardware tonemapping? |
There is also a solution which does not involve adding debian packages. The only file missing is /etc/OpenCL/vendors/nvidia.icd which has the following content: |
Hey @nyanmisaka (long time no see! ;) ) Would you mind sharing the effective ffmpeg command that this is passing to the GPU to avoid redundant memory copies for gpu memory back to system and back again between decoding-> openCL->encoding? |
Generally speaking, most HDR videos have a resolution of 4k or higher. In addition to OpenCL, the pipeline for transcoding 4k video also requires much more VRAM. As for the intel Gen 9/9.5 iGPU, it uses VRAM from memory. Therefore, its internal bandwidth and its OpenCL performance should be considered before reaching the VRAM bottleneck. I don't think the mainstream HD/UHD 630 iGPU can compare to a GTX 1050 in the number of simultaneous tone mapping processes.
Currently Intel only shipped TigerLake or newer iGPUs on some notebooks, which I don't plan to buy. It is reported that these new iGPUs will be in the 11th gen desktop Core processors, and I will look at them then. |
It's nice to hear that additional dependencies can be avoided by creating a small text file. I guess it contains manifest for nvidia opencl runtime. |
In the current ffmpeg, sharing P010 textures between opencl and other decoders/encoders are not available on nvidia and amd. Memory copy will inevitably happen. |
If i understand correctly, this is because, in the case of the intel system, it's using standard system ram, mapped to the iGPU, whereas the other 2 are typically discrete cards, moving data over the pcie bus. In the case of amd APU's or nvidia embedded devices, would the system not avoid the transfer between the 2 areas, being as they're using the same memory pool?
is this also the case for nv12? if so, could we convert to nv12 and remove 1 of the redundant copies? [Trying to think of how to lower overhead and broaden the usability to be as close to optimal on as many platforms as needed ] |
I think both Intel iGPU and AMD GPU/APU are capable of mapping data between HWA ASIC and OpenCL devices. Intel has implemented this in VAAPI through But for some reasons ffmpeg rejected the Here's a comparison,
And a nice example to show the real performance of AMD transcoder. I'll try to dig into AMD AMF and impl related functions into ffmpeg. As for Nvidia I don't see any reason for them to put extra resources on OpenCL beyond the basic CPU/d3d11<-->OpenCL data exchange. FYI https://stackoverflow.com/questions/65157235/pass-ffpmeg-opencl-filter-output-to-nvenc-without-hwdownload
Nope. |
In the meantime I'm trying to develop a cuda based tonemap filter. Do you have any experience with those filters? if so, would you mind taking a look at this issue in my repo? trying to create a low power embedded HW accelerated cluster. Issue is here: FCLC/Multi-Plexer#5 (comment) |
Changes
These PRs rely on the
tonemap_opencl
filter based on the hardware acceleration of the OpenCL device, can cooperate with the NVENC HEVC decoder to perform tone mapping from HDR to SDR while maintaining a decent transcoding speed.The current CPU-based tonemap method is very unsuitable for real-time transcoding in terms of speed.
Requirements
opencl
hwaccel type enabled.ffmpeg -hwaccels
Shows
Windows 10 + Driver 451.67
4K HEVC HDR10 -> 1080P 15M 20M
Sony_4K60_HDR_Camp.mp4
![cmp01](https://user-images.githubusercontent.com/14953024/85730563-53c3b400-b72c-11ea-9af9-05fd7bb9c3a0.png)
SONY.BRAVIA.OLED.4K.HDR.DEMO_Food .Fizzle.mp4
![cmp02](https://user-images.githubusercontent.com/14953024/85730742-7ce44480-b72c-11ea-835a-ec4bc27e73d8.png)
NVIDIA GTX 1050 4G
Stats(1050)
![gtx1050](https://user-images.githubusercontent.com/14953024/88537553-5aea2480-d040-11ea-879a-1a60f8dac92c.png)
NVIDIA GTX 1660
Stats(1660)
![stats_1660](https://user-images.githubusercontent.com/14953024/90014110-1a91d400-dcd9-11ea-92aa-0d4abf248562.png)
Dashboard
![dashboard](https://user-images.githubusercontent.com/14953024/85732005-9639c080-b72d-11ea-9657-9ee27086ad5f.png)
Issues
#415 was partially resolved with NVIDIA GPU.