d3d11vpp: add GPU scale option for intel and nvidia GPUs #14155

billxc · 2024-05-16T05:43:35Z

usage:
vf set d3d11vpp=scale=nvidia:scale-target

PS:
The pull requests is has not been rebased to one commit yet, as I still need some changes, and will rebase to one commit once ready for merge.

billxc · 2024-05-16T05:48:35Z

Quate the comment from #11390

@billxc First off, it's great that you're working on this, and thank you for looking at my original feedback. However, after taking an initial look at your tree, it looks like you have written it as a separate filter. For it to be mergeable, the functionality should be added to d3d11vpp. Were you just doing it as a separate filter as an initial development step? I can't tell.
Also, the filter's configuration interface should not hard code the "Super Resolution" concept - as I understand it, it's just a generic scaling function and vendors or anyone can write and plug in a scaling algorithm (I see you have added Intel as a second example), so it should just be a scale configuration. It should also support taking arbitrary width and height for the scaling output, as that is supported in the general case (does RTX Video only work for fixed resolutions?). Thanks!
Feel free to open a PR - it will be easier to discuss there.
Originally posted by @philipl in #11390 (comment)

Also to make it scalable for the future we could extend options of d3d11vpp with arbitrary GUID that user could provide. Would avoid having to recompile mpv for different variant of the same. I think AMD has some sort of super res too, which has to be enabled, because normally this interface just scales without fancy features.
Originally posted by @kasper93 in #11390 (comment)

billxc · 2024-05-16T06:06:05Z

@philipl

I have refacted the code to reuse d3d11vpp, also updatd the option name. There will be another change later to change SuperResolution to Scale to avoid potential copyright issue.

It should also support taking arbitrary width and height for the scaling output, as that is supported in the general case (does RTX Video only work for fixed resolutions?)

RTX video do support any resolution. Idealy, we should take the actual window size, and use them for the target height/width
but I am not quite familiar with MPV code, Would you help provide some examples for getting the actual window height and width?

@kasper93 The GPU API are not quite the same for different venders, as you can see from my code for nvidia and intel, they may be similar, but not the same, it is hard to simplely provide API for user to set the GUID, and it just works. AMD has super res which called FSR, but it is not a GPU API, the usage is not the same as intel/nvidia, there are some glsl shades implemented the FSR for MPV.

Also I have tried to use the intel auto HDR API, but failed, do we have examples for turning a SDR video to HDR video.

kasper93 · 2024-05-16T08:32:30Z

@kasper93 The GPU API are not quite the same for different venders, as you can see from my code for nvidia and intel, they may be similar, but not the same, it is hard to simplely provide API for user to set the GUID, and it just works. AMD has super res which called FSR, but it is not a GPU API, the usage is not the same as intel/nvidia, there are some glsl shades implemented the FSR for MPV.

I don't think they call their video hq scaler FSR in any documentation. Though indeed looking at it they did expose it through own interface, rather than rolling custom video processor in d3d11 video processing interface (to be confirmed if not available through it, not sure what browsers use). It could live in the same mpv filter (d3d11vpp), same as they integrated it in VLC.

See:
https://github.com/GPUOpen-LibrariesAndSDKs/AMF/blob/master/amf/doc/AMF_HQ_Scaler_API.md
https://code.videolan.org/videolan/vlc/-/merge_requests/4953

ruihe774 · 2024-05-16T08:55:53Z

The AMF doc says it supports

DirectX 11

DirectX 12

Vulkan

OpenCL

FWIW I'm curious about whether it is possible to be implemented as a cross-platform filter with both d3d11 and vulkan backends.

kasper93 · 2024-05-16T15:51:21Z

For AMD AMF stuff, it seems to be integrated in ffmpeg https://patchwork.ffmpeg.org/project/ffmpeg/list/?series=11632 so we can skip this part and have it for free once it get merged.

Jules-A · 2024-05-16T19:47:29Z

For AMD AMF stuff, it seems to be integrated in ffmpeg https://patchwork.ffmpeg.org/project/ffmpeg/list/?series=11632 so we can skip this part and have it for free once it get merged.

FFMPEG are being difficult again (https://www.phoronix.com/news/AMD-AMF-FFmpeg-Better-2024) so not really sure if it would get merged soon.

SaberDirewolf · 2024-05-16T22:50:20Z

@philipl

I have refacted the code to reuse d3d11vpp, also updatd the option name. There will be another change later to change SuperResolution to Scale to avoid potential copyright issue.

It should also support taking arbitrary width and height for the scaling output, as that is supported in the general case (does RTX Video only work for fixed resolutions?)

RTX video do support any resolution. Idealy, we should take the actual window size, and use them for the target height/width but I am not quite familiar with MPV code, Would you help provide some examples for getting the actual window height and width?

@kasper93 The GPU API are not quite the same for different venders, as you can see from my code for nvidia and intel, they may be similar, but not the same, it is hard to simplely provide API for user to set the GUID, and it just works. AMD has super res which called FSR, but it is not a GPU API, the usage is not the same as intel/nvidia, there are some glsl shades implemented the FSR for MPV.

Also I have tried to use the intel auto HDR API, but failed, do we have examples for turning a SDR video to HDR video.

Not sure if this is helpful, but was staring at how Chrome's Auto HDR works and is the original commit (there might be bug fixes after this commit): chromium/chromium@b2fae4c

Here is another example: Aleksoid1978/VideoRenderer#128

kasper93 · 2024-05-16T23:49:05Z

video/filter/vf_d3d11vpp.c

+            .scale = SUPER_RESOLUTION_OFF,
+            .scale_target = SUPER_RESOLUTION_AUTO,


This config interface is bloated. It should be simple and customizable.

scale should be floating point value of scaling that should be applied to source size. This can be extended to also include special options like target, which would get VO display size.

Current "scale" should be renamed to scale_method or similar. Although since currently none of the vendors offer selection, this can be made as a boolean value to enable high quality scalers.

Whole interface should work also without specifying those fancy GUIDs, d3d11vpp offers scaling as one of it's basic features, which should also be available for "normal" scaling.

Current "scale" should be renamed to scale_method or similar. Although since currently none of the vendors offer selection, this can be made as a boolean value to enable high quality scalers.

You mean none of the currently supported vendors? If this is expanded to AMD, they do support different methods with AMF_HQ_SCALER_ALGORITHM.

At current stage, there is no need for such option. scaling_algocan be added in the future when it will be useful.

kasper93 · 2024-05-16T23:51:35Z

video/filter/vf_d3d11vpp.c

    p->pool = mp_image_pool_new(f);
    mp_image_pool_set_allocator(p->pool, alloc_pool, f);
    mp_image_pool_set_lru(p->pool);

+    mp_refqueue_add_in_format(p->queue, IMGFMT_420P, 0);


This is big no-no in the filter itself. This is d3d11vpp processor and it accept only IMGFMT_D3D11 as an input. And hardware upload should be supported in different layer, possibly transparently, but not in this filer.

kasper93 · 2024-05-16T23:52:35Z

video/filter/vf_d3d11vpp.c

+                case SUPER_RESOLUTION_3X:
+                    window_w = 3 * in_fmt->w;
+                    window_h = 3 * in_fmt->h;
+                    break;


I already commented on the options, this is no go to maintain such list of arbitrary selected target resolutions.

kasper93 · 2024-05-16T23:53:44Z

video/filter/vf_d3d11vpp.c

+            SetSuperResNvidia(vf);
+            break;
+        case SUPER_RESOLUTION_INTEL:
+            SetSuperResIntel(vf);


Please follow https://github.com/mpv-player/mpv/blob/master/DOCS/contribute.md for codding style and naming. Generally following the style of current file is ok.

kasper93 · 2024-05-16T23:57:27Z

video/filter/vf_d3d11vpp.c

+        // mp_image_copy_attributes overwrites the height and width
+        // set it the size back if we are using scale
+        mp_image_set_size(out, p->out_params.w, p->out_params.h);
+        out->params.crop = bakup_crop;


typo, also crop value will not really be valid anymore after scaling. Crop value should be also rescaled with the size of the video.

kasper93 · 2024-05-16T23:57:51Z

video/filter/vf_d3d11vpp.c

+    ID3D11Texture2D *d3d_tex;
+    int d3d_subindex = 0;
+
+    if(in->imgfmt == IMGFMT_420P){


Wrong place to do any hardware upload.

video/filter/vf_d3d11vpp.c

kasper93 · 2024-05-17T00:00:06Z

video/filter/vf_d3d11vpp.c

+    if (p->sr_state != 0){
+        return;
+    }
+    GUID GUID_INTEL_VPE_INTERFACE = {


Use DEFINE_GUID macro, move it to the top of the file.

kasper93 · 2024-05-17T00:00:51Z

video/filter/vf_d3d11vpp.c

+		  0x4cbc,
+		  {0xa4, 0xd6, 0x98, 0x31, 0xa2, 0x16, 0x3a, 0xc3} };
+
+    enum : UINT {


Move all enum/struct definitions to the top or at least outside function.

kasper93 · 2024-05-17T00:03:59Z

video/filter/vf_d3d11vpp.c

@@ -72,6 +86,7 @@ struct priv {
    struct mp_image_pool *pool;

    struct mp_refqueue *queue;
+    int sr_state; // -1 for failed, 0 for off, 1 for on


This can be enum, tho avoid awkward comments explaining what each value mean.

kasper93 · 2024-05-17T00:07:13Z

Also needs fixes for gcc compilation and commits should be squashed into logical parts.

srk24 · 2024-05-17T01:39:54Z

is this helpful?
https://docs.nvidia.com/rtx/ngx/programming-guide/index.html#dlvsr

Hrxn · 2024-05-17T18:12:57Z

@srk24 What is this NGX? Is it even supported anymore?

philipl · 2024-05-17T21:57:11Z

@srk24 What is this NGX? Is it even supported anymore?

NGX is off topic for the work in this PR. It's nvidia's umbrella for some AI based image/video processing scalers - none of which are usable via the d3d11 scaling mechanism covered by this PR.

adjust order FIX: api usage adjust order update log refactor fix error logging logging fix error fix error update logging update update update update logging update update update Revert "update" This reverts commit 2642c41. update fixed 2K hardcode 2k test size test size test Revert "test" This reverts commit 596890b. Revert "test size" This reverts commit ef2b98a. Revert "test size" This reverts commit 38001eb. GNU GLOBAL LOGGING TEST logging fix logging Revert "fix logging" This reverts commit ec4f9c0. fix error revert logging remove logging because POC complete logging logging logging logging comment update update try fix update fix fix fix fix fix fix fix fix use 4K add option fix build more options fix build fix add defaults add vulkan for build add d3d11sr delete code remove processing rename Revert "rename" This reverts commit 82742d8. Revert "remove processing" This reverts commit c5acd59. Revert "delete code" This reverts commit 14250fb. Revert "add d3d11sr" This reverts commit 6bc9c1e. add d3d11sr disable deint fix blur fix size revert change fix warning remove logging add intel code rename field remove mode param remove unused variables Support SR selection add shorts revert d3d11vpp changes remove unused optmization handle SW decoding, dint not included stop processing if width or height is uneven cleanup cleanup Store SR state Revert "add vulkan for build" This reverts commit 0952b9f. revert unrelated code migrate back to d3d 11vpp

code format

billxc · 2024-05-18T13:34:57Z

updated the easy parts.
and still some TODOs:

Seeking correct place to do the uploading/downloading
Support any scale ratio
Support getting current VO size for scaling output

wadixx · 2024-05-28T18:22:33Z

updated the easy parts. and still some TODOs:

Seeking correct place to do the uploading/downloading

Support any scale ratio

Support getting current VO size for scaling output

Hey any update on TODOs?

billxc mentioned this pull request May 16, 2024

RTX Video Enhancement support #11390

Open

kasper93 requested changes May 17, 2024

View reviewed changes

billxc force-pushed the merge_upstream_sr branch from e2ae622 to 6c06704 Compare May 18, 2024 12:29

code style change

c59aa08

code format

billxc force-pushed the merge_upstream_sr branch from 84df7ac to c59aa08 Compare May 18, 2024 13:23

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

d3d11vpp: add GPU scale option for intel and nvidia GPUs #14155

d3d11vpp: add GPU scale option for intel and nvidia GPUs #14155

billxc commented May 16, 2024

billxc commented May 16, 2024

billxc commented May 16, 2024

kasper93 commented May 16, 2024 •

edited

ruihe774 commented May 16, 2024 •

edited

kasper93 commented May 16, 2024

Jules-A commented May 16, 2024

SaberDirewolf commented May 16, 2024

kasper93 May 16, 2024

Jules-A May 17, 2024

kasper93 May 17, 2024

kasper93 May 16, 2024

kasper93 May 16, 2024

kasper93 May 16, 2024

billxc May 18, 2024

kasper93 May 16, 2024

kasper93 May 16, 2024

kasper93 May 17, 2024

billxc May 18, 2024

kasper93 May 17, 2024 •

edited

billxc May 18, 2024

kasper93 May 17, 2024

billxc May 18, 2024

kasper93 commented May 17, 2024

srk24 commented May 17, 2024

Hrxn commented May 17, 2024

philipl commented May 17, 2024

billxc commented May 18, 2024

wadixx commented May 28, 2024

		.scale = SUPER_RESOLUTION_OFF,
		.scale_target = SUPER_RESOLUTION_AUTO,

d3d11vpp: add GPU scale option for intel and nvidia GPUs #14155

Are you sure you want to change the base?

d3d11vpp: add GPU scale option for intel and nvidia GPUs #14155

Conversation

billxc commented May 16, 2024

billxc commented May 16, 2024

billxc commented May 16, 2024

kasper93 commented May 16, 2024 • edited

ruihe774 commented May 16, 2024 • edited

kasper93 commented May 16, 2024

Jules-A commented May 16, 2024

SaberDirewolf commented May 16, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kasper93 May 17, 2024 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kasper93 commented May 17, 2024

srk24 commented May 17, 2024

Hrxn commented May 17, 2024

philipl commented May 17, 2024

billxc commented May 18, 2024

wadixx commented May 28, 2024

kasper93 commented May 16, 2024 •

edited

ruihe774 commented May 16, 2024 •

edited

kasper93 May 17, 2024 •

edited