RTX Video Enhancement support

[germn]

There was an RTX Video Enhancement release recently.
https://nvidia.custhelp.com/app/answers/detail/a_id/5448

Basically, it allows real-time AI video upscaling on recent RTX video cards.
It would be nice to have this feature available in mpv as well.

There's already a release for MPC with support of this:
https://github.com/emoose/VideoRenderer/releases/tag/rtx-1.0

[kasper93]

There is already vf_d3d11vpp.c so it is actually trivial to extend it to select correct post processing extension. I personally don't have hardware to work with and implementing it blindly is never fun, so let's wait for another brave soul to add this.

[JoseHidalgo]

Just wanted to support this idea. No doubt that a MPV hero will take on this mission. :) The new feature is supposed to work not only on 4xxx cards but on 3xxx cards also.

Caution, there may be some issues to tackle : memory leaks on some HEVC 10bit videos, or the enhancements not working on vertical videos. See here:
https://www.reddit.com/r/nvidia/comments/11e7e2h/comment/jacujw9/?utm_source=reddit&utm_medium=web2x&context=3
emoose/VideoRenderer#1 (comment)

[askasys-code]

I'm totally interessed to have this feature in mpv.
I tried RTX VSR on MPC-BE and almost completely beat many upscalers I use with mpv, like FSRCNNX. It is absolutely worth it.

[cloud11665]

I can take a look at this after getting access to a 40 series GPU

[Myaanee]

I can take a look at this after getting access to a 40 series GPU

If you need a tester @ me, I have a 4080

[NekuSoul]

I can take a look at this after getting access to a 40 series GPU

As a side-note to avoid possible misconceptions: RTX Video is also available for the RTX 30xx series.

[cloud11665]

I currently have a 2080, and I will have access to a 4070 in ~2-3 weeks

[askasys-code]

Tested on vlc-3.0.19 and it works. Now only mpv remain, I really hope it will be supported soon. Maybe with a lua script? A toggle option is mandatory in my opinion, because both MPC-HC and VLC have VSR always ON and it's annoying.

[360hzz]

yes we are waiting for MPV my RTX 3080

[saha1914]

there is already a VSR for VapourSynth https://github.com/HolyWu/vs-basicvsrpp only if someone would have build it

This doesn't seem to have anything to do with the NVIDIA VSR based on d3d11vpp.

really sorry ooooppps

[cloud11665]

update: I now have access to a 40-series GPU, just need to set up my devenv on windows, and I can take a closer look.

[deama]

update: I now have access to a 40-series GPU, just need to set up my devenv on windows, and I can take a closer look.

Any luck?

[radozd]

Aaaaand?

[kasper93]

Aaaaand?

Aaaaand, I can implement that, just need a GPU. We can discuss shipping/payment details in email. You can find my email address in my profile. (my nick on gmail)

[Holdo]

@kasper93 RTX VSR 1.5 now works on RTX 2000 series as well, don't need 4000 series.

[deama]

Aaaaand?

Aaaaand, I can implement that, just need a GPU. We can discuss shipping/payment details in email. You can find my email address in my profile. (my nick on gmail)

So I guess this was a joke or were you trying to scam someone?

[kasper93]

So I guess this was a joke or were you trying to scam someone?

Scam of what? It has been months and looks like no one capable of adding this feature to mpv cares to do so. Like I said, I don't have NVIDIA GPU, nor I intend to buy one, so I won't work on this feature, unless someone contribute the GPU. And since there is constant nagging in here to get it implemented I offered my help and stated my price. I was mostly joking, because I don't expect you guys understand how contracting works, but messages like Aaaaand? won't persuade anyone to work on this feature.

Look at my recent commit history in here if you call out scams... If anything I'm scammed of my free time working on this project.

[deama]

So I guess this was a joke or were you trying to scam someone?

Scam of what? It has been months and looks like no one capable of adding this feature to mpv cares to do so. Like I said, I don't have NVIDIA GPU, nor I intend to buy one, so I won't work on this feature, unless someone contribute the GPU. And since there is constant nagging in here to get it implemented I offered my help and stated my price. I was mostly joking, because I don't expect you guys understand how contracting works, but messages like Aaaaand? won't persuade anyone to work on this feature.

Look at my recent commit history in here if you call out scams... If anything I'm scammed of my free time working on this project.

I think it looks pretty good.

[Andarwinux]

VSR has no status feedback, so there is no reliable way to confirm VSR is active. This feature was designed primarily for Chromium, which is planning to move to D3D12, so D3D11VPP VSR will certainly be dropped eventually, or even removed from drivers at some point. Adding VSR may not be worth it at this time.

[kenpuluma]

So I guess this was a joke or were you trying to scam someone?

Scam of what? It has been months and looks like no one capable of adding this feature to mpv cares to do so. Like I said, I don't have NVIDIA GPU, nor I intend to buy one, so I won't work on this feature, unless someone contribute the GPU. And since there is constant nagging in here to get it implemented I offered my help and stated my price. I was mostly joking, because I don't expect you guys understand how contracting works, but messages like Aaaaand? won't persuade anyone to work on this feature.

Look at my recent commit history in here if you call out scams... If anything I'm scammed of my free time working on this project.

If you consider two messages between Jun and Oct as "constant nagging" and it bothers you, and you require sponsorship to start your work then probably you shouldn't spend your time for an open source project. Complains like yours won't persuade anyone away from asking for new features, explanations like @Andarwinux's will.

[tliron]

If you consider two messages between Jun and Oct as "constant nagging" and it bothers you, and you require sponsorship to start your work then probably you shouldn't spend your time for an open source project. Complains like yours won't persuade anyone away from asking for new features, explanations like @Andarwinux's will.

People like you make me hate volunteering my work for open source.

Your sense of entitlement is absurd. Nobody owes you this feature. And also nobody owes you an in-depth explanation of the technical or personal or monetary or whatever challenges keep it from being implemented as fast as you might like.

The feature was asked for. The issue was not closed, so it's welcome. If you want it so much, develop it yourself and submit a PR. Otherwise, sit yourself down and wait patiently and politely and gratefully for the project's volunteers to do the work for you, free of charge, while you type "And?".

[kenpuluma]

If you consider two messages between Jun and Oct as "constant nagging" and it bothers you, and you require sponsorship to start your work then probably you shouldn't spend your time for an open source project. Complains like yours won't persuade anyone away from asking for new features, explanations like @Andarwinux's will.

People like you make me hate volunteering my work for open source.

Your sense of entitlement is absurd. Nobody owes you this feature. And also nobody owes you an in-depth explanation of the technical or personal or monetary or whatever challenges keep it from being implemented as fast as you might like.

The feature was asked for. The issue was not closed, so it's welcome. If you want it so much, develop it yourself and submit a PR. Otherwise, sit yourself down and wait patiently and politely and gratefully for the project's volunteers to do the work for you, free of charge, while you type "And?".

I'm not begging you for development, I'm not even the one who typed "And". I'm just searching online and happened to found this issue, and it looks to me that the discussion was normal until somebody asks for a payment (I guess it's a joke).

If you hate volunteering your work for open source then don't do it, like you said nobody is forcing you.

[Andarwinux]

I'm not trying to convince people to give up with VSR, I just think people should be more patient with it. I'm sure no one want VSR add to mpv as a half-baked product, and then no one ever wanting to maintain it again, as is the case with d3d11vpp now.

[llyyr]

Complains like yours won't persuade anyone away from asking for new features

Patches welcome

[N-R-K]

[...] probably you shouldn't spend your time for an open source project

This is not your place to state what others should or should not do in their own time.

the discussion was normal until somebody asks for a payment (I guess it's a joke)

Perhaps you're from some communist utopia but asking for payment for your work is nothing abnormal around the rest of the world.

And if you're so bothered by someone not wanting to spend hundreds of dollars to implement a feature for free for randoms on the internet, then go ahead and be the change you wish to see in the world. Buy yourself a compatible gpu and send the patches.

[kenpuluma]

I really don't want to populate the Github issue with meaningless, unrelated messages in a but if you insist ok:

This is not your place to state what others should or should not do in their own time.

Agreed. Github issue is the place for tracking work and collaboration on the project, not for making jokes and spreading hates and stress.

Perhaps you're from some communist utopia but asking for payment for your work is nothing abnormal around the rest of the world.

Don't try to change the subject. It's definitely normal for asking payments for anything you have spent time on, lots of people work on open source projects for a living.
But the discussion went pretty off the track and became completely irrelevant about its original topic: "RTX Video Enhancement support" now, people started defending themselves on scam or not scam, be thankful to open source contributors or not , beginning from that payment joke.

If this is how you want this repository looks like, then it's fine, leave people there to fight against each others.

[ghost]

@kenpuluma

Hey.
Basically I'm not going to implement it (the NVIDIA VSR)
I know... UGH!! I know... I'm sorry!!!!
I'm just not gonna implement it is all
HAHAHAHAHAHA

[Andarwinux]

Intel actually also support VSR (Arc only) but like NVIDIA there is no status feedback and Intel's VSR has no gpu control panel switches or options and the effect is much weaker, you never know if VSR is active or not.

[kasper93]

VSR has no status feedback, so there is no reliable way to confirm VSR is active.

How the driver handles the scaling is indeed not visible. But adding generic d3d11vpp scaling is good either way. Even if not VSR, it would still scale, deepening on vendor with various quality.

Also note that passing frames through video processor can/will also apply other adjustments that they have in pipeline, which in most cases is not wanted.

This feature was designed primarily for Chromium, which is planning to move to D3D12, so D3D11VPP VSR will certainly be dropped eventually, or even removed from drivers at some point. Adding VSR may not be worth it at this time.

I think you infer too much. d3d11 or d3d12 is only an API, there is no need to remove the scaler from d3d11 video processor to make it work for d3d12 video processor. Unless there would be needed API extension.

Intel actually also support VSR (Arc only) but like NVIDIA there is no status feedback and Intel's VSR has no gpu control panel switches or options and the effect is much weaker, you never know if VSR is active or not.

Similar situation, we just request the scaling. It is on driver control panel to expose proper feedback/control.

[haasn]

Don't listen to paid AMD shills and false flaggers trying to subvert this issue

[kasper93]

@kasper93 RTX VSR 1.5 now works on RTX 2000 series as well, don't need 4000 series.

AMD has it's own upscaling now. You don't even need NVIDIA gpu to implement this. Just the latest AMD ones. (at least for the AMD, probably different guids and so on)

AMD Video Upscaling – Advanced video upscale algorithm to improve video playback image quality for AMD Radeon™ RX 7000 desktop series GPUs.

https://www.amd.com/en/support/kb/release-notes/rn-rad-win-24-1-1

...

[Andarwinux]

For Google Chrome and Microsoft Edge, 'Media Foundation for Clear' must be enabled. This setting can be configured in the browser settings accessed through chrome://flags/ or edge://flags/.

[Hrxn]

I assume this'll be the default setting for Chrome/Edge somewhat soon?

BTW, the description blurb for this flag is:

Enable/Disable the use of MediaFoundation for non-protected content playback on supported systems. – Windows

Implying that MediaFoundation is already used for Widevine content etc.

[Andarwinux]

But mpv doesn't seem to have any MediaFoundation support, what if AMD's Video Upscaling is only accessible through MediaFoundation?

[wadixx]

MPC-HC, Potplayer and VLC has already started using RTX VSR. Can we expect to see this in MPV soon?

[billxc]

Having study the code of MPC Render, and chromium. The magic code is quite easy

    constexpr GUID kNvidiaPPEInterfaceGUID = {
        0xd43ce1b3,
        0x1f4b,
        0x48ac,
        {0xba, 0xee, 0xc3, 0xc2, 0x53, 0x75, 0xe6, 0xf7}};
    constexpr UINT kStreamExtensionVersionV1 = 0x1;
    constexpr UINT kStreamExtensionMethodSuperResolution = 0x2;

    struct {
        UINT version;
        UINT method;
        UINT enable;
    } stream_extension_info = {kStreamExtensionVersionV1,
                                kStreamExtensionMethodSuperResolution,
                                1u};

    hr = ID3D11VideoContext_VideoProcessorSetStreamExtension(p->video_ctx,p->video_proc,&kNvidiaPPEInterfaceGUID,
        sizeof(stream_extension_info), &stream_extension_info);

I tried to add this code to d3d11vpp.c and enable the the filter by using vf set d3d11vpp=deint=no.
But nothing happened, no super res, even no crash.
I am new to MPV, will work to find out if the codes are included in the compile.

bcaefa6

emoose/VideoRenderer@a684e0a
chromium/chromium@46f000a#diff-993a15e7a79f90944c791b0c616715be012bf762dd8f7a18bc27e8a597fd18fb

[wadixx]

Having study the code of MPC Render, and chromium. The magic code is quite easy

    constexpr GUID kNvidiaPPEInterfaceGUID = {
        0xd43ce1b3,
        0x1f4b,
        0x48ac,
        {0xba, 0xee, 0xc3, 0xc2, 0x53, 0x75, 0xe6, 0xf7}};
    constexpr UINT kStreamExtensionVersionV1 = 0x1;
    constexpr UINT kStreamExtensionMethodSuperResolution = 0x2;

    struct {
        UINT version;
        UINT method;
        UINT enable;
    } stream_extension_info = {kStreamExtensionVersionV1,
                                kStreamExtensionMethodSuperResolution,
                                1u};

    hr = ID3D11VideoContext_VideoProcessorSetStreamExtension(p->video_ctx,p->video_proc,&kNvidiaPPEInterfaceGUID,
        sizeof(stream_extension_info), &stream_extension_info);

I tried to add this code to d3d11vpp.c and enable the the filter by using vf set d3d11vpp=deint=no. But nothing happened, no super res, even no crash. I am new to MPV, will work to find out if the codes are included in the compile.

bcaefa6

emoose/VideoRenderer@a684e0a chromium/chromium@46f000a#diff-993a15e7a79f90944c791b0c616715be012bf762dd8f7a18bc27e8a597fd18fb

Looking forward to hearing from you.

[billxc]

Update the status: Now the RTX Super Res works, but with a little hard code, and NOT ready for merge.

If you are curious about the code, just refer to https://github.com/billxc/mpv.

This relies on the D3D11VPP filter and seems that it cannot be used together with vapoursynth filter.
@kasper93 could you help confirm if it is expected, and is there any way to make both filters work together?

[wadixx]

Update the status: Now the RTX Super Res works, but with a little hard code, and NOT ready for merge.

If you are curious about the code, just refer to https://github.com/billxc/mpv.

This relies on the D3D11VPP filter and seems that it cannot be used together with vapoursynth filter. @kasper93 could you help confirm if it is expected, and is there any way to make both filters work together?

You are a hero. Hope @kasper93 can merge this into mpv.

[philipl]

Update the status: Now the RTX Super Res works, but with a little hard code, and NOT ready for merge.

If you are curious about the code, just refer to https://github.com/billxc/mpv.

This relies on the D3D11VPP filter and seems that it cannot be used together with vapoursynth filter. @kasper93 could you help confirm if it is expected, and is there any way to make both filters work together?

The vapoursynth filter is a CPU filter, so it's never going to work directly. You would need to do a GPU upload/download (depending on the order of operations) to the frames into the right place for the other filter. I expect ffmpeg has upload/download support for d3d11va so it ought to be possible, but you'd need to work out how to construct the correct filter chain yourself, and you may still find additional limitations getting in your way.

A mergeable implementation would require extending the d3d11vpp filter to have a scaling mode with proper specification of the output frame size/format/etc and then a way to select the scaling mode (of which RTX Video is just one of many).