DLSS Implementation

[DarkameMaster]

Would it be possible to add DLSS and make anything sharper? I'm just curious!

[nagadomi]

DLSS cannot be used for video, but RTX VSR could be supported.

If you only want to make the video sharper, you can use an unsharp mask with the vf (src) option: unsharp. For more details, please see the ffmpeg documentation.
It currently cannot be used with HWAccel=cuda, but I plan to add support for that later.

[nagadomi]

It might be possible to force it to work if the inputs are adjusted correctly, but I do not see a good reason to use it.

The estimated depth and motion vectors are not as accurate as those from a game engine, and they also add a large processing cost at high resolutions. Considering the intended use of the model, I assume it was not trained with low quality depth input, so I would not expect good results.

Using a VSR model designed for video should be more efficient and probably give better results.

That said, this is only my expectation because I have not benchmarked it myself. If you are interested, please try it yourself.

[DarkameMaster]

Thank you so much for your responses. I'll start looking into the logic of the code and see how it goes. If you could give me a hint on how the motion vector output works or where I should start looking, I'd really appreciate it :)

[nagadomi]

There are some mentions about optical flow in #619, but an AI explanation would probably be easier to understand. It is best to make a simple video conversion script yourself.

Also, removing compression artifacts is very important for video super resolution, but DLSS probably does not take that into account.
I think the VSR documentation mentions this specifically.
As for VSR, I have not tested it myself, but there is sample code in https://github.com/NVIDIA-Maxine/nvidia-vfx-python-samples/blob/main/video_super_resolution.py, so it should be easy to try.

[John9520]

Hello,
Nvidia Super Resolution can be add on OBS in downloading the video effect from this link:
https://www.nvidia.com/fr-fr/geforce/broadcasting/broadcast-sdk/resources/
In more you need to install the folder from this Github:
https://github.com/Bemjo/OBS-RTX-SuperResolution
A tutorial is available on the second link.
If you need more FPS and a better resolution Lossless Scaling can do the job too, you need to buy this software and launch OBS and Lossless Scaling as administrator and capture the [LosslessScaling.exe]: (null) on OBS once you used Lossless Scaling on the window of your real time conversion, I guess you can use that too on files converted with Iw3 GUI but I haven't tested.
Like this I can look videos with more FPS and a virtual enhancement of the resolution.

[DarkameMaster]

Yeah, I’ve already checked and tried those options. Currently, I’ve been using Lossless Scaling, but there’s a modder who has successfully implemented a DLSS mod for OBS that supports FG, and it’s way better than Lossless Scaling. However, it’s still unknown whether he’s going to release it publicly.

At least the modder left some hints. According to him, DLSS works surprisingly well with estimated depth vectors and optical flow vectors. So the technology to make this possible is already available, and someone has already managed to do it. I just wanted to make some progress too because I know this has a lot of potential.