[Feature request] Add Nvidia Maxine AI Video Upscaler

[wywywywy]

Is your feature request related to a problem? Please describe.

Nvidia has released a full SDK of their Maxine tech (also used in their Broadcast app and the up-coming browser video upscaling).

The SDK package includes pre-compiled binaries and models, so it should be easy to integrate with Waifu2x-Extension-GUI.

From my limited testing of real-life videos, the best quality is VideoEffectApp with SuperRes and mode 0, and the end result is slightly better than Waifu2x, but not quite as good as Real-ESRGAN. It is however at least 5x faster than other upscalers on my RTX3090!

(Pending more testing: comparison pics, actual timed durations, anime videos)

Describe the solution you'd like

Have Maxine as an additional upscaler, just like how we have ESRGAN.

Describe alternatives you've considered

The upscaler can be used standalone, if @AaronFeng753 decides not to implement it into Waifu2x-Extension-GUI.

SET PATH=%PATH%;..\external\opencv\bin;..\..\bin;
VideoEffectsApp.exe --effect=SuperRes --mode=0 --model_dir=..\..\bin\models --in_file=..\input\in.mp4 --resolution=1080 --out_file=out.mp4

Additional context

Limitation: The video can only be scaled to 1.3333x, 1.5x, 2x, 3x or 4x times the input, no arbitrary resolution.

General info: https://developer.nvidia.com/maxine

SDK package download: https://catalog.ngc.nvidia.com/orgs/nvidia/teams/maxine/resources/maxine_windows_video_effects_sdk_ga (Nvidia account required to download)

Usage guide: https://docs.nvidia.com/deeplearning/maxine/vfx-sdk-system-guide/index.html

Github but use the SDK download instead because that comes with everything: https://github.com/NVIDIA/MAXINE-VFX-SDK

SDK guide but probably not relevant to us: https://docs.nvidia.com/deeplearning/maxine/vfx-sdk-programming-guide/index.html

[AaronFeng753]

Thank you, I definitely will look into this

[wywywywy]

Comparison number 1

1. 624x336 source REAL-LIFE
2. 4x Maxine mode 0
3. 4x Waifu2x noise 0
4. 4x Waifu2x noise 1
5. 4x Waifu2x noise 2

https://imgsli.com/MTQ1MTAx/0/1

[wywywywy]

Comparison number 2

1. 624x336 source REAL-LIFE
2. 4x Maxine mode 0
3. 4x Waifu2x noise 0
4. 4x Waifu2x noise 1
5. 4x Waifu2x noise 2

https://imgsli.com/MTQ1MTA2/0/1

[wywywywy]

Comparison number 3 (spoiler: it's a disaster for Maxine)

1. 251x180 source ANIME
2. 4x Maxine mode 0
3. 4x Waifu2x noise 0
4. 4x Waifu2x noise 1
5. 4x Waifu2x noise 2

https://imgsli.com/MTQ1MTEw/0/1

[AaronFeng753]

Thanks, I've already started working on this

[AaronFeng753]

[AaronFeng753]

✅ https://github.com/AaronFeng753/Waifu2x-Extension-GUI/releases/tag/v3.98.18-beta

[wywywywy]

Good idea calling it "RTX SuperResolution" rather than "Nvidia Maxine". Much more direct and easier to understand.