Direct integration of NVIDIA NVENC H.264 and H.265 encoder

[Xaymar]

Integrating NVIDIA NVENC directly allows us far more control than we can have via FFmpeg, and may even allow us to edge out extra quality and performance from NVIDIA NVENC. This encoder should support the zero-copy encoding mode already possible with FFmpeg, as well as multi-GPU encoding mode. The multi-GPU mode should prioritize performance impact over latency and memory usage.

[Xaymar]

The following features are most likely not useful to this integration:

• MVC/Stereo Encoding is not supported by OBS Studio, though users could split the scene vertically to do this.
• SVC Encoding, as it is a needless waste of performance.
• Variable Framerate Encoding (VFR)due to OBS Studio running at a fixed time interval no matter the input, and NVIDIA NVENC not actually supporting it for Rate Control.
• Flexible Macroblock Ordering (FMO) since the underlying network protocol handles resilience against losses.
• Motion Estimation only encoding.
• Interlaced, Checkerboard, etc.

Possible extra interest by end-users in the following features:

• Asynchronous encoding for better GPU/Encoder utilization.
• Alpha channel support in HEVC.
• Dynamic Resolution and Bitrate changes.
• IDR/I-only encoding for lossless capture.
• Long Term Reference detection and support.
• Single/Multi-Slice and Intra-refresh coding.
• HDR and Full Range support

[Xaymar]

Regarding Color Format

At the current time, NVIDIA does not expose a 4:2:2 encoding mode for NVENC, only 4:2:0 and 4:4:4. Therefore the encoder should snap to the nearest lower or equal format, and expose it as a preferred format.

Therefore, the required color formats are: 4:2:0, 4:4:4, RGB

[Xaymar]

Regarding Multi-Slice and Intra-Refresh

These have no real usage for end users and are a nice to have, but not required for fulfilling this feature request. They allow users to have slightly more resilient or lower latency streams at a massive cost to quality. Additionally it is not confirmed if this encoding mode is even still supported by NVIDIA hardware at all.

[Xaymar]

Regarding Dynamic Resolution

As OBS Studio itself does not support resolution changes on-the-fly, this feature is a nice-to-have when tied into #848. It is not required for implementation.