[REQUEST] Upgrade NVIDIA video codec SDK to support the profile level for high resolution video

[karasusan]

This post is so helpful for us and many developers want this fix.
https://zenn.dev/blkcatman/articles/c2cb8517fd290b

Related issues posted from users.
Unity-Technologies/UnityRenderStreaming#517
FusedVR/VRStreaming#7

[vmohan7]

+1 Would be very appreciated

[doctorpangloss]

I implemented this fix but without a corresponding change to libwebrtc the SDP still says 5.x instead of 6.2. It is hard to say why it does or does not matter for the particular user.

[yvanzine]

I implemented this fix but without a corresponding change to libwebrtc the SDP still says 5.x instead of 6.2. It is hard to say why it does or does not matter for the particular user.

@doctorpangloss

libwebrtc does not support encoder levels higher than 5.2 and a constrained base profile. That is also a Mozilla specification for webrtc.

[yvanzine]

@doctorpangloss @karasusan

Can we reopen this bug BUG with not being able to exceed 2.5mbps ?

What is happening is that we are completely unable to set bitrate higher than 2.5mbps. This causes inability to increase resolution without running into artifacts. Context::SetRates is passing a maximum of 2.5mbps under all circumstances. I upgraded hardware encoder to level 6.2 but it is largely unnecessary because 5.2 can do 240 Gbps easily. This is probably caused by client or sdp negotiation. I debugged this extensively and higher resolution is locked away by this very low and limiting bitrate.

[karasusan]

@yvanzine
Sure. But I cannot promise when it will be fixed...
Our focus since the next month is performance improvement contains this issue.

[yvanzine]

@karasusan

I will try to explain about the upgrade to a newer nvidia encoder that supports level 6.2. The upgrade of nvcodec (nvidia driver 350+) prevents a few crashes but it does not bring about quality improvements. While the encoder is configured at 6.2 level, the libwebrtc library can only be configured at the maximum, 5.2 encoder level and constrained base profile. Still, this profile can accommodate 240 gbps at 4k resolution. The main problem is the maximum practical bitrate is never above 2.5mbps and we can never achieve higher quality at hd resolution or higher at higher framerates, e.g. 4k@30fps. The client via Context::SetRates is always sending less than 2.5mbps to reconfigure settings.

[karasusan]

I will check the bitrate limitation next month.

note:
Definitions of h264 levels in Chromium.
https://source.chromium.org/chromium/chromium/src/+/main:third_party/webrtc/api/video_codecs/h264_profile_level_id.h;l=32-50

[karasusan]

memo: WRS-151

[karasusan]

Made a PR draft to upgrade SDK but not tested yet.
#575

[karasusan]

We are checking 4k@30fps using H.264 5.1 profile level.
In WebRTC, encoder and decoder have to supports the same profile level to use.
https://github.com/Unity-Technologies/com.unity.webrtc/blob/develop/Plugin~/WebRTCPlugin/Codec/NvCodec/NvCodec.cpp#L145-L150