[BUG]: H264 encoding problem with NvCodec.

[SetoKaiba]

Package version

3.0.0-pre.4

Environment

* OS:Windows 10
* Unity version:Unity 2020.3

Steps To Reproduce

https://github.com/SetoKaiba/srs-unity
Use this project to produce the bug.

1. Run a srs server with docker, http://ossrs.io/lts/en-us/docs/v5/doc/getting-started
2. Use Streamer/SceneNormal to stream the unity camera to srs server.
3. Open http://localhost:8080/console/en_index.html#/streams?port=1985
4. You will see the stream can't be played
5. If you open the player before the streamer, the stream can be played. And if you open a player after that, the new player can't be played as well.
6. If I recompile the libwebrtc with rtc_use_h264=true proprietary_codecs=true ffmpeg_branding="Chrome", and disable the nvcoded encoder. The stream can be played no matter the play is before or after the push.

Current Behavior

No response

Expected Behavior

No response

Anything else?

No response

[winlinvip]

I think the root cause is the NVCodec does not support PLI, which is essential for WebRTC.

SRS is a WebRTC SFU, it works like this:

Publisher --WebRTC---> SRS ---WebRTC--> Player

Note: About the SFU, there is a post about it.

If start publisher before player, the player will request PLI to publisher, because in WebRTC the keyframe might lost or new participants will join the video room which need a new keyframe for decoding.

If publisher does not response for the PLI request, the player never get a keyframe and can't decode the video stream.

[SetoKaiba]

@winlinvip Hi, what's the following lines meaning? Does it matter? The following lines are printed all along with the stream pushing.

[2023-03-20 04:38:04.447][INFO][1][8600d62x] set ts=1345082689, header=32051, lost=32052
[2023-03-20 04:38:04.453][WARN][1][060vu898][11] avc ignore type=1 for no sequence header

[SetoKaiba]

@winlinvip @karasusan I found out the problem.

com.unity.webrtc/Plugin~/WebRTCPlugin/Codec/NvCodec/NvEncoderImpl.cpp

Lines 445 to 450 in 0bfa9ff

	picParams.encodePicFlags = 0;
	if (send_key_frame)
	{
	picParams.encodePicFlags = NV_ENC_PIC_FLAG_FORCEINTRA | NV_ENC_PIC_FLAG_FORCEIDR;
	m_configurations[0].key_frame_request = false;
	}

Change it to the lines below. Add output spspps solve the problem.

        picParams.encodePicFlags = 0;
        if (send_key_frame)
        {
            picParams.encodePicFlags = NV_ENC_PIC_FLAG_FORCEINTRA | NV_ENC_PIC_FLAG_FORCEIDR | NV_ENC_PIC_FLAG_OUTPUT_SPSPPS;
            m_configurations[0].key_frame_request = false;
        }

UPDATE: reminded by @johzzy. The SPSPPS is only needed for keyframe. It solves the problem with verification.

[karasusan]

@SetoKaiba @winlinvip
Hi, I am trying NV_ENC_PIC_FLAG_OUTPUT_SPSPPS flag with SRS. I checked it works well.
As @SetoKaiba said, the streaming with SRS is failed when not using NV_ENC_PIC_FLAG_OUTPUT_SPSPPS flag.

We will make the PR for solving this issue, do you have any concerns?

[winlinvip]

@winlinvip Hi, what's the following lines meaning? Does it matter? The following lines are printed all along with the stream pushing.

[2023-03-20 04:38:04.447][INFO][1][8600d62x] set ts=1345082689, header=32051, lost=32052
[2023-03-20 04:38:04.453][WARN][1][060vu898][11] avc ignore type=1 for no sequence header

The sequence header is SPS/PPS for H.264, or SPS/PPS/VPS for H.265, it's just a short name.

No sequence header means no SPS/PPS, so it's not able to decode the video stream.

The NV_ENC_PIC_FLAG_OUTPUT_SPSPPS should be set when encoder generate a IDR frames(generally keyframes). And for WebRTC, a keyframe also be sent with SPS/PPS, because the SPS/PPS might change.

I confirmed with @johzzy, it works for me, thanks a lot @SetoKaiba @karasusan

[winlinvip]

We are also working on adding WHIP support to FFmpeg. Please see ossrs/ffmpeg-webrtc#1. After this feature is merged, Unity will be able to communicate with FFmpeg applications.