Multiple freezes on WebRTC H264 stream

[NoOverflow]

Hi, first of all i've been using your library for a week and it's a real pleasure to use, thanks for that.

I've been trying to implement an H264 WebRTC stream using your library and OpenH264Lib,
I've got everything setup and the stream works fine, but there are some huges freezes when sending IDR frames,

Do you have any idea where that could come from ?

Right now i'm just sending each encoded frame using:
RTCPeerConnection.SendVideo(fps * 100, encodedFrameBuffer); Thread.Sleep(1000 / fps);

and encoder setup:

this.encoder.Setup(1224, 920, 5000 * 1000, 24, 2, new OnEncodeCallback(OnEncode));

I'm relatively new to encoding and webRTC so it might be coming from the values i'm using (such as bitrate or duration)

[sipsorcery]

Well done on getting OpenH264 wired up! Do you have any example code anywhere? Is it a C# OpenH264 wrapper? I'd love to test it.

Also to start with I'd switch to Timer callbacks instead of Thread.Sleep. The latter is not reliable enough for generating real-time media samples. Run the sample below to see for yourself. Hopefully you're using .NET Core. The Timer callbacks on .NET Framework are not as reliable.

Also the SendVideo call needs to be supplied with RTP timestamp units not milliseconds. If you're sending at 30 fps then set each timestamp unit is 3000 since the default video sampling rate is 90KHz, 90000 / 30 = 3000. You've gotten away with it in your example but if you changed the frame rate to something different like 5 or 10fps your calculation will be wrong.

Other than that without being able to test I don't have much of an idea where the freezes could come from. My H264 RTP packetization logic is very new and not that well tested and could certainly be faulty. The only testing I did was with FFmpeg and I guess it's possible OpenH264 could be providing NAL units differently.

If you're able it would be worth testing with my FFmpeg video end point and setting the codec to H264 if you can. That would help to rule out if your problem is at the sampling or encoding stage. The Mp4Source is the ideal example to test out FFmpeg with.

using System;
using System.Threading;

namespace test
{
    class Program
    {
        static void Main(string[] args)
        {
            Console.WriteLine("Timer callback test");

            //DateTime start = DateTime.Now;
            //TimerCallback callback = new TimerCallback((state) =>
            //{
            //    double period = DateTime.Now.Subtract(start).TotalMilliseconds;
            //    Console.WriteLine("{0}ms", period);
            //    start = DateTime.Now;
            //});

            //Timer timer = new Timer(callback, null, 0, 33);

            DateTime start = DateTime.Now;
            for (int i = 0; i < 1000; i++)
            {

                double period = DateTime.Now.Subtract(start).TotalMilliseconds;
                Console.WriteLine("{0}ms", period);
                start = DateTime.Now;
                Thread.Sleep(33);
            }

            Console.WriteLine("Press any key to exit.");
            Console.ReadLine();
        }
    }
}

[NoOverflow]

Hi again, thanks for the quick answer.

I used OpenH264Lib which is a wrapper around H264 Cisco DLL, I'll gladly make a repository showing how I did it.
Right now I'm using .net framework, but i'm thinking about switching to .net core since I may need the code to run on other platforms.

I think I tried with your FFMpeg endpoint too and ended up getting the same issue, i'll try again.

Right now the stream is coming from an IDS Camera, the stream is transformed to YUV4:2:0 using OpenCV then fed to the H264 encoder and then sent to SendVideo()

I'll let you know when I create the repository so you can take a look

[NoOverflow]

I pushed a little example I threw together quickly, don't mind the code quality.

A generator creates a YUV Test frame that is encoded with OpenH264Lib and sent using your library.
I've noticed the same problem with the freeze and figured out it was happening following this behaviour:

Stream works fine
Encoder generates IDR Frames and tries to send them
Stream freezes
Encoder continues then generates another set of IDR Frames
Stream resumes

Here is the repository (https://github.com/NoOverflow/SipSorceryH264Example)

[sipsorcery]

Great, thx.

I've started playing around with your sample and will massage it a bit to get it working with the test pattern video source.

I would definitely recommend simplifying your encode loop. It's going to be the critical path in your app and introducing queueing and callbacks just introduces unnecessary thread context switches and overhead plus the code is harder for you to write. Just keep it simple and all on the same thread your sampling your video capture device on:

sample video capture -> pixel conversion -> H264 encode -> RTP packetisation and send

No callbacks, no queues. If the app can't keep up with the above pipeline then all you can do is decrease the frame rate or resolution. You've added more overhead to the pipeline by doing:

Thread 1: sample video capture -> queue
Thread 2: dequeue -> pixel conversion -> encode
Thread 3: encode callback -> RTP packetisation and send

[NoOverflow]

Okay I'll keep that in mind, I won't use that ConcurrentQueue, however I did not use any thread in the example provided. If you're talking about the OnEncode callback, this is forced by OpenH264Lib and I unfortunately have no control on that :(

[sipsorcery]

I distilled your sample down a bit WebRTCTestPatternServerOpenH264. The advantage is that I know the test pattern can generate a reliable smooth source on my PC.

After wiring it up to the OpenH264 wrapper I do get similar behaviour to what you've described with the periodic short freezes. The good news is that the problem is now isolated to the OpenH264 encoder wrapper (I'd be pretty confident the openh264 dll is not the problem given its maturity).

I've only had a quick look at the wrapper but it does seem to be some some weird stuff and the OnEncode callback is the main thing. It doesn't need to do that it could just return the encoded sample directly from the Encode call.

As far as fixing it I'm getting away from Managed C++ as it's too problematic to distribute due to not being cross platform and requiring a specific version C++ runtime. I've deprecated my Managed C++ sipsorcery-media project for that reason. The better solution is to use PInvoke directly on the native dll. That's what I'm doing for libvpx in SIPSorceryMedia.Encoders.

But (as you're no doubt thinking) that's a lot more work and if you just want to get things working and only need to support Windows then fixing the Managed C++ OpenH264Lib is a reasonable option.

[NoOverflow]

Thanks a lot for taking your time with this. I will probably fork and modify the encoder wrapper directly to test, and see if it's worth removing it completely in order to use PInvokes. I will also try using another encoder such as x264 see if that changes anything.

Anyway I think since this issue does not concern your package but the encoder wrapper I'll close it. Keep up the good work !

[sipsorcery]

I forgot to say I did make a start on a PInvoke wrapper for OpenH264 a while ago. I didn't get very far before something more pressing came along. Just in case it's of any use.

[NoOverflow]

I forgot to say I did make a start on a PInvoke wrapper for OpenH264 a while ago. I didn't get very far before something more pressing came along. Just in case it's of any use.

Thanks, I'll try and use that and finish it if I ever need to, I'll let you know If I do !

[NoOverflow]

Just a little update, I tried to use another encoder (X264), and the issue disappeared, so you were right. The issue is indeed coming from OpenH264Lib !

[NoOverflow]

@sipsorcery Issue disappeared after updating from 4.0.79-pre to 4.0.85-pre, maybe related to #321 ?

[sipsorcery]

Nice to know, thanks. And it certainly could have been improved/fixed by #321. My original attempt at parsing H264 byte streams was faulty and @rafcsoares hopefully fixed it.

Different H264 encoders supply byte streams in different formats which makes it tricky to test. I did the original work and testing with FFMpeg which was using libx264 under the hood. I'm pretty sure @rafcsoares is using OpenH264 which would also add weight to the fact that it's likely what fixed this issue.

[NoOverflow]

Anyways, glad that is fixed. By the way you were right about C++/CLI wrappers, I'm going to stop using that and code one using P/Invokes, much better.

Have a nice day !