-
Notifications
You must be signed in to change notification settings - Fork 428
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Multiple freezes on WebRTC H264 stream #317
Comments
Well done on getting OpenH264 wired up! Do you have any example code anywhere? Is it a C# OpenH264 wrapper? I'd love to test it. Also to start with I'd switch to Also the Other than that without being able to test I don't have much of an idea where the freezes could come from. My H264 RTP packetization logic is very new and not that well tested and could certainly be faulty. The only testing I did was with FFmpeg and I guess it's possible OpenH264 could be providing NAL units differently. If you're able it would be worth testing with my FFmpeg video end point and setting the codec to H264 if you can. That would help to rule out if your problem is at the sampling or encoding stage. The Mp4Source is the ideal example to test out FFmpeg with. using System;
using System.Threading;
namespace test
{
class Program
{
static void Main(string[] args)
{
Console.WriteLine("Timer callback test");
//DateTime start = DateTime.Now;
//TimerCallback callback = new TimerCallback((state) =>
//{
// double period = DateTime.Now.Subtract(start).TotalMilliseconds;
// Console.WriteLine("{0}ms", period);
// start = DateTime.Now;
//});
//Timer timer = new Timer(callback, null, 0, 33);
DateTime start = DateTime.Now;
for (int i = 0; i < 1000; i++)
{
double period = DateTime.Now.Subtract(start).TotalMilliseconds;
Console.WriteLine("{0}ms", period);
start = DateTime.Now;
Thread.Sleep(33);
}
Console.WriteLine("Press any key to exit.");
Console.ReadLine();
}
}
} |
Hi again, thanks for the quick answer. I used OpenH264Lib which is a wrapper around H264 Cisco DLL, I'll gladly make a repository showing how I did it. I think I tried with your FFMpeg endpoint too and ended up getting the same issue, i'll try again. Right now the stream is coming from an IDS Camera, the stream is transformed to YUV4:2:0 using OpenCV then fed to the H264 encoder and then sent to SendVideo() I'll let you know when I create the repository so you can take a look |
I pushed a little example I threw together quickly, don't mind the code quality. A generator creates a YUV Test frame that is encoded with OpenH264Lib and sent using your library.
Here is the repository (https://github.com/NoOverflow/SipSorceryH264Example) |
Great, thx. I've started playing around with your sample and will massage it a bit to get it working with the test pattern video source. I would definitely recommend simplifying your encode loop. It's going to be the critical path in your app and introducing queueing and callbacks just introduces unnecessary thread context switches and overhead plus the code is harder for you to write. Just keep it simple and all on the same thread your sampling your video capture device on:
No callbacks, no queues. If the app can't keep up with the above pipeline then all you can do is decrease the frame rate or resolution. You've added more overhead to the pipeline by doing: Thread 1: |
Okay I'll keep that in mind, I won't use that ConcurrentQueue, however I did not use any thread in the example provided. If you're talking about the OnEncode callback, this is forced by OpenH264Lib and I unfortunately have no control on that :( |
I distilled your sample down a bit WebRTCTestPatternServerOpenH264. The advantage is that I know the test pattern can generate a reliable smooth source on my PC. After wiring it up to the OpenH264 wrapper I do get similar behaviour to what you've described with the periodic short freezes. The good news is that the problem is now isolated to the OpenH264 encoder wrapper (I'd be pretty confident the openh264 dll is not the problem given its maturity). I've only had a quick look at the wrapper but it does seem to be some some weird stuff and the As far as fixing it I'm getting away from Managed C++ as it's too problematic to distribute due to not being cross platform and requiring a specific version C++ runtime. I've deprecated my Managed C++ sipsorcery-media project for that reason. The better solution is to use But (as you're no doubt thinking) that's a lot more work and if you just want to get things working and only need to support Windows then fixing the Managed C++ OpenH264Lib is a reasonable option. |
Thanks a lot for taking your time with this. I will probably fork and modify the encoder wrapper directly to test, and see if it's worth removing it completely in order to use PInvokes. I will also try using another encoder such as x264 see if that changes anything. Anyway I think since this issue does not concern your package but the encoder wrapper I'll close it. Keep up the good work ! |
I forgot to say I did make a start on a PInvoke wrapper for OpenH264 a while ago. I didn't get very far before something more pressing came along. Just in case it's of any use. |
Thanks, I'll try and use that and finish it if I ever need to, I'll let you know If I do ! |
Just a little update, I tried to use another encoder (X264), and the issue disappeared, so you were right. The issue is indeed coming from OpenH264Lib ! |
@sipsorcery Issue disappeared after updating from 4.0.79-pre to 4.0.85-pre, maybe related to #321 ? |
Nice to know, thanks. And it certainly could have been improved/fixed by #321. My original attempt at parsing H264 byte streams was faulty and @rafcsoares hopefully fixed it. Different H264 encoders supply byte streams in different formats which makes it tricky to test. I did the original work and testing with FFMpeg which was using |
Anyways, glad that is fixed. By the way you were right about C++/CLI wrappers, I'm going to stop using that and code one using P/Invokes, much better. Have a nice day ! |
Hi, first of all i've been using your library for a week and it's a real pleasure to use, thanks for that.
I've been trying to implement an H264 WebRTC stream using your library and OpenH264Lib,
I've got everything setup and the stream works fine, but there are some huges freezes when sending IDR frames,
Do you have any idea where that could come from ?
Right now i'm just sending each encoded frame using:
RTCPeerConnection.SendVideo(fps * 100, encodedFrameBuffer); Thread.Sleep(1000 / fps);
and encoder setup:
this.encoder.Setup(1224, 920, 5000 * 1000, 24, 2, new OnEncodeCallback(OnEncode));
I'm relatively new to encoding and webRTC so it might be coming from the values i'm using (such as bitrate or duration)
The text was updated successfully, but these errors were encountered: