Profiling help - any simple way to see frame decode/display time?

[peterjk83]

Hi!

This is a great project - very helpful.

I have a live video stream set up and decoding/displaying properly.

My performance is a bit - well - crap right now. I'm guessing the frame decode and display time is longer than it needs to be, and so frames are buffering up and a long delay builds between where the server is in the stream and what the client is displaying.

So I fired up Chrome profiler. I guess the information I need is somewhere in here, but it also seems overkill for what I need now. I'm not an expert using it, it all seems rather confusing :) At the moment I'm in a cycle of tweaking encoder settings on the server side and observing the client side performance changes. I really just need to see the time it takes between a frame arriving and a frame being displayed on screen.

Is there any debug/timings mode in Broadway itself that might console.log per frame processing times? Or might someone be able to guide me on how to use Chrome's profiler to figure that out?

Thanks for any help! :) :)

[soliton4]

hey
i assume you are using worker threads since its the faster way to decode.
https://github.com/mbebenita/Broadway/blob/master/Player/Player.js#L160

have a look at the 2nd parameter of the decode function.
it will allow you to pass an object that you will get back in the onPictureDecoded callback.

note that you will get an array of info objects back. this represents the collection of info objects passed to the decoder before this frame was decoded. (only relevant if you use intermediary frames)
check if you get more than 1 object per frame. if so change your encoder parameters.

you can pass a time stamp in the info object. that will tell you how long decoding took.

use requestanimationframe to do the actual rendering. that way the rendering will not be a bottleneck.

note that non webgl rendering is way slower and the color conversion happens in the worker thread instead of webgl.

if decode time or latency is still to high consider combining multiple streams.
i had great results when splitting a fullscreen video stream into 4 parts.

i think that sums up about every performance tip i can give you

[peterjk83]

Thank you so much soliton4. I'm going to give all this a try and let you know how I get on :)

[peterjk83]

Just an update - I have the frametime capturing via the 'infos' object working, and it works perfectly! Finally can see some easy-to-comprehend numbers :)

I didn't know about requestanimationframe. I've implemented it like this:

window.requestAnimationFrame(function() { var d = new Date(); var n = d.getTime(); var bin = toUint8Array( e.data); var info = {timestamp:n}; player.decode(bin, info); });

i.e. I'm sticking the player.decode in the callback.

Is that how one would do it, or should I be doing something differently?

I should note on a first run this seemed to bring the decode time down by ~17% :) :)

Thanks so much for your help soliton!

[soliton4]

i can not give you a complete explaination of requestanimationframe.
read up on it and familiarize yourself with the reason why it exists.

it can help you to decouple the decoding from the rendering.

decode to memory using the player parameter render
https://github.com/mbebenita/Broadway/blob/master/Player/Player.js#L108

then call the renderFrame function in requestanimationframe.

i am not sure this will help you at all, just a pointer towards something you might want to experiment with. it only makes sense if your video streams fps is higher than your monitors refresh rate or your webgl rendering is significantly slow.

[peterjk83]

Thank you! I will do my research and experiment with this.

Right now my laptop is giving me ~30hz at 960x544 (quarter HD). Kind of hopeful I can do better. We shall see!

[cexcell]

@peterjk83 , do you have any improvements with it? I have pretty much the same problem. First frame is decoded with significant lag so stream has constant latency ~1 second.

[peterjk83]

Hey @cexcell - Through @soliton4's advice and some other experimentation, my decode speed is around 20ms per frame on my laptop. That's at fairly modest resolutions - quarter-HD or lower. I haven't yet looked into splitting the video into tiles to be processed by multiple workers, though - might be able to bump the resolution that way.

I suppose we should say that latency and decode speed are not quite the same thing - are you talking about, for example, input latency in a remote desktop type scenario? Or just the local decode speed?

If we're talking about total system latency from server to client, lots of things are at play.

If we're talking about local decode speed - aside from the things @soliton4 mentioned, I noticed that tweaking encode settings on the server side had a fairly dramatic impact on client decode speed. If bitrate was too high or too low, for example, it could affect client decode speed quite a bit. I also noticed that for whatever reason, in my case, using a web worker was actually a little slower than just doing everything in the main thread. I don't know the specifics of your encoder, but I noticed GOP and i-frame related settings had a bit impact on local decode performance also.

Overall I would advise tinkering with both server-side encode settings as well as the client to see what has the biggest impacts in your case.

[cexcell]

@peterjk83 , thank you for your answer. I'm trying to implement something like remote desktop with NVEnc accelerated encoding, so i have h264 frames from nvenc and send it via WebSocket to client side. First frame for client to decode is always IDR with SPS and PPS, and decode speed for this frame is awful (time difference between finish and start ~2500ms).
Again, thank you for your explanation. I will try to tweak some NVEnc parameters and maybe i will get acceptable results.

[peterjk83]

@cexcell I should say, I did notice often a slight delay in the initial frames on my feed, though things would quickly catch up and stabilise such that I wasn't 'stuck' with that delay through the remainder of the feed. It didn't have a knock-on effect on latency beyond the initial frames of the feed, but it was there for those first few frames.

Wishing you luck! 👍

[cexcell]

@peterjk83 , Thank you :-)

UPDATE: You were right about tweaking server-side. Now i start 60fps 1080p stream and it works almost perfectly! 'First frame' issue is still present but it doesn't affect total perfomance. Wow!

[peererror]

@peterjk83 @soliton4 @peterjk83 I am also experiencing the same issue can you please recommend some ffmpeg settings for 1080p video streaming it would be really help full for me .

[soliton4]

ffmpeg -y -i sourceFile -r 30000/1001 -b:a 2M -bt 4M -vcodec libx264 -pass 1 -coder 0 -bf 0 -flags -loop -wpredp 0 -an targetFile.mp4
copied from readme.markdown @peererror ;)

[soliton4]

@ASnegoff like what?

[soliton4]

not sure why this thread is still open. im gonna close it. if you still need help feel free to open a new one