Question about latency

[stellanhaglund]

I'm doing a quick test to stream a recorded screen over webrtc but I'm experiencing multi second latencies, if you have any ideas that would help a lot!

here's the poc for the capturing part.

    thread::spawn( move ||  {
        let displays = Display::all().unwrap();
        let display = displays.into_iter().nth(0).unwrap();
    
        let mut capturer = Capturer::new(display).unwrap();
        let width = capturer.width() as u32;
        let height = capturer.height() as u32;    
    
        let (vpx_codec, mux_codec) = match Codec::Vp9 {
            Codec::Vp8 => (vpx_encode::VideoCodecId::VP8, mux::VideoCodecId::VP8),
            Codec::Vp9 => (vpx_encode::VideoCodecId::VP9, mux::VideoCodecId::VP9),
        };    
    
        let mut vpx = vpx_encode::Encoder::new(vpx_encode::Config {
            width: width,
            height: height,
            timebase: [1, 1000],
            bitrate: 5000,
            codec: vpx_codec,
        }).unwrap();            

        let start = Instant::now();
        let mut yuv = Vec::new();
        let spf = Duration::from_nanos(1_000_000_000 / 30);

        loop {

            let now = Instant::now();
            let time = now - start;        
            
            match capturer.frame() {
                
                Ok(frame) => {

                    let ms = time.as_secs() * 1000 + time.subsec_millis() as u64;

                    convert::argb_to_i420(width as usize, height as usize, &frame, &mut yuv);
                    
                    for frame in vpx.encode(ms as i64, &yuv).unwrap() {
                        let owned = frame.data.to_vec();
                        frame_tx.send(owned).expect("msg");
                    }
                }
                _ => { }
            }        

            // let dt = now.elapsed();
            // if dt < spf {
            //     thread::sleep(spf - dt);
            // }            
        }

    });    
    

And here's the reciever.

    
        tokio::spawn(async move {
        // Wait for connection established
        let _ = notify_video.notified().await;

        let sleep_time = Duration::from_millis(
            ((1000 * 1) / 30) as u64,
        );
        let mut ticker = tokio::time::interval(sleep_time);

        for frame in frame_rx {

            let mut byte_frame = BytesMut::with_capacity(frame.len());
            byte_frame.put(&frame[..]);                

            video_track
                .write_sample(&Sample {
                    data: byte_frame.freeze(),
                    duration: Duration::from_secs(1),
                    ..Default::default()
                })
                .await?;                

            let _ = ticker.tick().await;

        }

        Result::<()>::Ok(())
    });
    

I've tried both with and without the waiting parts.

[stellanhaglund]

It's a bit strange too because stuff is only moving on the stream if I'm actively moving something on the computer screen but whats moving on the stream is the movements from like 5 seconds back.

[k0nserv]

Sounds like it might be something with the capturing of the display that's causing this issue. If the code is otherwise very similar to one of the play from disk examples that leads me to believe the issue is on the source side

[k0nserv]

Also, the timebase should match the negotiated timebase of the stream typically, 90000hz i.e. [1, 90000]

[stellanhaglund]

vpx_encode is from @astraw maybe he has some input?

[k0nserv]

Did you change the timebase?

[stellanhaglund]

Yes, no difference.
I also changed so that there is no frames sent on the channel until the peer-connection is connected, I thought maybe it's sending to many frames so it doesn't have time to send them all over the udp connection, but that didn't help either.

I also tried the library that vpx_encode is using for the screen capture.
https://github.com/quadrupleslap/scrap/blob/master/examples/ffplay.rs to see if the capturing is too slow.
But that worked in near realtime from a 4k screen.

The theories I've had is.

• That the the capture is sending too many frames that should be added to the video track.
• There is something off with the sent frames that makes the times differ.
• The encoding is too slow.

I'm not experienced enough in either vpx or the technical parts of webrtc yet though to know exactly where to look.

[k0nserv]

I think the other issue is the hardcoded duration of the samples. I know the examples do this, but it's a bit dodgy. What you ought to do instead is specify the correct duration of the samples

[stellanhaglund]

Great thanks I will check that out!
you mean this right?
duration: Duration::from_secs(1)

[k0nserv]

Yes!

[stellanhaglund]

So I made the duration use the same amount of milliseconds that the frame capture loop does, in this case 40 and there was a big difference.
I also tried to change from vp9 to vp8 and there was some additional difference.
So now theres still a lag, bigger than if I share my screen between two incognito tabs in chrome, and aren't they using some sort of SFU?
I was hoping that vp9 could be a good thing for a remote desktop thing, eventually actually more like a pixel streaming thing for the bevy game engine where you could stream the gpu output.
Similar to the unreal engine pixel streaming plugin.

But when I try that locally I don't even notice a lag.

[stellanhaglund]

Ok, one reason could be that the pixel streaming plugin uses Nvidia's NVENC hardware encoder.

[stellanhaglund]

Seems like ffmpeg has support for serveral hardware encoders, would be interesting to try.

[morajabi]

@stellanhaglund it would be fantastic if you could contribute this as a simple example once you're happy with it! 🤩