[BUG] high latency for inbound audio track

[AndrewMeadows]

We're seeing high latency (up to 2.0 seconds) between speaking into the microphone of the remote peer and playing the audio at the receiving peer . We think it is caused by a growing webrtc jitter buffer for the inbound audio track, however we are still trying to investigate and learn more.

To Reproduce
(1) Unity Peer1 on machine1 connect to C++ webrtc server
(2) Chrome browswer Peer2 on machine2 connect to C++ webrtc server
(3) Note: server does spatialization mixing and relays results to Peer1 and Peer2
(4) Peer1 speaks into mic...
(5) Peer2 hears audio with low latency (0.25 - 0.5 seconds)
(6) Peer2 speaks into mic ...
(7) ... Peer1 hears audio with high latency (1.0 - 2.0 seconds) BUG!

It is intermittent! Sometimes we can get a lower latency (0.25 - 0.5 seconds) by doing the following:
(8) Mute the mic on peer2 and wait several seconds.
(9) Unmute the mic on Peer2 and speak...
(10) ... Peer1 sometimes hears audio with lower latency but eventually the latency grows to high levels again.

Note: we also tested on Windows but there only noticed high latency: we couldn't use the mute/unmute trick to get intermittent low latency audio. We didn't try this for very hard: two or three times. We mostly just wanted to verify that it wasn't a linux-only problem.

Expected behavior
The latency for audio from Peer1 to Peer2 is low (0.25 - 0.5 seconds). We would expect similar latency in the other direction: Peer2 to Peer1.

Theory: is Unity draining webrtc's audio buffer too aggresively?
The behavior suggests the unity plugin is sometimes asking for audio from webrtc's inbound audio buffer too often, or in a bursty fashion, which could cause webrtc to supply "fake" audio data and grow its jitter buffer to compensate. When inbound audio is muted webrtc has an opportunity to flush its jitter buffer to be smaller which is maybe why it sometimes shows up as low latency for a while.

We are trying to investigate. The plan is to harvest webrtc stats to make a histogram of jitter buffer state. If we learn more we will post updates.

Environment:

• OS: Ubunutu-20.04
• Browser: Chrome 92.0.4515.159 (for Peer2)
• Unity Version: 2020.3.17f1
• Package version: 2.4.0-exp.4

[karasusan]

We recognize the issue that the audio streaming latency.
It is so helpful and waiting for your additional information.

[tomires]

Hello,
I have also encountered this issue. In addition to latency, parts of the incoming stream get repeated every ~5s making the stream barely usable (outgoing stream works just fine). We have tested this against our WebRTC implementation running on a webserver, but the same issue is present in the "Audio" example. Testing was done on iPad Pro 4th gen - please see the following example (I have slightly modified InputAudioStream by changing the type to 3D and placing it away from the camera so that we don't hear everything twice)

Archive.zip

I assume it is the same issue, but if you want I can open a separate bug report.

Environment:
OS: iOS 14.2
Unity Version: 2020.3.0f1
Package version: 2.4.0-exp.4

[AndrewMeadows]

TL;DR AFAICT the latency is not caused by a large WebRTC jitter buffer. It is something else.

I instrumented my client to grab WebRTC stats. Using the documentation about the various stats I was able to compute the latency caused by the jitter buffer itself. I plotted the latency and some other stats in real-time to see how they correlate with the latency I'm experiencing. Here is a typical graph at startup:

The graph is showing four distinct values together. The plots are scaled to span the full vertical range of the graph so the maximum value of each data set is printed at the top in corresponding color. The latency (white) plot has units of "seconds" and its peak in this snapshot was almost 50msec. It would remain around 50msec even when I heard latency of a second or more. I could not find evidence that the WebRTC jitter buffer is being terribly mishandled by Unity.

However, I could make the jitter buffer grow by moving the Unity Editor window out of focus. I'm using the Fxce Window Manager on Linux and when I shift mouse/keyboard focus to another window it will deschedule the Unity Editor window, making its frames slow down. And when I do that the WebRTC jitter buffer will grow. In the plot below I moved focus away from and back to the Unity Editor a few times.

However, even though the jitter latency grows to about half a second it doesn't really change the qualitative latency I'm hearing. It is a little difficult to measure latency accurately with an untrained ear, and I'm not well calibrated, but I don't think I would be able to tell with my eyes shut whether the Unity Editor window had focus or not. Can I tell the difference between 1.0 and 1.5 seconds of latency? Probably not.

If anyone is interested in reproducing these results: the code is open and I could supply instructions on how to run it. It would require someone to make a free demo account on the HighFidelity.com and setup a test space and test Javascript Web Token (JWT).

[tomires]

Made an example video to illustrate the issue. Hopefully it is the same issue as Andrew has discussed. In addition to latency, I am getting occasional repeats and echo.

Reference: https://www.youtube.com/watch?v=NI12WvjBFG4
Output: https://user-images.githubusercontent.com/7209990/134001222-3b8182ff-8427-4948-82e3-2258c8e77607.mp4

Tested on a built-in microphone of 2017 MBP, but I am getting the same issue on Windows (Editor+runtime) and iOS.

[AndrewMeadows]

@tomires, I watched the video. It looks to me like you have an open mic problem but with very high latency. If you isolate your mic from your speakers (i.e. use headphones) I would expect the echo to go away.

[howard-stearns]

If Andrew's idea is correct, then that would mean that @AndrewMeadows and @tomires examples are likely both illustrations of one underlying problem (extremely long latency) rather than there being two different problems (latency and echo/repeats).

[tomires]

I have been using headphones when recording my example. The echo is only a minor problem - the main issue being repeats and subsequent skips, please check the following timestamps in my video:

0:12-0:15 - repeat #1
0:20-0:22 - repeat #2
0:26-0:28 - repeat #3
0:30-0:32 - skip

[karasusan]

@leejeonghun requested to review from us the fix here.
If some people are interested in this fix, please review it. thanks.
#541

[IdolMenendez]

I mentioned this bug from #538 。Here is some details.
I Build
Two Server -> signal(c#) and stun
Two Client -> pushclient and pullclient

I add two track's at one peerconnection on both client.I did received the video and audio stream and two client could do some communicate.
But the Problem is : the audio always later than the video. So the feeling is becoming weird.Usually I say something ,my voice always quicker than my voice.And it's easy to notice.
So I think ,may be you can solve this bug.and it's better to give an example of audio-video scene. I think that'll be very helpful.
Thank you so much for create this repo.
Here is some my client code.may be it can offer some infomation.

    _peerConnection = new RTCPeerConnection(ref config)
    {
        OnTrack = evnt =>
        {
            if (evnt.Track is AudioStreamTrack audio)
            {
                //otherAudioSource = audio.Source;
                audio.OnAudioReceived += (renderer) =>
                {
                    outputAudioSource.clip = renderer;
                    outputAudioSource.loop = true;
                    outputAudioSource.Play();
                };
                Debug.LogError("Poll:otherAudioSource");

            
            Debugger.LogIfLevel(LoggerType.Debug, () => $"Add Track:{evnt.Track}");
            if (!(evnt.Track is VideoStreamTrack video) ) return ; 
            if( video.IsDecoderInitialized ) return;
            otherView.texture = video.InitializeReceiver(1280, 720);
            
            evnt.Streams.First().OnRemoveTrack = ev =>
            {
                Debugger.LogIfLevel(LoggerType.Debug, () => $"Un track");
                otherView.texture = null;
                ev.Track.Dispose();
            };
            Debugger.LogIfLevel(LoggerType.Debug, () => $" video :{video.Id}");
        },
        OnIceCandidate = ice =>
        {
            _channel.Route(remoteId, new IceCandidate {IceCandidate_ = JsonUtility.ToJson(ice)});
        },
        OnConnectionStateChange = state =>
        {
            Debugger.LogIfLevel(LoggerType.Debug, () => $"State:{state.ToString()}");
        },
        OnNegotiationNeeded = () =>
        {
            Debugger.LogIfLevel(LoggerType.Debug, () => $"OnNegotiationNeeded()");
           
        },
        OnDataChannel = (channel) =>
        {
            //_dataChannel = channel;
            channel.OnMessage = (msg) =>
            {
                var any = Any.Parser.ParseFrom(msg);
                Debugger.Log(any);
            };
        }
        
    };
    
    var videoTrack = new VideoStreamTrack(webCamTexture);
    _peerConnection.AddTrack(videoTrack);
    
    _sendStream = new MediaStream();
    m_audioTrack = new AudioStreamTrack(inputAudioSource);
    _peerConnection.AddTrack(m_audioTrack, _sendStream);
    
    var initData = new RTCDataChannelInit();
    _dataChannel =_peerConnection.CreateDataChannel("Data", initData);

    _channel.OnOffer.Subscribe(d =>
    {
        Debugger.Log($"{d.Data.type} sdp {d.Data.sdp}");
        offerDes =  new RTCSessionDescription
        {
            type =  RTCSdpType.Offer,
            sdp =  d.Data.sdp
        };
        remoteId = d.FromID;
        receive .gameObject.SetActive(true);
        return false;
    });
    
    _channel.OnIceCandidate.Subscribe(d =>
    {
        _peerConnection.AddIceCandidate(d.Data);
        return false;
    });

[tomires]

@leejeonghun requested to review from us the fix here. If some people are interested in this fix, please review it. thanks. #541

Thanks for a reply! I have tested the changes made in @leejeonghun 's PR, unfortunately the performance seems to be similar to what I have described earlier.

https://user-images.githubusercontent.com/7209990/137645113-723559ac-a54a-4a95-87c8-2c096a7e490a.mp4
(reference audio track is the same as before)