Big Input Interruptions Analyzed

[WaltzBinaire]

We have found a bigger unnecessary bug that is driving us crazy since weeks. About the way the input communication from the Webserver to Unity works.

You can test this bug on WebRTC 2.0.0 vs WebRTC 2.0.1 - 2.0.5. There must be a signaling change that was introduced in 2.0.1 - which destroys the communication.

To Reproduce
Start Demo - add a rotating cube - play - rotate your camera for a while. You create continuous signals lets say via dragging - while looking at the rotating cube.

After a minute you will see an input lag/ stuttering - while you will see the cube continuing to rotate. This means that the image transfer is perfectly fine - but the signals are

• discarded
• queued the wrong way
• not communicated at all periodically

I believe there must be somewhere a one liner of code - that was introduced in 2.0.1 - that causes this problem - that many people on the forum refer to as "image lack" while it actually is an input bug - totally ruins the experience - and could be fixed real quick. Please help out here

Environment (please complete the following information):
Windows
chomre
Unity Version 2019.3.9
Package version 2.0.1-preview

[karasusan]

Do you mean that the input lag caused by the changes of signaling feature?
I think it is difficult to consider a causal relationship between the two.

No major changes between 2.0.0-preview and 2.0.1-preview, so it should be reproduced with the 2.0.0-preview.

[WaltzBinaire]

Hey! Thanks for getting back. Yes we believe it must be somehow signaling - we can reproduce this by coming from 200 (everything fine) to 201 (shows effect after 1 minute)

[eirikwah]

EDIT: I did not have this same bug. I have registered a new bug instead, as #180

We are seeing the same behaviour. I had not reported it yet because I though the error was in our own code, and I wanted to do some more tests.

But to be clear: @WaltzBinaire: Am I correct in assuming that you mean that the bug is in the receiving and/or parsing of the inputChannel data? Note that the word "signaling" in WebRTC-context is mostly used about the setup of the WebRTC communication (through a "signaling server"), and not for the sending of control signals over the WebRTC data channel(s) when the WebRTC connection has been set up.

[WaltzBinaire]

hey @eirikwah - absolutly - thanks for clarifying my sloppy expression which can lead to confusion. It has nothing to do with signaling server - it is related to incoming inputchannel data - that has a lacky behavior introduced between 200 and 201

[thomasvanta]

Hi @karasusan !
I'm seeing the same issue.
I've been able to get rid of it in editor:

• if I have an InputDeviceDebuggerWindow open through the Input Debug.
  If the window is executing 'onGUI', then the problem disappears, in the moment I close or put the window in the background, the issue comes back.
  So I'm guessing that the event tracing that happens in 'initializaWith()' inside 'OnGUI()' is somehow able to manage or flush the queue of state events on the unmanaged C++ side.

[karasusan]

Thanks for your cooperation.

I guess the issue caused by Input System. I am currently investigating the issue.

[karasusan]

I am checking the issue with the latest version of package (2.1.0-preview.1) but not reproduced.
Unity 2019.4.7f1 on windows and google chrome. I operated keyboard to move camera during 5 minutes.
Could you tell me how to reproduce it?

[thomasvanta]

Hi Karasusan, you mean the dev branch on github? the latest in package manager is 2.0.5 preview for webrtc and 2.0.2 preview for unity render streaming.

My setup is 2019.3 and 2.0.2 of render streaming.

I'd love to test 2.1 if an official release is made.

[karasusan]

Thanks @thomasvanta,
I am preparing to release the QA version 2.1.0-preview.2.

Please try it if you can.

[thomasvanta]

I was checking the HDRP sample and couldn't replicate there either.
It seems to happen with hardware encoding and Complex scenes with highly detailed environments.
Let me know what I should look into specifically while testing 2.10.-preview.2 when is out.
Thanks

[karasusan]

The latest version of the template project is here
https://github.com/Unity-Technologies/UnityRenderStreaming/releases/tag/2.1.0-preview

[thomasvanta]

I'm testing and it looks much better now.
The bitrate ramps up now from around 20 MB to 90 in ~30s

I see that in CameraStreamer.cs the bitrate parameter is not working.
In RenderStreaming, the string replace for max bitrate doesn't seem to have any effect either.
should I replace in line 177 pc.AddTrack(track) for something like:

var sender = pc.AddTrack(track);  
var parameters = sender.GetParameters();  
foreach(var encoding in parameters.GetEncodings )  
{  
encoding.maxBitrate = //my bitrate here  
}

Or where would be the right place to plug it?
Thanks

[karasusan]

@thomasvanta
I am testing the RTCRtpSender.SetParameters API and I guess the API behavior is weird as you said.

[eirikwah]

My previous message stating that I also experienced this bug was wrong. Sorry for my confusion. I have registered a new bug instead, as #180

[WaltzBinaire]

@karasusan we see that the quality of the video got much better and it seems as if it was connected to the interruptions. Which is really great as it pushed the quality by a huge margin. However we also stumble across the issues mentioned by @thomasvanta -

1. the bitrate doesnt drop now - but it ramps up slowly in the beginning for about 25 seconds - it seems as if the bitrate is slowly interpolating from 0 - to the actual value. Should we open a new bug for this one?

[karasusan]

@WaltzBinaire

Should we open a new bug for this one?

Sure, although we are investigating this issue already,
it would be glad for me that you make a new one for the bug.

[WaltzBinaire]

@karasusan very cool - i will open a new bug with description from our side - furthermore i would consider the input lag to be resolved as we were not able to create this again with the new 2.1.0 preview.4 - SO SOLVED - THANK YOU

[karasusan]

I reopened the issue because of the comment.
#182 (comment)

[WaltzBinaire]

@karasusan where you able to recreate the issue?

[karasusan]

@WaltzBinaire
I will try to reproduce the issue this week.

[WaltzBinaire]

@karasusan thank you!

[WaltzBinaire]

cant reproduce this issue anymore with latest adjustments here #191 - Thank you @karasusan