[BUG] Unable to resize canvas without framerate drop, using VideoStreamTrack and ReplaceTrack

[yvanzine]

@karasusan asked me to create an issue for this in github, so I am filing a proper bug. All the up-to-date description is in this entry on the forum:

https://forum.unity.com/threads/resize-videostreamtrack-and-replacetrack-framerate-drops.1145414/

I will be retesting this functionality to bring myself up to date with how it is currently working.

I see this as a way forward to dynamically resize canvas via webrtc Replace Track API.

But we are unable to call replace track without a framerate drop to 1fps, so it is unusable. I will also try to create a small patch to demonstrate this functionality once I get it running.

[kannan-xiao4]

memo: WRS-227

[karasusan]

@yvanzine
I think this issue should fix on the WebRTC package, not Render Streaming package.

[yvanzine]

@karasusan you are right, the bug to fix is in the webrtc package, however I implemented the calls to the existing ReplaceTrack api in the RenderStreaming package. I have previously posted some of the code details on that in the forum link.

[karasusan]

@yvanzine
I am checking this issue here. I found the crash bug when using NvDecoder. We are investigating it.
One question, could you tell me what codec you are using?

[yvanzine]

@karasusan

I am using h264 codec 6.2 profile.

I want to mention that I am also running 1920x1080 resolution. I have had crashes with earlier codecs at this resolution. It is I believe necessary to cap framerate to 60fps, to avoid codec crashes. Unity webrtc codec code is bounded high, at 240 fps and it overflows buffer size at that framerate. The 6.2 codec is documented to handle it at that framerate but it crashes for me, so I limited it to 60 and then, no crashes.

[yvanzine]

@karasusan

The replace track functionality must handle arbitrary size buffer. Power of 64 dimensions is not sufficient for most use cases.

[karasusan]

@yvanzine
One question, what version of WebRTC package are you using? I am sure that the current implementation can not choice the profile level 6.2.

I implemented the feature of controlling framerate here. It have not released yet. I checked the feature in the Latency sample, and I checked that the condition(H264, 1920x1080, 60FPS) works fine. As a side note, it doesn't work with 90FPS because of the profile level limitation.

The replace track functionality must handle arbitrary size buffer. Power of 64 dimensions is not sufficient for most use cases.

OK, I will check it with other resolutions. Thanks.

[karasusan]

@yvanzine
#737 is merged into the develop branch. Please try the branch and let me give your feedback.

[kannan-xiao4]

memo: URS-449