Remote video is frozen after returning from background

[etown]

Frequently after extended time outside of the app, when returning to app remote video view shows a frozen frame. Audio is unaffected. This has been observed on many devices with latest SDK.

[piyushtank]

@etown Thanks for reaching out. In order to debug this problem, can you provide following information ?

1. The Room SID (TVIRoom.sid)
2. The Device and iOS version
3. Video SDK version

Also, is it possible to for you to share the debug logs with us?

Best,
Piyush

[vidhyasri2106]

I am getting the same issue often. Did you get to solve this?

[piyushtank]

@vidhyasri2106 Thanks for reaching out. Unfortunately we haven't been able to reproduce the problem yet. Is it possible for you to provide following when the problem occers -

The Room SID (TVIRoom.sid)
The Device and iOS version
Video SDK version

Best,
Piyush

[vidhyasri2106]

It is getting frozen often. Please find the details below, Room SID - RMc3454c17f2333f2556061c52356ebba6 Device – iPhone X iOS version – 12.1.4 Video SDK Version - '2.10' Also it is automatically getting unsubscribed from Video and Audio track often. Can you please help?

[vidhyasri2106]

Any update on this?

[piyushtank]

@vidhyasri2106 Thanks for the information and we are investigation the problem. We have observed an issue in our SDK's network manager module where Wifi to LTE results into one way media, we think the root cause could be the same for the issue that you are observing.

We are working on the issue and I will get back to you as soon as I have the update.

Thanks again,
Piyush

[vidhyasri2106]

Thanks. I will wait for your update.

[piyushtank]

Hello,

We have found a bug in our infrastructure (media server). It is reproducible in following network handoff scenario:

1. A Group, or Small Group Rooms is created Ad-Hoc or via the REST API.
2. Alice connects to a Room with Wifi and LTE on.
3. Bob connects to the Room.
4. Alice switches off Wifi

After network handoff, Bob can see and hear Alice, but Alice's can not see or hear Bob. Alice observes frozen screen on her phone.

The fix for this problem has been tested with existing SDKs and it is going to be deployed to our media server in the next few days. We believe the media server fix should resolve the issue reported by @vidhyasri2106 and @etown on this ticket, and do not expect that any SDK updates will be required.

I am sorry for the inconvenience caused by this bug. I will post the update on this ticket once the fix is deployed.

Best,
Piyush

[piyushtank]

@vidhyasri2106 This ticket is specific to network handover and backgrounding scenario. I have moved your comments from here to this ticket to avoid confusion for readers of this ticket. Lets use this ticket for the ARKit freeze issue you are facing without backgrounding and without network handover.

[piyushtank]

Hello,

Our media server team has deployed the fix for this ticket. The discussed Wifi+LTE -> LTE network handoff scenario should work as expected now.

Try it out, and Let us know if you have any questions.

Best,
Piyush

[NaiyerAghaz]

Yes @piyushtank I am also facing this issue on remote view, how I will fix this?

[Horatiu]

@piyushtank I am also facing this issue with latest SDK. What is the solution for this? re-enabling the video track when app becomes active as opposed to app entering foreground?

[piyushtank]

@NaiyerAghaz Apologies for the delay. Can you provide the detailed scenario and the RoomSID when you observed the issue?

@Horatiu Can you provide the detailed scenario and the RoomSID when you observed the issue?

[Horatiu]

@piyushtank I have fixed it by enabling the video track after applicationDidBecomeActive notification fires as opposed to applicationWillMoveToForeground notification. The way I was able to reproduce was this:

1. In applicationDidEnterBackground notification handler I would set videoTrack?.isEnabled = false and this would disable the video
2. In applicationWillEnterForeground notification handler I would set videoTrack?.isEnabled = true

I would notice that the videoTrack would freeze even though at the other end I got the didEnableVideoTrack callback. The reason I need to enable and disable when user moves between app states is because I want a specific UI at the remote participant end and I would implement didEnableVideoTrack/didDisableVideoTrack there to achieve the wanted UX.

To fix it I removed enabling the video track in applicationWillEnterForeground and instead I am enabling it in applicationDidBecomeActive notification handler and that seems to work.

This issue is with the iOS SDK.

[piyushtank]

@Horatiu You should be able to enable/disable a video track while going to foreground/background. We would like to fix the issue in SDK but when I tried to reproduce, I couldn't. So, I have created a branch on our VideoQuickstart sample app to reproduce the bug. Can you try it out and let me know.

[Horatiu]

@piyushtank actually the above described 'fix' did not work. If I background one or both participants for 2-3 minutes when I foreground either of them the local track will be 'freezed' even though callbacks that video track was enabled would be received at the other end. I'll try to repro on your branch a little later today.

[Horatiu]

@piyushtank Upon further investigation I am finding that when moving from background to foreground I get cameraSourceDidFail with error message from AVFoundation: "The operation could not be completed" and that is when the call is placed in this state.

Trying now to recover the call by rebuilding the cameraSource and localVideoTrack from scratch and republishing it when this happens. Will let you know if I manage to fix it that way.

[raylillywhite]

@piyushtank I'm also experiencing the issue that @etown mentioned. I've filed a Twilio support ticket #4302109 with a call SID, debug logs and a video of it slowly recovering. This seems different than the issue Horatiu is mentioning, since the frozen video is the remote one for me and etown. We can reproduce this every time we background the app for 5+ minutes (but backgrounding the app briefly is fine). After foregrounding, the remote video feed will be frozen for a minute or two before rendering a few frames and freezing again. And each time it freezes, it freezes for a slightly shorter duration until it fully recovers around 5 minutes.

[piyushtank]

@raylillywhite Thanks. Followed up on the twilio support ticket.

[halldorlogi]

Any news on a fix here ?

[piyushtank]

@halldorlogi We haven't been able to reproduce the problem and our team is busy upgrading the webrtc to latest release. If you have room sid and steps to reproduce, we can take a look. Also reproduction with quickstart will be helpful.

[piyushtank]

Closing the ticket due to no activity. Feel free to reopen if you run into issue or open a new ticket.