Froze video capturing during Video call

[ketanEcreate]

Froze video capturing, during video call put call in background to foreground so, do not start video streaming.

i share my logs

2019-12-17 19:01:17.685790+0530 VideoCallKitQuickStart[1270:1088816] DEBUG:TwilioVideo:[Platform](Thread 0x0x157e789b0): Application will enter foreground.
2019-12-17 19:01:17.685954+0530 VideoCallKitQuickStart[1270:1088816] INFO:TwilioVideo:[Platform](Thread 0x0x157e789b0): Manually restarting interrupted session.
2019-12-17 19:01:17.755665+0530 VideoCallKitQuickStart[1270:1088816] DEBUG:TwilioVideo:[Platform](Thread 0x0x157e789b0): Application will enter foreground.
2019-12-17 19:01:17.755883+0530 VideoCallKitQuickStart[1270:1088816] INFO:TwilioVideo:[Platform](Thread 0x0x157e789b0): Manually restarting interrupted session.
2019-12-17 19:01:17.756087+0530 VideoCallKitQuickStart[1270:1089276] INFO:TwilioVideo:Platform: Session interrupted with reason: 1
2019-12-17 19:01:17.756172+0530 VideoCallKitQuickStart[1270:1089276] INFO:TwilioVideo:Platform: ... Capture pipeline did stop.
2019-12-17 19:01:17.756240+0530 VideoCallKitQuickStart[1270:1089276] DEBUG:TwilioVideo:Platform: Teardown video output pipeline ...
2019-12-17 19:01:17.756303+0530 VideoCallKitQuickStart[1270:1089276] DEBUG:TwilioVideo:Platform: Finished video output pipeline teardown.
2019-12-17 19:01:17.756365+0530 VideoCallKitQuickStart[1270:1089276] DEBUG:TwilioVideo:Platform: Video pipeline did finish running
2019-12-17 19:01:17.764960+0530 VideoCallKitQuickStart[1270:1089276] DEBUG:TwilioVideo:Platform: No background task to stop.
2019-12-17 19:01:17.765275+0530 VideoCallKitQuickStart[1270:1089276] INFO:TwilioVideo:Platform: Session stopped running
2019-12-17 19:01:18.460713+0530 VideoCallKitQuickStart[1270:1089276] DEBUG:TwilioVideo:Platform: Session encountered a runtime error: Error Domain=AVFoundationErrorDomain Code=-11800 "The operation could not be completed" UserInfo={NSLocalizedFailureReason=An unknown error occurred (-12780), NSLocalizedDescription=The operation could not be completed, NSUnderlyingError=0x1599c2170 {Error Domain=NSOSStatusErrorDomain Code=-12780 "(null)"}}
2019-12-17 19:01:18.460873+0530 VideoCallKitQuickStart[1270:1089276] INFO:TwilioVideo:Platform: ... Capture pipeline did stop.
2019-12-17 19:01:18.460955+0530 VideoCallKitQuickStart[1270:1089276] DEBUG:TwilioVideo:Platform: Teardown video output pipeline ...
2019-12-17 19:01:18.461032+0530 VideoCallKitQuickStart[1270:1089276] DEBUG:TwilioVideo:Platform: Finished video output pipeline teardown.
2019-12-17 19:01:18.461105+0530 VideoCallKitQuickStart[1270:1089276] DEBUG:TwilioVideo:Platform: Video pi2019-12-17 19:01:18.461731+0530 VideoCallKitQuickStart[1270:1088816] DEBUG:TwilioVideo:[Platform](Thread 0x0x157e789b0): Application will enter foreground.
peline did finish running
2019-12-17 19:01:18.462691+0530 VideoCallKitQuickStart[1270:1089276] DEBUG:TwilioVideo:Platform: No background task to stop.
2019-12-17 19:01:18.462883+0530 VideoCallKitQuickStart[1270:1089276] ERROR:TwilioVideo:Platform: Fatal runtime capture error: Error Domain=AVFoundationErrorDomain Code=-11800 "The operation could not be completed" UserInfo={NSLocalizedFailureReason=An unknown error occurred (-12780), NSLocalizedDescription=The operation could not be completed, NSUnderlyingError=0x1599c2170 {Error Domain=NSOSStatusErrorDomain Code=-12780 "(null)"}}, code -11800
2019-12-17 19:01:18.463373+0530 VideoCallKitQuickStart[1270:1089276] DEBUG:TwilioVideo:Platform: Session encountered a runtime error: Error Domain=AVFoundationErrorDomain Code=-11800 "The operation could not be completed" UserInfo={NSLocalizedFailureReason=An unknown error occurred (-12780), NSLocalizedDescription=The operation could not be completed, NSUnderlyingError=0x159a2d460 {Error Domain=NSOSStatusErrorDomain Code=-12780 "(null)"}}
2019-12-17 19:01:18.467970+0530 VideoCallKitQuickStart[1270:1089276] INFO:TwilioVideo:Platform: ... Capture pipeline did stop.
2019-12-17 19:01:18.468227+0530 VideoCallKitQuickStart[1270:1089276] ERROR:TwilioVideo:Platform: Fatal runtime capture error: Error Domain=AVFoundationErrorDomain Code=-11800 "The operation could not be completed" UserInfo={NSLocalizedFailureReason=An unknown error occurred (-12780), NSLocalizedDescription=The operation could not be completed, NSUnderlyingError=0x159a2d460 {Error Domain=NSOSStatusErrorDomain Code=-12780 "(null)"}}, code -11800
2019-12-17 19:01:18.480047+0530 VideoCallKitQuickStart[1270:1088816] DEBUG:TwilioVideo:[Platform](Thread 0x0x157e789b0): No manual restart of the session required.
2019-12-17 19:01:18.4619182019-12-17 19:01:18.502820+0530 VideoCallKitQuickStart[1270:1088816] Camera source failed with error: The operation could not be completed
2019-12-17 19:01:18.584974+0530 VideoCallKitQuickStart[1270:1088816] Camera source failed with error: The operation could not be completed
+0530 VideoCallKitQuickStart[1270:1090919] DEBUG:TwilioVideo:Platform: Session encountered a runtime error: Error Domain=AVFoundationErrorDomain Code=-11800 "The operation could not be completed" UserInfo={NSLocalizedFailureReason=An unknown error occurred (-12780), NSLocalizedDescription=The operation could not be completed, NSUnderlyingError=0x159ad3810 {Error Domain=NSOSStatusErrorDomain Code=-12780 "(null)"}}
2019-12-17 19:01:18.588548+0530 VideoCallKitQuickStart[1270:1088816] DEBUG:TwilioVideo:[Platform](Thread 0x0x157e789b0): Default audio device received UIApplicationDidBecomeActiveNotification.
2019-12-17 19:01:18.594590+0530 VideoCallKitQuickStart[1270:1090919] INFO:TwilioVideo:Platform: ... Capture pipeline did stop.
2019-12-17 19:01:18.615303+0530 VideoCallKitQuickStart[1270:1090919] DEBUG:TwilioVideo:Platform: Teardown video output pipeline ...
2019-12-17 19:01:18.615513+0530 VideoCallKitQuickStart[1270:1090919] DEBUG:TwilioVideo:Platform: Finished video output pipeline teardown.
2019-12-17 19:01:18.615585+0530 VideoCallKitQuickStart[1270:1090919] DEBUG:TwilioVideo:Platform: Video pipeline did finish running
2019-12-17 19:01:18.615649+0530 VideoCallKitQuickStart[1270:1090919] DEBUG:TwilioVideo:Platform: No background task to stop.
2019-12-17 19:01:18.615909+0530 VideoCallKitQuickStart[1270:1090919] ERROR:TwilioVideo:Platform: Fatal runtime capture error: Error Domain=AVFoundationErrorDomain Code=-11800 "The operation could not be completed" UserInfo={NSLocalizedFailureReason=An unknown error occurred (-12780), NSLocalizedDescription=The operation could not be completed, NSUnderlyingError=0x159ad3810 {Error Domain=NSOSStatusErrorDomain Code=-12780 "(null)"}}, code -11800
2019-12-17 19:01:18.620740+0530 VideoCallKitQuickStart[1270:1090919] DEBUG:TwilioVideo:Platform: Session encountered a runtime error: Error Domain=AVFoundationErrorDomain Code=-11800 "The operation could not be completed" UserInfo={NSLocalizedFailureReason=An unknown error occurred (-12780), NSLocalizedDescription=The operation could not be completed, NSUnderlyingError=0x159a90960 {Error Domain=NSOSStatusErrorDomain Code=-12780 "(null)"}}
2019-12-17 19:01:18.620938+0530 VideoCallKitQuickStart[1270:1090919] INFO:TwilioVideo:Platform: ... Capture pipeline did stop.
2019-12-17 19:01:18.621087+0530 VideoCallKitQuickStart[1270:1090919] ERROR:TwilioVideo:Platform: Fatal runtime capture error: Error Domain=AVFoundationErrorDomain Code=-11800 "The operation could not be completed" UserInfo={NSLocalizedFailureReason=An unknown error occurred (-12780), NSLocalizedDescription=The operation could not be completed, NSUnderlyingError=0x159a90960 {Error Domain=NSOSStatusErrorDomain Code=-12780 "(null)"}}, code -11800
2019-12-17 19:01:18.622238+0530 VideoCallKitQuickStart[1270:1090919] INFO:TwilioVideo:Platform: Session stopped running
2019-12-17 19:01:18.573673+0530 VideoCallKitQuickStart[1270:1090819] DEBUG:TwilioVideo:Core: Received message:
{"type":"heartbeat"}
2019-12-17 19:01:18.631019+0530 VideoCallKitQuickStart[1270:1090819] DEBUG:TwilioVideo:Core: Sending message (21 bytes):
{"type":"heartbeat"}
2019-12-17 19:01:18.704659+0530 VideoCallKitQuickStart[1270:1091048] INFO:TwilioVideo:Platform: Session interruption ended
2019-12-17 19:01:18.704818+0530 VideoCallKitQuickStart[1270:1091048] INFO:TwilioVideo:Platform: Session started running
2019-12-17 19:01:18.707640+0530 VideoCallKitQuickStart[1270:1091048] DEBUG:TwilioVideo:Platform: Video pipeline will start running.
2019-12-17 19:01:18.709340+0530 VideoCallKitQuickStart[1270:1091048] DEBUG:TwilioVideo:Platform: Video device will start running: <AVCaptureFigVideoDevice: 0x157debe10 [Front Camera][com.apple.avfoundation.avcapturedevice.built-in_video:1]>.
2019-12-17 19:01:23.041784+0530 VideoCallKitQuickStart[1270:1090819] DEBUG:TwilioVideo:Core: Received message:
{"type":"heartbeat"}
2019-12-17 19:01:23.442736+0530 VideoCallKitQuickStart[1270:1090819] DEBUG:TwilioVideo:Core: Sending message (21 bytes):
{"type":"heartbeat"}
2019-12-17 19:01:27.614268+0530 VideoCallKitQuickStart[1270:1090819] DEBUG:TwilioVideo:Core: Received message:

Video iOS SDK
3.0.1 via CocoaPods

Xcode
10.1

iOS Device
iPhone 5s, iPhone X

[ketanEcreate]

Hello @ceaglest, @piyushtank, @paynerc
Can you please check this issue?

[andschdk]

I am also experiencing issues when backgrounding and returning to foreground.

[andschdk]

Any news on this one @piyushtank ?

[andschdk]

This is printed in the console when it happens:

ERROR:TwilioVideo:[Platform](0x16d0c7000): Fatal runtime capture error: Error Domain=AVFoundationErrorDomain Code=-11800 "The operation could not be completed" UserInfo={NSLocalizedFailureReason=An unknown error occurred (-12780), NSLocalizedDescription=The operation could not be completed, NSUnderlyingError=0x2812990b0 {Error Domain=NSOSStatusErrorDomain Code=-12780 "(null)"}}, code -11800

[andschdk]

I did some more investigation here.

One is not able to reproduce this in the sample project because the local preview (VideoView) is always present.

However most apps will create this view when needed (i.e. starting a call), and remove it again when done (ending a call). This process can be repeated.

I did a few modifications to the sample project (VideoQuickStart) in order to reproduce the bug we experience in our app.
The changes allows one to add/remove the VideoView mimicking the use case described above.
Usually this works fine:

1. Start camera
2. Background app
3. Foreground app
4. Video is not frozen

However second time usually fails:

5. Disconnect (also removes VideoView)
6. Start camera
7. Background app
8. Foreground app
9. Video is frozen (if not try repeat 5->10 again)

Note: You don't have to connect to any room to reproduce this bug.

Link to modded sample project: andschdk@df6941c

video	-
	<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<

[ceaglest]

Hey @andschdk,

Thanks for investigating the frozen video. I looked at the code you provided and it seems like the stopPreview method is just stopping rendering but not the TVICameraSource. Calling this method, and then startPreview is going to cause a new TVICameraSource to be created without stopping the old one.

Somewhat recently, we updated the example code to stop TVICameraSource in deinit. What if the stopPreview() method that you wrote also performed this step? I think this would resolve the issue.

https://github.com/twilio/video-quickstart-ios/blob/master/VideoQuickStart/ViewController.swift#L44

Best,
Chris

[andschdk]

Hi @ceaglest
Thank you so much for your reply.

This resolved the issue 👍
Any reason why this is not happening automatically when CameraSource is deinitialized?

[ceaglest]

Great to hear @andschdk!

TVICameraSource has a strong reference internally that is broken by calling stopCapture. When we adopted VideoSource in WebRTC this seemed like the safest memory model at the time, but the API has caused some confusion. We are looking to make improvements in this area.

[amritpalsingh88]

Hi @ceaglest , I am also experiencing the same issue. Video freezes when the application is brought to the foreground from the background. This happens occasionally. Most of the times it works fine. The solution that you have suggested above seems to be working if the local video is stopped manually on the tap of the button. But why would this solution work if the app is sent to background and again to foreground?

[ceaglest]

Hi @amritpalsingh88,

But why would this solution work if the app is sent to background and again to foreground?

The problem can occur if you've leaked a TVICameraSource by destroying it without calling stopCapture(). After that point when you background you have 1 or more interrupted AVCaptureSessions. The behavior of which AVCaptureSession is going to come back may not be deterministic when there is more than 1, so you might end up with the leaked AVCaptureSession getting resumed instead of the one you really want.

Best,
Chris

[paynerc]

All,

Today we released Twilio Video iOS 3.2.5 which ensures that as TVICameraSources are deallocated the associated AVCaptureSession is stopped and the strong reference that @ceaglest mentioned is removed. When this automatic cleanup is being performed during the dealloc of TVICameraSource you will see Error level messages in your log files before and after the underlying AVCaptureSession is stopped and cleaned up. This serves as a reminder that while we did add this cleanup mechanism in TVICameraSource, it is preferable that you call either stopCaptureWithCompletion: or stopCapture when you are done capturing from the camera and at a time that is best in your application's lifecycle.

Ryan