[iOS 13.0, 13.1-beta] LocalVideoTrack content might be distorted after videoDeviceInUseByAnotherClient interruption

[kselvin]

Description

Note : this only happens in iOS 13, it does not when running the same exact build in iOS 12

Local Video Track becomes distorted after being interrupted and then uninterrupted specifically with the reason videoDeviceInUseByAnotherClient

(Note: this will not occur from interruption by putting the app in background and then foreground)

[Description of the issue]

Steps to Reproduce

1. Connect two users in a group room (make sure you are publishing the local video track to both users)
2. Use the camera in-app for something else -- in my case the user opens up to a separate camera manager to take a video of just yourself, which is why the Twilio video track gets interrupted.
3. Stop the newly created capture session --> Twilio will automatically uninterrupt the connection and start publishing the localVideoTrack again

The localVideoTrack is distorted (looks squished)

Expected Behavior

localVideoTrack is not distorted

Reproduces How Often

Every time on iOS 13

Video iOS SDK

2.9 via CocoaPods

Xcode

10.3 and Xcode 11 Beta (tried downloading from app store build that was created in 10.3, then tried also running the same code in Xcode 11 Beta)

iOS Version

13.0, 13.1 Beta

iOS Device

iPhone 7+, iPhone 7

[ceaglest]

Hi @kselvin,

Thanks for the report! I'll try to reproduce the videoDeviceInUseByAnotherClient interruption case locally. I tried backgrounding interruptions on iOS 13.0-beta8 and didn't notice distortions in that case.

The localVideoTrack is distorted (looks squished)

Are you using TVIVideoView as a TVIVideoRenderer? In the meantime, one possible solution would be to use TVICameraPreviewView to render your local camera feed instead. There is an example here.

Best,
Chris

[kselvin]

I believe I am using the TVIVideoView as the TVIVideoRenderer

if let myView = TVIVideoView.init(frame: CGRect.zero, delegate:nil) {
              
               myView.contentMode = .scaleAspectFill
               localVideoTrack?.addRenderer(myView)
               myView.shouldMirror = true
               myView.frame = readerBannerView.auditionerVideoView.bounds
}

The distortion is actually being seen on both ends -- the remote user is seeing me distorted on his//her screen as well

Will using TVICameraPreviewView solve the issue in this case?

[kselvin]

An interesting finding

If I interrupt the camera another way (like backgrounding), the localVideoTrack renders properly on uninteruption and then it ALWAYS renders properly no matter how i interrupt it (including using the camera for something else, which is the source of the original problem)

[ceaglest]

Hello,

Thank you for the detailed information! I'll be sharing more on potential solutions as soon as I can.

Thanks,
Chris

[kselvin]

Thank you! Much appreciated

I have tried a lot of ideas (aside from purposely disconnecting and reconnecting which would really hurt the user experience given how many times it would happen throughout a session) -- but unfortunately no luck yet

I will let you know if if i find anything new

[ceaglest]

Hi @kselvin,

10.3 and Xcode 11 Beta (tried downloading from app store build that was created in 10.3, then tried also running the same code in Xcode 11 Beta)

iOS Version
13.0, 13.1 Beta

iOS Device
iPhone 7+, iPhone 7

I tried to reproduce this issue using an iPhone 7 Plus with the iOS 13.0 GM seed, Xcode 11.0 and Twilio Video 2.10.1. I used two approaches:

1. Interrupt TVICameraSource by creating and starting another TVICameraSource.
2. Interrupt TVICameraSource by creating and starting an AVCaptureSession with a video data output.

Here is the modified Quickstart example. Tap the local camera preview to start an interruption.

2019-09-11 13:47:43.691618-0700 VideoQuickStart[1377:33717] Camera source was interrupted with reason 3.
2019-09-11 13:47:45.972392-0700 VideoQuickStart[1377:33717] Camera source was interruption ended.
2019-09-11 13:47:55.682147-0700 VideoQuickStart[1377:33717] Camera source was interrupted with reason 3.
2019-09-11 13:47:58.135718-0700 VideoQuickStart[1377:33717] Camera source was interruption ended.

Both code paths result in the interruption starting. After a few seconds I stop the new Source/Session, the interruption ends, and video looks as expected for local and remote parties. Can you please take a look at the code snippet and suggest how I can modify it to reproduce the issue?

Thank you,
Chris

[ceaglest]

Hi,

One more thought, would it be possible to provide debug level logs from your app from before the interruption begins, to after the interruption ends and the video distortion has begun?

Best,
Chris

[ceaglest]

Hey @kselvin,

No need for logs, I can reproduce the problem now and I have pushed another commit to the test branch. The piece of code that is important is how the VideoDataOutput in the interrupting session is configured:

let vdo = AVCaptureVideoDataOutput()
session.addOutput(vdo)

// Causes distortion after the interruption ends.
if #available(iOS 13.0, *) {
    vdo.automaticallyConfiguresOutputBufferDimensions = false
    vdo.deliversPreviewSizedOutputBuffers = false
}

I am investigating how to properly configure the AVCaptureVideoDataOutput used by TVICameraCapturer and TVICameraSource to handle this interruption scenario in iOS 13. Thanks again for bringing this issue to our attention.

Best,
Chris

[kselvin]

Awesome, thank you! Taking a look at the test branch now

[kselvin]

Sorry I didn't think to give this detail sooner but when it gets interrupted it is because i am switching to use this Camera Manager library https://github.com/imaginary-cloud/CameraManager

Which is actually using AVCaptureMovieFileOutput

I do not have the deepest understanding of Camera features but I don't see the type AVCaptureVideoDataOutput anywhere in the CameraManager.swift file (but I think it inherits from AVCaptureVideoDataOutput)

[ceaglest]

Hi @kselvin,

Thanks for the extra info about your setup.

Which is actually using AVCaptureMovieFileOutput

Good to know. It seems like the issue is arising because of the preview buffer / full size buffer requirements of our VideoDataOutput and an interaction with other AVCaptureSessions that have different requirements. There are new VDO properties in iOS 13 that we haven't adopted yet, and setting them appropriately might resolve the problem.

Best,
Chris

[kselvin]

Thanks Chris!

Please let me know if there is anything else I can do/look into to help and please keep me posted!

[ceaglest]

Hi @kselvin,

I am trying out a fix which resets the AVCaptureDevice's active format when the interruption ends. This seems to resolve the image distortion that occurs after specific combos of AVCaptureSessions interrupt each other in iOS 13.

You can expect a fix for this bug to be part of 2.10.2 and 3.0.0-beta5. I will post again once testing with internal stakeholders completes and these updates are closer to release.

Best,
Chris

[kselvin]

Amazing, thank you!!

[kselvin]

Hi @ceaglest,

Any update on this?

Would love to update Twilio and submit new version of my app ASAP

Thanks!

[ceaglest]

Hi @kselvin,

We have released a fix in 3.0.0-beta5, which you are welcome to try. I am also back porting the fix for release in 2.10.2, which we expect next week.

I apologize for the delay in getting a fix for 2.x, as I know you originally reported the problem with 2.9.

Best,
Chris

[kselvin]

Awesome thank you! Will check it out.

What are the reasons it's still considered a beta and not just 3.0? Would it be smarter to wait until 2.10?

[ceaglest]

Hello,

Thanks for asking. We are nearly ready to GA the 3.0 release, however we are making one more API change related to TVICameraSource in order to better support UIWindowScene on iOS 13. We will be releasing a 3.0.0-beta6 next week, and expect to promote that beta release to 3.0.0.

Have a good weekend,
Chris

[kselvin]

Sounds great, thanks for the update

[kselvin]

I was able to get the 3.0.0beta working and it fixed the issue :), but we have decided for now it is still better for us to wait for the back port fix on 2.10.0

[ceaglest]

Thanks for confirming the fix with the 3.0 beta! We are in the process of back porting it to 2.10.2.

[kselvin]

Hey,

Any update on this? Unfortunately enough of our users are starting to update to iOS 13 that some have noticed the issue.

[kselvin]

Decided to migrate to 3.0.0 now that it's not in beta anymore anyway 😊

[paynerc]

This fix for this issue has been back ported to 2.10.2 which was released today.

Please don't hesitate to reach out if you have any further issues!

Thank you,

Ryan