Distorted attached view to TVILocalVideoTrack

[alexiss03]

Hi!

I have TVIVideoView attached to a TVILocalVideoTrack. This TVIVideoView is a subview of a parent UIView. This TVIVideoView is horizontally or vertically flipped and the contentMode is set to scaleToFill after doing the following steps.

STEPS TO REPLICATE:

1. Set the contentMode of the parent UIView to scaleAspectFill.
2. Set the TVILocalVideoTrack isEnabled property to false.
3. Set the isHIdden property of the parent UIView to true.
4. Apply a transform of CGAffineTransform.init(scaleX: 0, y: 0) to the parent UIView.
5. Set the TVILocalVideoTrack isEnabled property to true.
6. Apply a transform of CGAffineTransform.init(scaleX: 1, y: 1) to the parent UIView.
7. Set the isHidden property of the parent UIView to false.

Hoping you can help me this. Thanks you!

Best,
Lexi

[paynerc]

Hi @alexiss03!

I'm working on reproducing this issue and have a couple questions for you.

1. Can you give me a little more information on what you mean by the distorted view?
2. Do you have mirroring on on the TVIVideoView?
3. Are you performing these steps all at once in response to a UI action or something?

Thanks!

Ryan

[alexiss03]

Hi Ryan!

Thanks for the timely response.

1. The initial 'TVIVideoView' that is set to 'scaleAspectFill' before performing the said steps above results to either horizontally or vertically flipped and the 'contentMode' is set to 'scaleToFill' without changing it in the process.
2. No.
3. I am performing this in response to a UI action.

I've created a fork of your repository for this issue, you may view it here in where I've edited the VideoQuickStart project. You should tap the Disable button more than twice to replicate the issue. This issue is intermittent but is more likely to be replicated after a few tries.

Best,
Lexi

[alexiss03]

Hi Ryan,

Just checking if you were able to replicate this issue. Thanks!

Best,
Lexi

[piyushtank]

Hi Lexi,

I performed following tests on your branch but could not reproduce the problem.

1. Tapped Disable button more than twice when the device is not connected to a Room.
2. Tapped Disable button more than twice when the device is connected to a Room.
   I ran these tests on iPhone 6 plus and iPad Air 2 multiple times but the problem did not reproduce.

Is it possible for you to provide the screenshot or video when problem reproduces? Also, can you send the device and iOS information where you are running the tests?

Thanks,
Piyush

[alexiss03]

Hi Piyush,

Thanks for the reply. I attached here a video of the issue. I put the video file inside the zipped file to be able to upload here in Github. Another thing, I forgot to tell you before that this issue is easily replicated on iPhone 5 devices while difficult on newer ones such as iPhone 6. The device used in the video is iPhone 5 running on iOS 10.2.1. After tapping the Disable button, you can see that the initial local video track preview was flipped -90 degrees and the ratio of the video was also changed. The code run is the same as the one I referenced in the previous comment.

iPhone 5 10.2.1 TwilioVideo Distorted Local Video Track.zip

Best,
Lexi

[piyushtank]

Thanks @alexiss03 Yes, I am able to reproduce the problem on your branch now.

The problem you have found in Video SDK is a frame rotation bug in our OpenGL ES Renderer pipeline while previewing the local camera feed. The problem happens when you show the view and enable the local video track after disabling it.

I was able to reproduce the problem on your branch without applying any changes. Then I modified toggleMic method on your branch to following when Open GL ES Renderer is used, and I could still reproduce the problem -

@IBAction func toggleMic(sender: AnyObject) {
    self.localVideoTrack?.isEnabled = self.previewViewHolder.isHidden
    if(self.previewViewHolder.isHidden) {
        self.previewViewHolder.isHidden = false
    }
    else {
        self.previewViewHolder.isHidden = true
    }
}

Video SDK supports two video renderer types: Metal and OpenGL ES. For 64-bit devices, the default renderer SDK uses is Metal Renderer, otherwise, OpenGL ES Renderer is used. Metal Renderer is not supported on non 64-bit devices. So, for iPhone 5, SDK uses Open GL ES Renderer pipeline for video rendering, where we have bug in our SDK and that's why the problem is reproducible on iPhone 5 but not on iPhone 6. (Here is the documentation)

We will prioritize the issue and release the fix very soon. In the meantime is it possible for you to apply one of the following workarounds? :

Workaround - 1 - Instead of enabling and disabling the track, can you remove the track and add it back in your app?

Workaround -2 - ImplementingTVIVideoViewDelegate rotation APIs. Unfortunately, our current sample apps do not demonstrate usage of these APIs yet, but we are planning to add a sample app soon. I can help using rotation APIs in your app if you decided to use it as workaround the problem on non 64-bit devices. see this

I apologize for the inconvenience. I will let you know once the fix becomes available.
Let me know if you have any questions.

[alexiss03]

Hi Piyush,

Thank you for your timely and detailed response. I'll try the workarounds you suggested and will let you know what will work on my end.

Best,
Lexi

[alexiss03]

Hi Piyush,

I tried implementing the Workarounds you suggested.

Workaround 1 won't fit in the desired behavior of my app. Yes, the TVILocalVideoTrack will not be visible to the app user but will still be visible to the other participants in the Room, even after removing it.

As forWorkaround 2, the method videoView:videoOrientationDidChange: was not called after the rotation has changed.

Best,
Lexi

[piyushtank]

@alexiss03 When local video track gets disabled, other participants in the Room receives black frames. The renderer added to local video track on local device receives black frames as well.

The root cause of the problem is, on Open GL ES renderer, after the local video track is disabled, frame rotation gets incorrect values. That's why TVIVideoViewDelegate's rotation delegate method is not called reliably.

At this point, I would recommend waiting for a few days until we fix the problem. As mentioned earlier, we have prioritized this issue and the fix will be released in a few day. I will update you when we release the fix.

Thanks for your patience, and I apologize for the inconvenience!

[alexiss03]

Hi Piyush,

I'm looking forward on your fix. Thanks for the urgent response!

Cheers,
Lexi

[paynerc]

@alexiss03,

I just wanted to touch base to let you know that we just released 1.0.1 of Twilio Video and it contains the fix for the issue that you reported.

Let us know if you have any further issues!

Ryan

[alexiss03]

Hi Ryan,

The fix works perfectly on my end. Thanks!

Best,
Lexi

[kselvin]

Hi,

I am now experiencing a similar issue on iOS 13 Beta. (Everything works perfectly on iOS 12 still)

Everything works as intended initially, but when the camera source gets interrupted then uninterrupted (which i do at points intentionally in my app),

and the function cameraSourceInterruptionEnded(_ source: TVICameraSource)
is called, the local video track is distorted

Any help would be greatly appreciated

Thank you!

[ceaglest]

Hi @kselvin,

Could you please open up a separate issue for the distortion you are seeing on iOS 13? It would be great to know exact reproduction steps, for example would backgrounding and foregrounding the QuickStart example cause the same distortion to occur? On what device model and SDK version does the bug occur?

Thank you,
Chris

[kselvin]

It is not actually in the example app it is in my own application -- however I can confirm this is not an issue when running iOS 12 on the same device model (iPhone 7 Plus)

To replicate

1. Camera source interrupted with reason videoDeviceInUseByAnotherClient -- I use the camera for something else (within my app) while still in the chat room
2. I exit out of my other camera view and receive : camera source uninterrupted -- other users can see the current user again
   Current user's videotrack is distorted until i actively disconnect and reconnect again to the session

Using:
iPhone 7 Plus running iOS 13.1 Beta, build compiled in Xcode 10 (also tried in Xcode 11 Beta with same result)

Note: Backgrounding and foregrounding does NOT cause the issue -- only when interruption is caused with reason = "used by another client"

Where would be a better place to write this?

Thanks a ton!

• Keith

[ceaglest]

Hi @kselvin,

Thank you! If possible could you open an issue in: https://github.com/twilio/twilio-video-ios/issues with this information.

I'd be happy to look into it. I tried the backgrounding and foregrounding case on iOS 13.0-beta8, but as you mentioned I was not able to reproduce the corruption that way.

Best,
Chris

[kselvin]

Thanks doing it now!