Disabling local video track causes crash

[MilanNosal]

I am using TwilioVideo for calls over the internet. I am trying to support disabling video.

Based on the example code, I expected this would do the trick:

localVideoTrack?.isEnabled = false

Where localVideoTrack is an instance of TVILocalVideoTrack created using:

let camera = TVICameraCapturer(source: .frontCamera, delegate: self)
localVideoTrack = TVILocalVideoTrack.init(capturer: camera!)

I used basically the same approach with the TVILocalAudioTrack to mute and unmute microphone, which works like a charm.

However, with video, when I call:

localVideoTrack?.isEnabled = false

I get a crash:

2018-09-25 17:58:47.270895+0200 App[714:101541] -[TVIRTCI420Buffer coreVideoFrameBuffer]: unrecognized selector sent to instance 0x280d8ce00
2018-09-25 17:58:47.289063+0200 App[714:101541] Encountered an uncaught exception. All Mixpanel instances were archived.
2018-09-25 17:58:47.289128+0200 App[714:101541] *** Terminating app due to uncaught exception 'NSInvalidArgumentException', reason: '-[TVIRTCI420Buffer coreVideoFrameBuffer]: unrecognized selector sent to instance 0x280d8ce00'
*** First throw call stack:
(0x18f53bef8 0x18e709a40 0x18f453154 0x18f541810 0x18f5434bc 0x1043b754c 0x1043b6cac 0x1043f4524 0x10460e358 0x10461ed0c 0x104431f58 0x104433d18 0x10443bf24 0x109068de4 0x109070e88 0x109071b7c 0x10907bc18 0x18f1560f0 0x18f158d00)
libc++abi.dylib: terminating with uncaught exception of type NSException

Any idea why this is happening?

[paynerc]

@MilanNosal

Can you tell me what version of TwilioVideo you are using where you see this crash occurring?

Thanks,

Ryan

[MilanNosal]

I used 2.3, yesterday I have updated to 2.4 and the crash still occurs.. Could it be something with renderers assigned to track? Or the room settings?

[paynerc]

Thanks for getting back so quick. I don't believe it has anything to do with room settings. I just tested this in our internal application and I am not seeing the crash.

Is this crash occurring on the device where you are disabling the local video or on the remote participant side? Are you able to share the source showing how you are rendering the local video track?

I would like to try and get a setup where I can reproduce the crash you are seeing locally.

Thank you,

Ryan

[MilanNosal]

@paynerc
Hi! So I've just found out that the same error happens in SampleHandler.swift that's part of your example application when you try to disable the track there. it's from the ReplayKit example, there the broadcast extension. I believe you can replicate it very easily yourself, but here's the code with a couple of changes made by me to be able to try it out (to pinpoint my changes, I've added couple of FIXME: comments on places where I have added or changes something.. as you can see it's really almost no changes at all).

One thing that may be relevant is that even when you disable the local video track before connecting to room, it crashes only after the twilio connects to the room. Could it be the problem with room configuration, or with access token?

//
//  SampleHandler.swift
//  BroadcastExtension
//
//  Created by Piyush Tank on 7/1/18.
//  Copyright © 2018 Twilio. All rights reserved.
//

import Accelerate
import TwilioVideo
import ReplayKit

class SampleHandler: RPBroadcastSampleHandler, TVIRoomDelegate, TVIVideoCapturer {

    public var isScreencast: Bool = true

    // Video SDK components
    public var room: TVIRoom?
    weak var captureConsumer: TVIVideoCaptureConsumer?

    static let kDesiredFrameRate = 30
    static let kDownScaledFrameWidth = 540
    static let kDownScaledFrameHeight = 960

    let audioDevice = ExampleCoreAudioDevice()
    
    // FIXME: Added reference so I get back to it later
    var localScreenTrack: TVILocalVideoTrack?

    public var supportedFormats: [TVIVideoFormat] {
        get {
            /*
             * Describe the supported format.
             * For this example we cheat and assume that we will be capturing the entire screen.
             */
            let screenSize = UIScreen.main.bounds.size
            let format = TVIVideoFormat()
            format.pixelFormat = TVIPixelFormat.formatYUV420BiPlanarFullRange
            format.frameRate = UInt(SampleHandler.kDesiredFrameRate)
            format.dimensions = CMVideoDimensions(width: Int32(screenSize.width), height: Int32(screenSize.height))
            return [format]
        }
    }

    override func broadcastStarted(withSetupInfo setupInfo: [String : NSObject]?) {

        TwilioVideo.audioDevice = ExampleCoreAudioDevice(audioCapturer: self)

        // User has requested to start the broadcast. Setup info from the UI extension can be supplied but optional.
        
        // FIXME: My own access token
        let accessToken = "eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCIsImN0eSI6InR3aWxpby1mcGE7dj0xIn0.eyJqdGkiOiJTSzJiNTgyZDc0ZDVlMDc2OTA0MmUxYzIxZTdiYWE1MWU5LTE1Mzg0MTg4NzkiLCJncmFudHMiOnsiaWRlbnRpdHkiOiIxNTIxMyIsInZpZGVvIjp7InJvb20iOiI2Rjk3OTM3Ri1EQzJCLTQ4MTQtQUI5Mi0wQjU2QzI3NDRCQkUifX0sImlhdCI6MTUzODQxODg3OSwiZXhwIjoxNTM4NDIyNDc5LCJpc3MiOiJTSzJiNTgyZDc0ZDVlMDc2OTA0MmUxYzIxZTdiYWE1MWU5Iiwic3ViIjoiQUNiYmJlNzZhODg1N2E0NzY0ZmZhZGUzODU2MDY1N2ZjZiJ9.Q6wcAVUisyJQRrGbQJ0qfU_LRS9qmLKyOZ__vVnRbZU";

        // FIXME: storing reference to local track
        localScreenTrack = TVILocalVideoTrack(capturer: self)
        let h264VideoCodec = TVIH264Codec()
        let localAudioTrack = TVILocalAudioTrack()
        let connectOptions = TVIConnectOptions.init(token: accessToken) { (builder) in

            // Use the local media that we prepared earlier.
            builder.audioTracks = [localAudioTrack!]
            builder.videoTracks = [self.localScreenTrack!]

            // Use the preferred video codec
            builder.preferredVideoCodecs = [h264VideoCodec] as! [TVIVideoCodec]

            // The name of the Room where the Client will attempt to connect to. Please note that if you pass an empty
            // Room `name`, the Client will create one for you. You can get the name or sid from any connected Room.
            
            // FIXME: I commented out the if, since I'm using the same way in both cases + my own name of a room for which access token is valid
//            if #available(iOS 12.0, *) {
                builder.roomName = "6F97937F-DC2B-4814-AB92-0B56C2744BBE"
//            } else {
//                builder.roomName = setupInfo?["RoomName"] as? String
//            }
        }

        // Connect to the Room using the options we provided.
        room = TwilioVideo.connect(with: connectOptions, delegate: self)

        // User has requested to start the broadcast. Setup info from the UI extension can be supplied but optional.
        print("broadcastStarted")
    }

    override func broadcastPaused() {
        // User has requested to pause the broadcast. Samples will stop being delivered.
    }

    override func broadcastResumed() {
        // User has requested to resume the broadcast. Samples delivery will resume.
    }

    override func broadcastFinished() {
        // User has requested to finish the broadcast.
    }

    override func processSampleBuffer(_ sampleBuffer: CMSampleBuffer, with sampleBufferType: RPSampleBufferType) {
        RPScreenRecorder.shared().isMicrophoneEnabled = true
        switch sampleBufferType {
        case RPSampleBufferType.video:
            if ((captureConsumer != nil) && room?.state == .connected) {
                processVideoSampleBuffer(sampleBuffer)
            }
            break
        case RPSampleBufferType.audioApp:
            if (room?.state == .connected) {
                ExampleCoreAudioDeviceRecordCallback(sampleBuffer)
            }
            break
        case RPSampleBufferType.audioMic:
            if (room?.state == .connected) {
                ExampleCoreAudioDeviceRecordCallback(sampleBuffer)
            }
            break
        }
    }

    var tempPixelBuffer : CVPixelBuffer?;

    func startCapture(_ format: TVIVideoFormat, consumer: TVIVideoCaptureConsumer) {
        captureConsumer = consumer
        captureConsumer!.captureDidStart(true)

        CVPixelBufferCreate(kCFAllocatorDefault,
                            480,//self.width,
            640, //self.height,
            kCVPixelFormatType_420YpCbCr8BiPlanarFullRange,
            nil,
            &tempPixelBuffer)
    }

    func stopCapture() {
        print("Stop capturing.")
    }

    // MARK:- Private
    func processVideoSampleBuffer(_ sampleBuffer: CMSampleBuffer) {
        //        let  imageBuffer = sampleBuffer.imageBuffer!
//        let pixelBuffer = sampleBuffer.imageBuffer!
        // FIXME: got error on previous line, had to replace it with this:
        guard let pixelBuffer: CVPixelBuffer = CMSampleBufferGetImageBuffer(sampleBuffer) else { return }
        var outPixelBuffer : CVPixelBuffer? = nil

        CVPixelBufferLockBaseAddress(pixelBuffer, [])

        let pixelFormat = CVPixelBufferGetPixelFormatType(pixelBuffer)

        if (pixelFormat != kCVPixelFormatType_420YpCbCr8BiPlanarFullRange) {
            assertionFailure("Extension assumes the incoming frames are of type NV12")
        }

        let status = CVPixelBufferCreate(kCFAllocatorDefault,
                                         SampleHandler.kDownScaledFrameWidth,
                                         SampleHandler.kDownScaledFrameHeight,
                                         pixelFormat,
                                         nil,
                                         &outPixelBuffer);
        if (status != kCVReturnSuccess) {
            print("Failed to create pixel buffer");
        }

        CVPixelBufferLockBaseAddress(outPixelBuffer!, []);

        // Prepare source pointers.
        var sourceImageY = vImage_Buffer(data: CVPixelBufferGetBaseAddressOfPlane(pixelBuffer, 0),
                                         height: vImagePixelCount(CVPixelBufferGetHeightOfPlane(pixelBuffer, 0)),
                                         width: vImagePixelCount(CVPixelBufferGetWidthOfPlane(pixelBuffer, 0)),
                                         rowBytes: CVPixelBufferGetBytesPerRowOfPlane(pixelBuffer, 0))

        var sourceImageUV = vImage_Buffer(data: CVPixelBufferGetBaseAddressOfPlane(pixelBuffer, 1),
                                          height: vImagePixelCount(CVPixelBufferGetHeightOfPlane(pixelBuffer, 1)),
                                          width:vImagePixelCount(CVPixelBufferGetWidthOfPlane(pixelBuffer, 1)),
                                          rowBytes: CVPixelBufferGetBytesPerRowOfPlane(pixelBuffer, 1))

        // Prepare out pointers.
        var outImageY = vImage_Buffer(data: CVPixelBufferGetBaseAddressOfPlane(outPixelBuffer!, 0),
                                      height: vImagePixelCount(CVPixelBufferGetHeightOfPlane(outPixelBuffer!, 0)),
                                      width: vImagePixelCount(CVPixelBufferGetWidthOfPlane(outPixelBuffer!, 0)),
                                      rowBytes: CVPixelBufferGetBytesPerRowOfPlane(outPixelBuffer!, 0))

        var outImageUV = vImage_Buffer(data: CVPixelBufferGetBaseAddressOfPlane(outPixelBuffer!, 1),
                                       height: vImagePixelCount(CVPixelBufferGetHeightOfPlane(outPixelBuffer!, 1)),
                                       width:vImagePixelCount( CVPixelBufferGetWidthOfPlane(outPixelBuffer!, 1)),
                                       rowBytes: CVPixelBufferGetBytesPerRowOfPlane(outPixelBuffer!, 1))


        var error = vImageScale_Planar8(&sourceImageY,
                                        &outImageY,
                                        nil,
                                        vImage_Flags(0));
        if (error != kvImageNoError) {
            print("Failed to down scale luma plane ")
            return;
        }

        error = vImageScale_CbCr8(&sourceImageUV,
                                      &outImageUV,
                                      nil,
                                      vImage_Flags(0));
        if (error != kvImageNoError) {
            print("Failed to down scale chroma plane")
            return;
        }

        CVPixelBufferUnlockBaseAddress(outPixelBuffer!, []);
        CVPixelBufferUnlockBaseAddress(pixelBuffer, []);

        let time: CMTime = CMSampleBufferGetPresentationTimeStamp(sampleBuffer)
        let frame = TVIVideoFrame(timestamp: time,
                                  buffer: outPixelBuffer!,
                                  orientation: TVIVideoOrientation.up)

        captureConsumer?.consumeCapturedFrame(frame!)
    }

    // MARK:- TVIRoomDelegate

    func didConnect(to room: TVIRoom) {
        print("didConnectToRoom")
        
        // FIXME: Connect to the room and try to disable the track..
        // When you disable it earlier, it still crashes only after connecting to the room
        self.localScreenTrack?.isEnabled = false
    }

    func room(_ room: TVIRoom, didFailToConnectWithError error: Error) {
        print("didFailToConnectWithError")
    }

    func room(_ room: TVIRoom, didDisconnectWithError error: Error?) {
        print("didDisconnectWithError")
    }

    func room(_ room: TVIRoom, participantDidConnect participant: TVIRemoteParticipant) {
        print("participantDidConnect")
    }

    func room(_ room: TVIRoom, participantDidDisconnect participant: TVIRemoteParticipant) {
        print("participantDidDisconnect")
    }
}

[aastlind]

Having the same issue here, version 2.4.1.
The other part of the conversation understands that the track is disabled.

This is a collapsed log to see some timestamps, first row is our own log that indicates that the track has been disabled.

2018-10-03 17:30:00.321818+0200 ***[12242:3536754] ***: [Twilio_Room_Local_Event] {"event":"disabledVideoTrack"}
2018-10-03 17:30:00.341165+0200 ***[12242:3537154] -[TVIRTCI420Buffer coreVideoFrameBuffer]: unrecognized selector sent to instance 0x281d58f40

[paynerc]

I am digging into this right now and will post as soon as I am able to replicate.

Quick question, what iOS versions are you seeing this on?

Ryan

[MilanNosal]

Both my devices are on iOS 12 GM

[aastlind]

Great!
In my case 12.0 also. Haven't tested it on 11.x.

[paynerc]

OK. That's good to know. I will test on the various devices I have and see if I can reproduce and if there is any iOS version relation.

[paynerc]

@MilanNosal, @aastlind

Also, what device models are you testing on?

[aastlind]

iPhone 8 Plus.
I’ll have a iPhone 6 with iOS 11.x available tomorrow, will test on that also.

[paynerc]

OK. I have tested on both an iPhone 6Plus and an iPhone X with iOS 12 and am not seeing anything yet. I don't have an iPhone 8 Plus.. I have a 7 Plus with iOS 11 and was also not able to reproduce the issue.

A few more questions for you on this:

1. Do you have a RoomSid for a room that you were connected to when the crash occurred? I would like to pull the server logs to see the room configuration, etc. If you have a RoomSid, you can email it to support@twilio.com and reference this ticket and then let me know that you sent it so I know to look for it.
2. When the crash occurs, can you capture the stack trace of the crash? You can type bt in the debugger when the crash occurs and just attach that to this issue. Or if you are running outside of the debugger, the actual crash report would help a ton. I can resymbolicate that and hopefully get a better idea of what actually is happening.

I am interested to hear if the crash is related to the iPhone 8 Plus or happens on other devices as well.

Ryan

[aastlind]

Stack trace (bt):

* thread #72, queue = 'EncoderQueue', stop reason = signal SIGABRT
  * frame #0: 0x000000018a509104 libsystem_kernel.dylib`__pthread_kill + 8
    frame #1: 0x000000018a588a00 libsystem_pthread.dylib`pthread_kill$VARIANT$armv81 + 296
    frame #2: 0x000000018a460d78 libsystem_c.dylib`abort + 140
    frame #3: 0x0000000189b28f78 libc++abi.dylib`abort_message + 132
    frame #4: 0x0000000189b29120 libc++abi.dylib`default_terminate_handler() + 304
    frame #5: 0x0000000189b41e48 libobjc.A.dylib`_objc_terminate() + 124
    frame #6: 0x0000000189b350fc libc++abi.dylib`std::__terminate(void (*)()) + 16
    frame #7: 0x0000000189b34a40 libc++abi.dylib`__cxa_throw + 132
    frame #8: 0x0000000189b41b84 libobjc.A.dylib`objc_exception_throw + 380
    frame #9: 0x000000018a88b154 CoreFoundation`-[NSObject(NSObject) doesNotRecognizeSelector:] + 140
    frame #10: 0x000000018a979810 CoreFoundation`___forwarding___ + 1412
    frame #11: 0x000000018a97b4bc CoreFoundation`_CF_forwarding_prep_0 + 92
    frame #12: 0x00000001057c354c TwilioVideo`-[TVIVideoEncoderH264 resetCompressionSessionIfNeededForPool:withFrame:] + 352
    frame #13: 0x00000001057c2cac TwilioVideo`-[TVIVideoEncoderH264 encode:codecSpecificInfo:frameTypes:] + 224
    frame #14: 0x0000000105800524 TwilioVideo`webrtc::(anonymous namespace)::ObjCVideoEncoder::Encode(webrtc::VideoFrame const&, webrtc::CodecSpecificInfo const*, std::__1::vector<webrtc::FrameType, std::__1::allocator<webrtc::FrameType> > const*) + 332
    frame #15: 0x0000000105a1a358 TwilioVideo`webrtc::VCMGenericEncoder::Encode(webrtc::VideoFrame const&, webrtc::CodecSpecificInfo const*, std::__1::vector<webrtc::FrameType, std::__1::allocator<webrtc::FrameType> > const&) + 212
    frame #16: 0x0000000105a2ad0c TwilioVideo`webrtc::vcm::VideoSender::AddVideoFrame(webrtc::VideoFrame const&, webrtc::CodecSpecificInfo const*) + 732
    frame #17: 0x000000010583df58 TwilioVideo`webrtc::VideoStreamEncoder::EncodeVideoFrame(webrtc::VideoFrame const&, long long) + 1312
    frame #18: 0x000000010583fd18 TwilioVideo`rtc::ClosureTask<webrtc::VideoStreamEncoder::OnFrame(webrtc::VideoFrame const&)::$_6>::Run() + 144
    frame #19: 0x0000000105847f24 TwilioVideo`rtc::TaskQueue::Impl::TaskContext::RunTask(void*) + 60
    frame #20: 0x000000010a954de4 libdispatch.dylib`_dispatch_client_callout + 16
    frame #21: 0x000000010a95ce88 libdispatch.dylib`_dispatch_lane_serial_drain + 720
    frame #22: 0x000000010a95db7c libdispatch.dylib`_dispatch_lane_invoke + 460
    frame #23: 0x000000010a967c18 libdispatch.dylib`_dispatch_workloop_worker_thread + 1220
    frame #24: 0x000000018a58e0f0 libsystem_pthread.dylib`_pthread_wqthread + 312
    frame #25: 0x000000018a590d00 libsystem_pthread.dylib`start_wqthread + 4

I also emailed the RoomSid to support@twilio.com (support ticket 1528481).

[etown]

See this on multiple devices

[ceaglest]

Hey @aastlind,

Thanks for the stack trace. I believe that, in 2.3.0 and later, we are making a bad assumption in the H.264 encoder about frame types, and formats. Disabling the Track causes software I420 buffers to be produced for the black frames, and our encoder was not expecting this type.

Regards,
Chris

[aastlind]

@ceaglest: Only to confirm, I just run a test removing H.264 as preferred video codec (leaving VP9 and VP8 as remaining preferred codecs) and then it works fine to disable the track.

Good to see that you know where to start! :)

[piyushtank]

@aastlind I apologize for the issue. Yes, we have a bug in our H.264 video encoder. I am working on the fix and aiming to release it in a couple of days. Thanks.

[paynerc]

@MilanNosal @aastlind

We released Twilio Video iOS 2.5.1 today which addresses this crash.

Let us know if this has resolved your issue and then we'll close out this issue.

Thanks for your patience while we diagnosed and fixed the issue.

Ryan

[MilanNosal]

@paynerc
The issue is resolved, thank you for support and quick resolution.

[riddhixtraining]

HI @paynerc

I am getting this crash again with 'TwilioVideo', '~> 5'
I am testing in
iPad(6th Generation)
iOS : 15.3.1

When I try to disable localVideoTrack, app just freeze in debug mode and on TestFlight build it crashes. I have that crash log if you want.

Also, I have sent an email of Sid

Thank you

[paynerc]

@riddhixtraining,

Tagging @piyushtank on this issue as I am no longer with Twilio.

Ryan