Crash with 960x540 video & network congestion

[aobondar]

Hi there!

I'm using TwillioVideo GA lib for 1-to-1 video conversation. It used to work well, but when i set the 16:9 video dimension for capturer it started crashing. The same scenario: after ~10 secs the video quality starts degradating, and after the calle have a crash with the same callstack:

Thread 15Queue : com.apple.avfoundation.videodataoutput.bufferqueue (serial)
#0	0x000000010256dde8 in twilio::media::CVPixelBufferCreateWithVideoFrameBuffer(webrtc::VideoFrameBuffer const*, CVPlanarPixelBufferInfo_YCbCrPlanar*) ()
#1	0x000000010256e544 in twilio::media::VideoRendererNativeAdapter::OnFrame(webrtc::VideoFrame const&) ()
#2	0x000000010282d0e4 in rtc::VideoBroadcaster::OnFrame(webrtc::VideoFrame const&) ()
#3	0x000000010282e080 in cricket::VideoCapturer::OnFrame(webrtc::VideoFrame const&, int, int) ()
#4	0x000000010254c794 in twilio::media::CoreVideoCapturer::OnCapturedFrame(TVIVideoFrame*) ()
#5	0x000000010254c578 in twilio::media::CoreVideoCapturer::CopyCapturedFrame(TVIVideoFrame*) ()
#6	0x000000010254ce38 in -[TVICoreVideoCapturer consumeCapturedFrame:] ()
#7	0x0000000102549cec in -[TVICameraCapturePipeline captureOutput:didOutputSampleBuffer:fromConnection:] ()
#8	0x0000000193c8f6bc in -[AVCaptureVideoDataOutput _handleRemoteQueueOperation:] ()
#9	0x0000000193c8f4f8 in __47-[AVCaptureVideoDataOutput _updateRemoteQueue:]_block_invoke ()
#10	0x000000018edf4d94 in __FigRemoteOperationReceiverCreateMessageReceiver_block_invoke ()
#11	0x000000018ee13dcc in __FigRemoteQueueReceiverSetHandler_block_invoke.2 ()
.....
#23	0x000000018b52ed8c in start_wqthread ()

All the other threads are waiting. I user swift3, latest available version of lib. Your help would be appreciated!!!

[aobondar]

Alse there were a record in log:

ERROR:TwilioVideo:[Platform]:Fatal runtime capture error: Error Domain=AVFoundationErrorDomain Code=-11800 "The operation could not be completed" UserInfo={NSUnderlyingError=0x17064d500 {Error Domain=NSOSStatusErrorDomain Code=-12785 "(null)"}, NSLocalizedFailureReason=An unknown error occurred (-12785), NSLocalizedDescription=The operation could not be completed}, code -11800

[piyushtank]

@aobondar Setting video constraints should not cause a crash. Can you provide more details about your use case and how you are applying video constraints? Also, can you send details about the device and iOS version you are running your app on, along with a stack trace of all the threads? Thanks.

[aobondar]

Probably, you're right, but when i removed the constraints, i started working well again. Reproduced 4/4.
Here's a snippet, called on didConnect(to room: TVIRoom).

camera = TVICameraCapturer()
let constraints = TVIVideoConstraints(block: { (builder: TVIVideoConstraintsBuilder) in
    //builder.aspectRatio = TVIAspectRatio16x9
    // commented out to avoid crashes
})
localVideoTrack = TVILocalVideoTrack(capturer: camera!, enabled: true, constraints: constraints)
if let unwrVideo = localVideoTrack {
    localMedia?.addVideoTrack(unwrVideo)
}

Reproducing on iOS 10.1 and 10.2. iPhone 6 & iPhone 6s. Crashdump is in attachment
crashdump.txt

Update: oops. Here's symbolificated one.
symbolificated_dump.txt

[piyushtank]

@aobondar We have removed TVILocalMedia and refactored few APIs recently. I recommend using TwilioVideo GA release 1.0.0 in your application.

Changelogs - https://www.twilio.com/docs/api/video/changelogs/ios#100-april-26-2017

I have created following branch where 16:9 aspect ratio video constraints is applied on Quickstart. Try running the quickstart from the branch and let me know if you still get the crash on setting the constraints.

Branch: https://github.com/ptankTwilio/video-quickstart-swift/tree/video-constraints-test

Code which sets up the up the constraint: https://github.com/ptankTwilio/video-quickstart-swift/blob/video-constraints-test/VideoQuickStart/ViewController.swift#L174

[aobondar]

Ah, i see, sorry for misinformation - it was a nonrefactored snipped, just before mowing to GA. The correct one is:

camera = TVICameraCapturer()
let constraints = TVIVideoConstraints(block: { (builder: TVIVideoConstraintsBuilder) in
    //builder.aspectRatio = TVIAspectRatio16x9
    // commented out to avoid crashes
})
localVideoTrack = TVILocalVideoTrack(capturer: camera!, enabled: true, constraints: constraints)
if let unwrVideo = localVideoTrack {
    // localParticipator is obtained from room object
    localParticipator?.addVideoTrack(unwrVideo)
}

Thank you for branch, i think i'll be able to continue investigation after several days, and i will send more info ASAP.

[piyushtank]

@aobondar I have used the code snippet you have sent to add and remove video on branch video-constraints-test (the branch I sent you in my last post). The Video constraints worked as expected, no crash observed.

Branch: Branch: https://github.com/ptankTwilio/video-quickstart-swift/tree/video-constraints-test

Quickstart app uses the code snippet you sent:
https://github.com/ptankTwilio/video-quickstart-swift/blob/video-constraints-test/VideoQuickStart/ViewController.swift#L152-L174

Please try running Quickstart app from this branch and let me know if you are still observing the crash.

[aastlind]

In one of our test runs (1.0.0 GA) we experienced the same crash (iPhone SE), but we use the following constraints:

builder.minSize = TVIVideoConstraintsSize960x540;
builder.maxSize = TVIVideoConstraintsSize1280x720;
builder.minFrameRate = TVIVideoConstraintsFrameRateNone;
builder.maxFrameRate = TVIVideoConstraintsFrameRateNone;

And this was the call stack:

Crashed: com.apple.avfoundation.videodataoutput.bufferqueue
0  TwilioVideo                    0x1003f1de8 twilio::media::CVPixelBufferCreateWithVideoFrameBuffer(webrtc::VideoFrameBuffer const*, CVPlanarPixelBufferInfo_YCbCrPlanar*) + 52
1  TwilioVideo                    0x1003f2544 twilio::media::VideoRendererNativeAdapter::OnFrame(webrtc::VideoFrame const&) + 340
2  TwilioVideo                    0x1006b10e4 rtc::VideoBroadcaster::OnFrame(webrtc::VideoFrame const&) + 228
3  TwilioVideo                    0x1006b2080 cricket::VideoCapturer::OnFrame(webrtc::VideoFrame const&, int, int) + 228
4  TwilioVideo                    0x1003d0794 twilio::media::CoreVideoCapturer::OnCapturedFrame(TVIVideoFrame*) + 476
5  TwilioVideo                    0x1003d0578 twilio::media::CoreVideoCapturer::CopyCapturedFrame(TVIVideoFrame*) + 288
6  TwilioVideo                    0x1003d0e38 -[TVICoreVideoCapturer consumeCapturedFrame:] + 52
7  TwilioVideo                    0x1003cdcec -[TVICameraCapturePipeline captureOutput:didOutputSampleBuffer:fromConnection:] + 472
8  AVFoundation                   0x18bb9e310 -[AVCaptureVideoDataOutput _handleRemoteQueueOperation:] + 308
9  AVFoundation                   0x18bb9e14c __47-[AVCaptureVideoDataOutput _updateRemoteQueue:]_block_invoke + 100
10 CoreMedia                      0x186a73f68 __FigRemoteOperationReceiverCreateMessageReceiver_block_invoke + 260
11 CoreMedia                      0x186a92e9c __FigRemoteQueueReceiverSetHandler_block_invoke.2 + 224
12 libdispatch.dylib              0x1830069a0 _dispatch_client_callout + 16
13 libdispatch.dylib              0x183013604 _dispatch_continuation_pop + 448
14 libdispatch.dylib              0x18301fc1c _dispatch_source_latch_and_call + 204
15 libdispatch.dylib              0x1830088a0 _dispatch_source_invoke + 804
16 libdispatch.dylib              0x183014964 _dispatch_queue_serial_drain + 560
17 libdispatch.dylib              0x18300a2cc _dispatch_queue_invoke + 884
18 libdispatch.dylib              0x183014964 _dispatch_queue_serial_drain + 560
19 libdispatch.dylib              0x18300a2cc _dispatch_queue_invoke + 884
20 libdispatch.dylib              0x183015950 _dispatch_root_queue_drain_deferred_item + 256
21 libdispatch.dylib              0x18301c170 _dispatch_kevent_worker_thread + 760
22 libsystem_pthread.dylib        0x18320f08c _pthread_wqthread + 772
23 libsystem_pthread.dylib        0x18320ed7c start_wqthread + 4

[aobondar]

@ptankTwilio Sorry for not answering so long. I've encountered the same crash on given branch. Here i attach the stacktraces, captured from XCode on crash, and all the build symbols for investigation.

https://drive.google.com/open?id=0B2OVY0Xc-YAucVpFbk5DN0FVcUk

[ceaglest]

Thank you for the feedback @aobondar and @aastlind! @ptankTwilio is looking into this, and may reach out for more info today.

[ceaglest]

Quick update: I was able to reproduce the crash in poor network conditions when frames are downscaled at the front of the capture pipeline (not the encoder). I'm looking into a fix this afternoon, and am planning to include it in our next 1.0.1 release.

A temporary workaround (though not a great one) is to choose 1280x720 instead of 960x540.

builder.minSize = TVIVideoConstraintsSize1280x720;
builder.maxSize = TVIVideoConstraintsSize1280x720;

[ceaglest]

I've fixed this crash, and our QA team will begin internal validation tomorrow. Expect the fix to land in 1.0.1 over the next few business days.

[aobondar]

@ceaglest, thank you for help! Looking forward for new build!

[ceaglest]

Hi @aobondar and @aastlind,

We've just released 1.0.1 of Twilio Video which contains the fix for this crasher. Feel free to capture at 960x540 in whatever network conditions you want using the new release.

https://www.twilio.com/docs/api/video/changelogs/ios#101-may-15-2017

Best,
Chris Eagleston

[aastlind]

Nice work, thank you so much!

[aobondar]

@ceaglest, thank you very much!

[ceaglest]

Closing this issue, since it was fixed in 1.0.1.