Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Crash with 960x540 video & network congestion #116

Closed
aobondar opened this issue Apr 27, 2017 · 16 comments

Comments

Projects
None yet
4 participants
@aobondar
Copy link

commented Apr 27, 2017

Hi there!

I'm using TwillioVideo GA lib for 1-to-1 video conversation. It used to work well, but when i set the 16:9 video dimension for capturer it started crashing. The same scenario: after ~10 secs the video quality starts degradating, and after the calle have a crash with the same callstack:

Thread 15Queue : com.apple.avfoundation.videodataoutput.bufferqueue (serial)
#0	0x000000010256dde8 in twilio::media::CVPixelBufferCreateWithVideoFrameBuffer(webrtc::VideoFrameBuffer const*, CVPlanarPixelBufferInfo_YCbCrPlanar*) ()
#1	0x000000010256e544 in twilio::media::VideoRendererNativeAdapter::OnFrame(webrtc::VideoFrame const&) ()
#2	0x000000010282d0e4 in rtc::VideoBroadcaster::OnFrame(webrtc::VideoFrame const&) ()
#3	0x000000010282e080 in cricket::VideoCapturer::OnFrame(webrtc::VideoFrame const&, int, int) ()
#4	0x000000010254c794 in twilio::media::CoreVideoCapturer::OnCapturedFrame(TVIVideoFrame*) ()
#5	0x000000010254c578 in twilio::media::CoreVideoCapturer::CopyCapturedFrame(TVIVideoFrame*) ()
#6	0x000000010254ce38 in -[TVICoreVideoCapturer consumeCapturedFrame:] ()
#7	0x0000000102549cec in -[TVICameraCapturePipeline captureOutput:didOutputSampleBuffer:fromConnection:] ()
#8	0x0000000193c8f6bc in -[AVCaptureVideoDataOutput _handleRemoteQueueOperation:] ()
#9	0x0000000193c8f4f8 in __47-[AVCaptureVideoDataOutput _updateRemoteQueue:]_block_invoke ()
#10	0x000000018edf4d94 in __FigRemoteOperationReceiverCreateMessageReceiver_block_invoke ()
#11	0x000000018ee13dcc in __FigRemoteQueueReceiverSetHandler_block_invoke.2 ()
.....
#23	0x000000018b52ed8c in start_wqthread ()

All the other threads are waiting. I user swift3, latest available version of lib. Your help would be appreciated!!!

@aobondar

This comment has been minimized.

Copy link
Author

commented Apr 27, 2017

Alse there were a record in log:

ERROR:TwilioVideo:[Platform]:Fatal runtime capture error: Error Domain=AVFoundationErrorDomain Code=-11800 "The operation could not be completed" UserInfo={NSUnderlyingError=0x17064d500 {Error Domain=NSOSStatusErrorDomain Code=-12785 "(null)"}, NSLocalizedFailureReason=An unknown error occurred (-12785), NSLocalizedDescription=The operation could not be completed}, code -11800

@piyushtank

This comment has been minimized.

Copy link
Collaborator

commented Apr 28, 2017

@aobondar Setting video constraints should not cause a crash. Can you provide more details about your use case and how you are applying video constraints? Also, can you send details about the device and iOS version you are running your app on, along with a stack trace of all the threads? Thanks.

@aobondar

This comment has been minimized.

Copy link
Author

commented Apr 28, 2017

Probably, you're right, but when i removed the constraints, i started working well again. Reproduced 4/4.
Here's a snippet, called on didConnect(to room: TVIRoom).

camera = TVICameraCapturer()
let constraints = TVIVideoConstraints(block: { (builder: TVIVideoConstraintsBuilder) in
    //builder.aspectRatio = TVIAspectRatio16x9
    // commented out to avoid crashes
})
localVideoTrack = TVILocalVideoTrack(capturer: camera!, enabled: true, constraints: constraints)
if let unwrVideo = localVideoTrack {
    localMedia?.addVideoTrack(unwrVideo)
}

Reproducing on iOS 10.1 and 10.2. iPhone 6 & iPhone 6s. Crashdump is in attachment
crashdump.txt

Update: oops. Here's symbolificated one.
symbolificated_dump.txt

@piyushtank

This comment has been minimized.

Copy link
Collaborator

commented Apr 29, 2017

@aobondar We have removed TVILocalMedia and refactored few APIs recently. I recommend using TwilioVideo GA release 1.0.0 in your application.

Changelogs - https://www.twilio.com/docs/api/video/changelogs/ios#100-april-26-2017

I have created following branch where 16:9 aspect ratio video constraints is applied on Quickstart. Try running the quickstart from the branch and let me know if you still get the crash on setting the constraints.

Branch: https://github.com/ptankTwilio/video-quickstart-swift/tree/video-constraints-test

Code which sets up the up the constraint: https://github.com/ptankTwilio/video-quickstart-swift/blob/video-constraints-test/VideoQuickStart/ViewController.swift#L174

@aobondar

This comment has been minimized.

Copy link
Author

commented Apr 29, 2017

Ah, i see, sorry for misinformation - it was a nonrefactored snipped, just before mowing to GA. The correct one is:

camera = TVICameraCapturer()
let constraints = TVIVideoConstraints(block: { (builder: TVIVideoConstraintsBuilder) in
    //builder.aspectRatio = TVIAspectRatio16x9
    // commented out to avoid crashes
})
localVideoTrack = TVILocalVideoTrack(capturer: camera!, enabled: true, constraints: constraints)
if let unwrVideo = localVideoTrack {
    // localParticipator is obtained from room object
    localParticipator?.addVideoTrack(unwrVideo)
}

Thank you for branch, i think i'll be able to continue investigation after several days, and i will send more info ASAP.

@piyushtank

This comment has been minimized.

Copy link
Collaborator

commented May 1, 2017

@aobondar I have used the code snippet you have sent to add and remove video on branch video-constraints-test (the branch I sent you in my last post). The Video constraints worked as expected, no crash observed.

Branch: Branch: https://github.com/ptankTwilio/video-quickstart-swift/tree/video-constraints-test

Quickstart app uses the code snippet you sent:
https://github.com/ptankTwilio/video-quickstart-swift/blob/video-constraints-test/VideoQuickStart/ViewController.swift#L152-L174

Please try running Quickstart app from this branch and let me know if you are still observing the crash.

@aastlind

This comment has been minimized.

Copy link

commented May 9, 2017

In one of our test runs (1.0.0 GA) we experienced the same crash (iPhone SE), but we use the following constraints:

builder.minSize = TVIVideoConstraintsSize960x540;
builder.maxSize = TVIVideoConstraintsSize1280x720;
builder.minFrameRate = TVIVideoConstraintsFrameRateNone;
builder.maxFrameRate = TVIVideoConstraintsFrameRateNone;

And this was the call stack:

Crashed: com.apple.avfoundation.videodataoutput.bufferqueue
0  TwilioVideo                    0x1003f1de8 twilio::media::CVPixelBufferCreateWithVideoFrameBuffer(webrtc::VideoFrameBuffer const*, CVPlanarPixelBufferInfo_YCbCrPlanar*) + 52
1  TwilioVideo                    0x1003f2544 twilio::media::VideoRendererNativeAdapter::OnFrame(webrtc::VideoFrame const&) + 340
2  TwilioVideo                    0x1006b10e4 rtc::VideoBroadcaster::OnFrame(webrtc::VideoFrame const&) + 228
3  TwilioVideo                    0x1006b2080 cricket::VideoCapturer::OnFrame(webrtc::VideoFrame const&, int, int) + 228
4  TwilioVideo                    0x1003d0794 twilio::media::CoreVideoCapturer::OnCapturedFrame(TVIVideoFrame*) + 476
5  TwilioVideo                    0x1003d0578 twilio::media::CoreVideoCapturer::CopyCapturedFrame(TVIVideoFrame*) + 288
6  TwilioVideo                    0x1003d0e38 -[TVICoreVideoCapturer consumeCapturedFrame:] + 52
7  TwilioVideo                    0x1003cdcec -[TVICameraCapturePipeline captureOutput:didOutputSampleBuffer:fromConnection:] + 472
8  AVFoundation                   0x18bb9e310 -[AVCaptureVideoDataOutput _handleRemoteQueueOperation:] + 308
9  AVFoundation                   0x18bb9e14c __47-[AVCaptureVideoDataOutput _updateRemoteQueue:]_block_invoke + 100
10 CoreMedia                      0x186a73f68 __FigRemoteOperationReceiverCreateMessageReceiver_block_invoke + 260
11 CoreMedia                      0x186a92e9c __FigRemoteQueueReceiverSetHandler_block_invoke.2 + 224
12 libdispatch.dylib              0x1830069a0 _dispatch_client_callout + 16
13 libdispatch.dylib              0x183013604 _dispatch_continuation_pop + 448
14 libdispatch.dylib              0x18301fc1c _dispatch_source_latch_and_call + 204
15 libdispatch.dylib              0x1830088a0 _dispatch_source_invoke + 804
16 libdispatch.dylib              0x183014964 _dispatch_queue_serial_drain + 560
17 libdispatch.dylib              0x18300a2cc _dispatch_queue_invoke + 884
18 libdispatch.dylib              0x183014964 _dispatch_queue_serial_drain + 560
19 libdispatch.dylib              0x18300a2cc _dispatch_queue_invoke + 884
20 libdispatch.dylib              0x183015950 _dispatch_root_queue_drain_deferred_item + 256
21 libdispatch.dylib              0x18301c170 _dispatch_kevent_worker_thread + 760
22 libsystem_pthread.dylib        0x18320f08c _pthread_wqthread + 772
23 libsystem_pthread.dylib        0x18320ed7c start_wqthread + 4
@aobondar

This comment has been minimized.

Copy link
Author

commented May 9, 2017

@ptankTwilio Sorry for not answering so long. I've encountered the same crash on given branch. Here i attach the stacktraces, captured from XCode on crash, and all the build symbols for investigation.

https://drive.google.com/open?id=0B2OVY0Xc-YAucVpFbk5DN0FVcUk

@ceaglest

This comment has been minimized.

Copy link
Member

commented May 9, 2017

Thank you for the feedback @aobondar and @aastlind! @ptankTwilio is looking into this, and may reach out for more info today.

@ceaglest ceaglest added the bug label May 11, 2017

@ceaglest ceaglest assigned ceaglest and unassigned piyushtank May 11, 2017

@ceaglest

This comment has been minimized.

Copy link
Member

commented May 11, 2017

Quick update: I was able to reproduce the crash in poor network conditions when frames are downscaled at the front of the capture pipeline (not the encoder). I'm looking into a fix this afternoon, and am planning to include it in our next 1.0.1 release.

A temporary workaround (though not a great one) is to choose 1280x720 instead of 960x540.

builder.minSize = TVIVideoConstraintsSize1280x720;
builder.maxSize = TVIVideoConstraintsSize1280x720;
@ceaglest

This comment has been minimized.

Copy link
Member

commented May 12, 2017

I've fixed this crash, and our QA team will begin internal validation tomorrow. Expect the fix to land in 1.0.1 over the next few business days.

@aobondar

This comment has been minimized.

Copy link
Author

commented May 12, 2017

@ceaglest, thank you for help! Looking forward for new build!

@ceaglest ceaglest changed the title Crash with 16:9 video dimension Crash with 960x540 video, network congestion and downscaling May 13, 2017

@ceaglest ceaglest changed the title Crash with 960x540 video, network congestion and downscaling Crash with 960x540 video & network congestion May 13, 2017

@ceaglest

This comment has been minimized.

Copy link
Member

commented May 15, 2017

Hi @aobondar and @aastlind,

We've just released 1.0.1 of Twilio Video which contains the fix for this crasher. Feel free to capture at 960x540 in whatever network conditions you want using the new release.

https://www.twilio.com/docs/api/video/changelogs/ios#101-may-15-2017

Best,
Chris Eagleston

@aastlind

This comment has been minimized.

Copy link

commented May 15, 2017

Nice work, thank you so much!

@aobondar

This comment has been minimized.

Copy link
Author

commented May 15, 2017

@ceaglest, thank you very much!

@ceaglest

This comment has been minimized.

Copy link
Member

commented May 19, 2017

Closing this issue, since it was fixed in 1.0.1.

@ceaglest ceaglest closed this May 19, 2017

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.