New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Time lag between addRenderer and actual video is shown on TVIVideoView. #211

Closed
mitolog opened this Issue Dec 14, 2017 · 15 comments

Comments

Projects
None yet
4 participants
@mitolog
Copy link

mitolog commented Dec 14, 2017

@mitolog commented on Fri Dec 15 2017

Description

There is a time lag after videoTrack.addRenderer(self.remoteView!) until participant's video
is actually shown on the remoteView, say 3~7 sec on VideoCallKitQuickStart project. On my own project, it last about 30 sec maximum. It's depends on network bandwidth. But, I don't want to expose a blank view, which shows only remoteView background color, to the user.

How can I detect actual timing that the remote participant's video is shown on remoteView?
Or, Is there any ideas to prevent showing blank view until actual video is shown, or remove time lag?

room type: group
video codec: VP8
recording: disabled
maximum participants: 50

I tried with 2 device, and 2 situations:
wifi - wifi
wifi - 3G

In both situation, time lags are exist though durations are differed.

Code

I assume time lag will start from this code:

ViewController.swift:323 on VideoCallKitQuickStart

    func participant(_ participant: TVIParticipant, addedVideoTrack videoTrack: TVIVideoTrack) {
        logMessage(messageText: "Participant \(participant.identity) added video track")

        if (self.participant == participant) {
            setupRemoteVideoView()
            videoTrack.addRenderer(self.remoteView!)
        }
    }

Versions

TwilioVideo 1.3.8

Xcode

9.0

iOS Versions / iOS Devices

10.2: iPhone6 plus
10.0.1: iPhone5


@bchen-twilio commented on Fri Dec 15 2017

Hi @mitolog

Thanks for choosing Twilio Programmable Video SDK. Can you create a ticket in the Swift Video Quickstart repo?


@mitolog commented on Fri Dec 15 2017

@bchen-twilio
sorry I just posted support ticket.


@bchen-twilio commented on Fri Dec 15 2017

@mitolog no worries.
By the way, have you heard about the Programmable Voice SDK? You can create your awesome VoIP app using the Voice SDK with less than hundred lines of codes ;)

-bobie


@mitolog commented on Fri Dec 15 2017

@bchen-twilio
Thanks for the info. Does Voice SDK supports also Video? I took a glance of Voice SDK but I need Video functions, so I chosen Video SDK.


@mitolog commented on Fri Dec 15 2017

sorry, I mistaken. it's voice-quickstart-swift repository :( I should have posted to video-quickstart-swift...


@bchen-twilio commented on Fri Dec 15 2017

Unfortunately the Voice SDK doesn't support video and the various media capabilities like the Video SDK does. The Voice mainly provides mobile applications the ability to integrate with the TwiML functions of Twilio, and VoIP is one of the great features.


@mitolog commented on Fri Dec 15 2017

@bchen-twilio Thanks for the info again. Let me try if needed in the future! Can you remove this issue? or Should I?


@bchen-twilio commented on Fri Dec 15 2017

You are welcome!
Happy to close the issue for you and please feel free to reach out to us in you run into any problem using the SDKs.

Cheers.

@ceaglest ceaglest self-assigned this Dec 14, 2017

@ceaglest

This comment has been minimized.

Copy link
Member

ceaglest commented Dec 14, 2017

Hey @mitolog,

There is a time lag after videoTrack.addRenderer(self.remoteView!) until participant's video
is actually shown on the remoteView, say 3~7 sec on VideoCallKitQuickStart project. On my own project, it last about 30 sec maximum. It's depends on network bandwidth. But, I don't want to expose a blank view, which shows only remoteView background color, to the user.

30s is a long time to wait for video, that sounds like it might be a poor connectivity issue. If you can share a Room SID where it took this long for video to arrive I'd be happy to dig in further. You mention that 3-7 seconds elapses in VideoCallKitQuickStart between the track being added and video shown. This still seems a little bit long, but is possible given the 3G network and in a Group Room where a keyframe might not be available from the remote Participant immediately.

How can I detect actual timing that the remote participant's video is shown on remoteView?
Or, Is there any ideas to prevent showing blank view until actual video is shown, or remove time lag?

The track added event indicates that the media has been negotiated, but not that the first frame has arrived. My suggestion would be to implement TVIVideoViewDelegate and respond to videoViewDidReceiveData:. This event will be fired once when the first frame arrives at the renderer. You could consider hiding your view until that point so you can guarantee that you will see useful content once its visible.

extension ViewController: TVIVideoViewDelegate {
    func videoViewDidReceiveData(_ view: TVIVideoView) {
        // This method is called just once. Make the view visible or animate a transition onscreen here.
        view.isHidden = false
    }
}

TVIVideoViewDelegate docs

Regards,
Chris Eagleston

@ceaglest ceaglest added the question label Dec 14, 2017

@mitolog

This comment has been minimized.

Copy link

mitolog commented Dec 14, 2017

@ceaglest
Hi, Thanks for the quick response. I didn't know TVIVideoViewDelegate. I'll try this. Thanks.
Then about 30 sec elapsed time, It's appreciated if you can dig down.

@piyushtank

This comment has been minimized.

Copy link
Collaborator

piyushtank commented Dec 14, 2017

@mitolog What you have shared is a secret key. Can you post Room-SID? Room-Sid can be retrieved by room.sid(). https://media.twiliocdn.com/sdk/ios/video/releases/1.3.8/docs/Classes/TVIRoom.html#//api/name/sid

Can you please post room-sid for the call where you are observing the issue?

@mitolog

This comment has been minimized.

Copy link

mitolog commented Dec 14, 2017

@ptankTwilio oops.. thanks. When I run into long elapsed time issue, I'll post here again, cause I couldn't identify which one. I'll try workaround as ceaglest suggested above, at this time.

As my reference, Is it also the case with audio? I mean the time between addedAudioTrack called and actual audio reached.

@ceaglest

This comment has been minimized.

Copy link
Member

ceaglest commented Dec 14, 2017

As my reference, Is it also the case with audio? I mean the time between addedAudioTrack called and actual audio reached.

Yes the audioTrackAdded event is very similar, in that negotiation has completed but media is not flowing.

@mitolog

This comment has been minimized.

Copy link

mitolog commented Dec 14, 2017

Ok. I assume, If there are both video/audio track, it's same timing as videoViewDidReceiveData, cause these should be synchronized. But how about only audio track is sent from remote? Can I detect audio track is actually reached? cause I think I have to show loading view or some info to the user until actual media has reached. If it's a subtle time, I can skip it though...

@ceaglest

This comment has been minimized.

Copy link
Member

ceaglest commented Dec 14, 2017

@mitolog,

The timing should be very similar if your app always shares both Audio / Video using TVIConnectOptions. Waiting for the first video frame to be received should be close enough to when audio first arrives. We do have TVIAudioSink which allows you to receive audio samples, but you may get silent audio from the mixer before receiving real encoded audio frames so I don't think the same heuristic can be used there.

Regards,
Chris

@mitolog

This comment has been minimized.

Copy link

mitolog commented Dec 14, 2017

I got it! Thanks @ceaglest.

@mitolog

This comment has been minimized.

Copy link

mitolog commented Dec 16, 2017

@ceaglest
Hi, I encountered long time lag issue with room:
RM4330a7d870efd0a7677a19324feea558

it lasted about 30 sec or so. At that time, I set TVIVideoConstraints as below:

        let videoConstraints = TVIVideoConstraints { (constraints) in
            constraints.maxSize = TVIVideoConstraintsSize480x360
            constraints.minSize = TVIVideoConstraintsSizeNone
            constraints.maxFrameRate = TVIVideoConstraintsFrameRate10
            constraints.minFrameRate = TVIVideoConstraintsFrameRate10
        }

where 2 participants are in the room:
iPhone6s-plus: 3G band
iPhone5: wifi with tethering of 3G

@sathishvgs

This comment has been minimized.

Copy link

sathishvgs commented Jan 6, 2018

Hi, @mitolog @ceaglest I'm facing the same time lag issue. I tried this but it's not working for me.

    func videoViewDidReceiveData(_ view: TVIVideoView) {
        // This method is called just once. Make the view visible or animate a transition onscreen here.
        view.isHidden = false
    }
}

@ceaglest

This comment has been minimized.

Copy link
Member

ceaglest commented Jan 13, 2018

Hi,

@sathishvgs I followed up in the issue you opened.

@mitolog Sorry for not following up, this got lost in the shuffle over the holiday break.

Over the past month we've made a number of changes to the congestion control algorithm used in our Group Rooms media servers. You may get better results today especially if both devices are on 3G.

If you only need 2 Participants and don't require features like recording you may have a better experience using Peer-to-Peer Rooms. There is a bit more info about the distinction between the two here.

I took a look at your traffic, and don't see much media flowing between Participants. Also, I can see that both of your Participants are connecting with paused video Tracks. You might want to consider enabling the video tracks before connecting to the Room to improve call setup time.

If you're interested here is a summary of the Traffic.

Summary for room name: C532664A-7DE8-40B1-9ADA-7552FB38111A | sid: RM4330a7d870efd0a7677a19324feea558 | account sid: ACxxx
Topology: group | Adhoc: True | TURN: True | Recording: False | Video Codecs: VP8 | Media Region: us1
Participants:
P1: name: nbJiSkr6u3g0jP62Ue59ZDTBULA3 | sid: PA28622ba3eca130642ba4dc246cd78813 | call sid: CA3b3323af3160bd898198c7577f2bb892 | sdp format: Plan B
P2: name: VYrgF1cYstTnDnsGV4igd449zWS2 | sid: PAe76cae5dc409f86c78f4fbf8f3e56d92 | call sid: CA5917a116ff40a65324ce8068a076a8b5 | sdp format: Plan B
2017-12-16T19:13:42.425Z: [add_labeled_room]
2017-12-16T19:13:42.428Z: P1 sent connect with audio + video (paused) tracks
2017-12-16T19:13:42.428Z: P1 sent offer for peer connection id d1696612-895f-eaf0-ef96-cefaef881c76 revision: 1
2017-12-16T19:13:42.428Z: [room_created]
2017-12-16T19:13:42.957Z: P1 received connected
2017-12-16T19:13:42.957Z: P1 received answer for peer connection id d1696612-895f-eaf0-ef96-cefaef881c76 revision: 1
2017-12-16T19:13:42.957Z: P1 published audio track name: 882b1d59ff12c45b81b132cd7718539cbdc4 sid: MTdd653a686bfdc56e698a50fa6bcbd658 revision: 1
2017-12-16T19:13:42.957Z: P1 published video track name: 48366dcb56084bd9c2c8a716dae0e9a9ab8b sid: MT0238ada8568330b66c55c5ec364da43e revision: 1
2017-12-16T19:13:43.714Z: P1 sent ice complete for peer connection id d1696612-895f-eaf0-ef96-cefaef881c76 with 2 ice candidates [ 1 host, 1 srflx, 0 relay ] revision: 3
2017-12-16T19:13:45.579Z: P2 sent connect with audio + video (paused) tracks
2017-12-16T19:13:45.579Z: P2 sent offer for peer connection id 86b09409-0f2f-ab31-f4cc-e916fd19976e revision: 1
2017-12-16T19:13:45.598Z: P2 received connected
2017-12-16T19:13:45.598Z: P2 received answer for peer connection id 86b09409-0f2f-ab31-f4cc-e916fd19976e revision: 1
2017-12-16T19:13:45.598Z: P2 published audio track name: 527cd23794ff12fd161e1603c6ee6d79c5ca sid: MTd42fac32685b7a11673c94120f699c59 revision: 1
2017-12-16T19:13:45.598Z: P2 published video track name: e8ab4ec48811c57508fc2e773a61ef780a68 sid: MT225cbedcb371fd4ce8ce4fa14ed3da09 revision: 1
2017-12-16T19:13:45.598Z: P1 received offer for peer connection id d1696612-895f-eaf0-ef96-cefaef881c76 revision: 2
2017-12-16T19:13:45.598Z: P1 subscribed to track sid: MTd42fac32685b7a11673c94120f699c59 revision: 1
2017-12-16T19:13:45.598Z: P1 subscribed to track sid: MT225cbedcb371fd4ce8ce4fa14ed3da09 revision: 1
2017-12-16T19:13:45.599Z: P2 received offer for peer connection id 86b09409-0f2f-ab31-f4cc-e916fd19976e revision: 2
2017-12-16T19:13:45.599Z: P2 subscribed to track sid: MT0238ada8568330b66c55c5ec364da43e revision: 1
2017-12-16T19:13:45.599Z: P2 subscribed to track sid: MTdd653a686bfdc56e698a50fa6bcbd658 revision: 1
2017-12-16T19:13:46.564Z: P2 sent answer for peer connection id 86b09409-0f2f-ab31-f4cc-e916fd19976e revision: 2
2017-12-16T19:13:46.631Z: P1 sent answer for peer connection id d1696612-895f-eaf0-ef96-cefaef881c76 revision: 2
2017-12-16T19:13:46.653Z: P2 sent ice complete for peer connection id 86b09409-0f2f-ab31-f4cc-e916fd19976e with 2 ice candidates [ 1 host, 1 srflx, 0 relay ] revision: 3
2017-12-16T19:13:46.845Z: [ice_connection_state_change] for P1 with state CONNECTED
2017-12-16T19:13:49.772Z: [ice_connection_state_change] for P2 with state CONNECTED
2017-12-16T19:13:59.277Z: [ice_connection_state_change] for P2 with state COMPLETED
2017-12-16T19:13:59.349Z: [ice_connection_state_change] for P1 with state COMPLETED
2017-12-16T19:14:30.061Z: P1 sent disconnect
2017-12-16T19:14:30.064Z: P1 received disconnected
2017-12-16T19:14:30.068Z: [group_billing_event] for P1 with bytes sent 81779, bytes received 225103, recording time in secs 0
2017-12-16T19:14:30.068Z: P2 received offer for peer connection id 86b09409-0f2f-ab31-f4cc-e916fd19976e revision: 3
2017-12-16T19:14:30.532Z: P2 sent answer for peer connection id 86b09409-0f2f-ab31-f4cc-e916fd19976e revision: 3
2017-12-16T19:14:30.538Z: [ice_connection_state_change] for P2 with state CONNECTED
2017-12-16T19:14:31.555Z: P2 sent disconnect
2017-12-16T19:14:31.557Z: P2 received disconnected
2017-12-16T19:14:31.559Z: [group_billing_event] for P2 with bytes sent 80578, bytes received 80909, recording time in secs 0

Let me know if you're still experiencing this issue, and I'll do anything I can to help.

Best,
Chris Eagleston

@mitolog

This comment has been minimized.

Copy link

mitolog commented Jan 17, 2018

@ceaglest Thanks for the reply and investigation.

P1 sent connect with audio + video (paused) tracks

I'm now tried in p2p mode, but no logs are there describing paused on TVILogLevel.all.
For possible future issue, in what situation do you suspect paused status happen?

I've used TVILocalVideoTrack init as below, where I set enabled to true.

        let videoConstraints = TVIVideoConstraints { (constraints) in
            constraints.maxSize = TVIVideoConstraintsSize1280x960
            constraints.minSize = TVIVideoConstraintsSize640x480
            constraints.maxFrameRate = TVIVideoConstraintsFrameRate30
            constraints.minFrameRate = TVIVideoConstraintsFrameRate15
        }
        localVideoTrack = TVILocalVideoTrack(capturer: camera!,
                                             enabled: true,
                                             constraints: videoConstraints)

If the enabled is not relevant, I suspect that video camera permission alert, which is shown when it's first call. but not sure...

@ceaglest

This comment has been minimized.

Copy link
Member

ceaglest commented Jan 24, 2018

Hi @mitolog,

My apologies for the late response.

The only way that a track should be enabled or disabled after creation is by toggling the Track's enabled property. Are you doing this in your app?

It would be helpful if you could try the QuickStart app and let us know if the issue is reproducible there.

Best,
Chris

@mitolog

This comment has been minimized.

Copy link

mitolog commented Jan 25, 2018

@ceaglest
Thanks for the reply!

Lately, after moving to p2p mode, the connection time has much faster. I haven't seen any disable state on log. Then I didn't set enabled property other than preparing VideoTrak, so it may be my fault during twilio experiment.

Now it seems we can close this issue. Thanks a lot again!

@ceaglest

This comment has been minimized.

Copy link
Member

ceaglest commented Jan 25, 2018

I'm glad to hear that you're having good results with p2p rooms. We are still actively improving our group rooms media servers during the beta period, so there may be some changes to how quickly keyframes are requested and video starts in the future.

Best of luck with your project!

@ceaglest ceaglest closed this Jan 25, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment