Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Using Twilio Android SDK from an Android Service #73

Closed
FranziscaZarrai opened this issue Mar 7, 2017 · 18 comments

Comments

@FranziscaZarrai
Copy link

commented Mar 7, 2017

We're experiencing some troubles with the Twilio Video-SDK when using it based on a Service. The use-case is as follows:

The user gets authenticated with Twilio (JWT token is provided by our authentication server - links to it's source code at the end of this post), joins his own room (each user has one room based on his username and some other identifiers) then moves the App into the background. A service listens for room related events and notifies the user when someone joins his room (i.e. tries to call him by picking him from a list of people that are registered). Then the callee clicks on the notification and the call dialog opens. Note, that the callee was already in his own room before opening the call dialog, since he was notified of the call attempt, when someone else joined his room.

At this point the following issues come up (more or less randomly):

In onConnected when iterating over the participants it happens, that the same participants exists twice in the list and the other participant does not exist at all when calling room.getParticipants().entrySet (where aRoom = an instance of com.twilio.video.Room)

The first call between two devices is usually successful, subsequent calls may exhibit above behavior in about 30%-50% of the cases. This was found out by simply printing each participant returned by room.getParticipants().entrySet. Another issue was that in the android video quickstart only the first participant in the list found in onConnected(Room room)is passed on to the private void addParticipant(Participant participant) method. However it also seems the first participant is not always the caller, but can be the callee as well, in which case the callee would call private void addParticipant(Participant participant) passing itself.

We have opensourced our mobile client using the twilio SDK as well as our authentication server, which is based on the original twilio-provided Apache Spark authentication server sample (for Java). You can find bot of them here:

Twilio Android Project
Spark based authentication server

Please let us know if we can provide any further details.

@aaalaniz

This comment has been minimized.

Copy link
Member

commented Mar 7, 2017

Thanks for writing in. Allow me to look through the application source provided and see what I can find. Thanks for the detailed breakdown of the issues you are seeing. I'lll provide an update soon.

@aaalaniz aaalaniz self-assigned this Mar 7, 2017

@aaalaniz

This comment has been minimized.

Copy link
Member

commented Mar 7, 2017

Duplicate Participant Issue
When you see duplicate Participant s do they have matching SIDs and identities? You can invoke Participant#getSid() and Participant#getIdentity() to check. This could be an issue of "ghost" participants that we sometimes see due to some backend implementation details. I will see if this could be a client side issue.

Absent Participant
When you say a participant is not present in Room#getParticipants() are you saying that a participant successfully connects to a Room and that is not being reflected to other participants in a Room? I'm trying to understand the notion of "calling" you describe. We do not have this concept with new Rooms API. If a participant has not finished connecting to a Room they will not show up in Room#getParticipants. Once a participant does successfully connect to a Room all other participants in the Room will receive a callback Room.Listener#onParticipantConnected. At that point if you invoke Room#getParticipants then the new participant should be present.

Quickstart Findings
We only add the first participant because the quickstart is only meant to show a simple 1:1 video example. Note that when you connect to a Room a Participant represents other clients that have connected to that Room but NOT yourself. So if you connect to a Room of which you are the only participant, getParticipants should return an empty Map. The LocalParticipant object is meant to represent your presence in a Room. You can retrieve your LocalParticipant once you have a connected to a Room using Room#getLocalParticipant.

I hope this makes sense. Let me know if I can clarify anything.

Thanks!

@FranziscaZarrai

This comment has been minimized.

Copy link
Author

commented Mar 10, 2017

@aaalaniz i'll get to responding to the open questions this weekend. That aside I saw you mentioned an alternative way of handling one-to-one-calls here: #74 (comment) by using google cloud messaging. Using a background running service (as we do) seems a little easier, since it does not involve yet another endpoint (google cloud messaing) but it makes me wonder if being permanently connected to one room (via the service) could be an issue since you didn't suggest using a service as an alternative approach?

@aaalaniz

This comment has been minimized.

Copy link
Member

commented Mar 10, 2017

Hey @FranziscaZarrai

I do not recommend being permanently connected to a Room for two reasons:

  1. Holding a long standing connection will cause unnecessary battery drain.
  2. We have a 4 hour cap on a Participant being connected to a Room. Even if we lifted this cap, I still do not suggest doing this because of reason one.
@FranziscaZarrai

This comment has been minimized.

Copy link
Author

commented Mar 11, 2017

@aaalaniz Ok so regarding the question Duplicate Participant Issue.

Say one phone with the Identitiy test-accountFiigp (we use roomnames == user identity) connects to a room of another person (who was already in the room) Identitiy test-accountLkqoa thus sending a "call request" to the participant with identity test-accountLkqoa. Now if i list the participants on the phone with Identitiy test-accountLkqoa (the one who received the call and was thus already in the room) I get:

// That's the room both are connected to
CallActivity: Listing participants for room test-accountLkqoa
// On the one "receiving the call" (i.e. who was already in the room) we get distinct identities and SIDs
// when listing participants:
CallActivity: Identity: test-accountLkqoa - SID: PAbc1197c769c6a3d944ca61cc1303eb73
CallActivity: Identity: test-accountFiigp - SID: PA6d7fc780e29470317bb7850f340ed684

Based on the above i get both: My own identity (test-accountLkqoa) and the identity of the other person in the room who just joined (test-accountFiigp).

// That's the room both are connected to
CallActivity: Listing participants for room test-accountLkqoa
// However on the device that connected to the other phone, i.e. joined the room we get the same identity twice (identity of the room we joined/person we called), albeit with different SIDs:
CallActivity: Identity: test-accountLkqoa - SID: PA3dde7e27287354d4d121c8cbe47d83be
CallActivity: Identity: test-accountLkqoa - SID: PAbc1197c769c6a3d944ca61cc1303eb73

Based on the above I only get the identity of the other person in the room (not my own) but I get it twice.

The following snippet from our project produced the above log output (called from onConnected of the room listener)

https://gist.github.com/e107c186f24063756d031f8c4080c3a5

The above example did not work, i.e. I was not able to get the remote video stream on either device.

Regarding the question for Absent Participant: At the time of writing i was not aware, that our own participant should not even be in that list (which makes sense). Based on my above findings this works for the one who is joining a new room, but we still got the issue of duplicate participants there (see log output above). However in the first case - where the person was already in a room and someone else joined it looks like we ourselves are listed in that room as well. Based on your comment regarding who shows up in the participant list, your sample code in the Quickstart Findings with adding the first participant in the list makes sense. However since I experienced the issue of having myself and the other person in the room it could actually happen, that I try to add myself if I just take the first one (that would have been the case in my first log output further up).

@aaalaniz

This comment has been minimized.

Copy link
Member

commented Mar 14, 2017

Hey @FranziscaZarrai

Thanks for investigating this a bit further. I'm pretty sure this is a bug. I'm going to test with our video application and see if I can reproduce and root cause the problem. I'll update this issue for any new findings.

Thanks again!

@aaalaniz

This comment has been minimized.

Copy link
Member

commented Mar 22, 2017

Hi @FranziscaZarrai

Can you try the 1.0.0-beta13 release and see if you are still encountering duplicate participants? We made a lot of changes in our core library and I'm wondering if some of these issues may have been resolved.

Thanks!

@FranziscaZarrai

This comment has been minimized.

Copy link
Author

commented Mar 23, 2017

@aaalaniz Thank you very much for the update! Will verify it asap and get back to you.

@dreamerns

This comment has been minimized.

Copy link
Contributor

commented Mar 29, 2017

@FranziscaZarrai I've looked into this ticket and the only way I was able to reproduce it is if one of the participants (test-accountLkqoa in you example) abruptly left the room and came back. We do have a timeout on our backend (#80) that can take up to 120 seconds to detect that participant has left the room. During that time participant appears to be still in the room (ghost participant). What happens in your case is that test-accountLkqoa abruptly leaves the room (without calling disconnect) and then connects to same room again. This is why test-accountLkqoa can see itself and other participant in the list, and test-accountFiigp can see 2x test-accountLkqoa (real one and ghost). You might want to check if your service might have crashed and restarted or similar event happened that can trigger test-accountLkqoa to re-connects to room withour properly leaving it. Also, what @aaalaniz already mentioned, being permanently connected to the room is not recommended approach. Hope this helps, let me know if you were able to fix this problem.

Thanks,
Nikola

@FranziscaZarrai

This comment has been minimized.

Copy link
Author

commented Mar 31, 2017

@dreamerns Thank you for the additional input! I'll check this out as well and see if this is what's going on with our service.

On that note: What if a for instance the internet disconnects and then before the 120 seconds are over, the participants reconnects? I'd say that is a possible scenario - how would you suggest handling that? An artificial timeout on the client side? Because 2 minutes is a lot :) .

@dreamerns

This comment has been minimized.

Copy link
Contributor

commented Mar 31, 2017

That is also possible scenario. If one of participant looses connection, it would remain in room as "ghost" for next 2 minutes. Unfortunately, we don't have good solution right now for this problem, but our backend team is working on it and hopefully there will be a fix soon. As workaround you could put some additional timeout on client side that would cut down time. Something like a client ping.

Thanks,
Nikola

@FranziscaZarrai

This comment has been minimized.

Copy link
Author

commented Apr 1, 2017

@dreamerns @aaalaniz

I gradually updated - starting with 1.0.0-beta11, then 1.0.0-beta13 and then 1.0.0-beta15. The relevant ones being the latter two:

1.0.0-beta13 - works great
1.0.0-beta15 - works sometimes, crashes on same occasions and sometimes doesn't add the video stream

Here is one of the crashes from beta15 (Sony Xperia Z5 Compact calls Google Pixel - the crash occurs on the Pixel - both run Android Version 7):

04-01 02:09:27.644 653-731/? E/ANDR-PERF-OPTSHANDLER: Warning: Resource [2, 0] not supported for core 1. Instead use resource for core 0
04-01 02:09:27.644 653-731/? E/ANDR-PERF-RESOURCEQS: Failed to apply optimization [2, 2, 0]
04-01 02:09:28.706 660-660/? E/OMX-VDEC-1080P: Extension: OMX.google.android.index.storeANWBufferInMetadata not implemented
04-01 02:09:28.723 660-11192/? E/OMX-VDEC-1080P: Extension: OMX.google.android.index.storeANWBufferInMetadata not implemented
04-01 02:09:28.724 660-11192/? E/OMX-VDEC-1080P: Extension: OMX.google.android.index.configureVideoTunnelMode not implemented
04-01 02:09:28.724 660-11192/? E/OMX-VDEC-1080P: Extension: OMX.google.android.index.useAndroidNativeBuffer is supported
04-01 02:09:28.724 660-1924/? E/OMX-VDEC-1080P: Extension: OMX.google.android.index.describeHDRStaticInfo not implemented
04-01 02:09:28.725 660-27022/? E/OMX-VDEC-1080P: get_parameter: unknown param 6f600007
04-01 02:09:28.725 660-27022/? E/OMXNodeInstance: getParameter(29405dd:qcom.decoder.vp8, ParamVideoAndroidVp8Encoder(0x6f600007)) ERROR: UnsupportedIndex(0x8000101a)
04-01 02:09:28.728 660-791/? E/OMX-VDEC-1080P: Does not handle dataspace request
04-01 02:09:28.728 660-791/? E/OMXNodeInstance: getConfig(29405dd:qcom.decoder.vp8, ??(0x7f00005e)) ERROR: UnsupportedSetting(0x80001019)
04-01 02:09:28.786 660-1920/? E/OMX-VDEC-1080P: Does not handle dataspace request
04-01 02:09:28.786 660-1920/? E/OMXNodeInstance: getConfig(29405dd:qcom.decoder.vp8, ??(0x7f00005e)) ERROR: UnsupportedSetting(0x80001019)
04-01 02:09:28.821 1074-20268/? E/ActivityManager: applyOptionsLocked: Unknown animationType=0
04-01 02:09:28.976 656-656/? E/mm-camera: <CPP   ><ERROR> 2208: cpp_module_notify_add_stream: failed, CPP current version can not support rotation
04-01 02:09:29.119 23980-23980/com.myproject E/LocalMedia: Applying VideoConstraints closest to 640x480@30 FPS.
04-01 02:09:29.147 23980-25882/com.myproject E/rtc: #
                                                                    # Fatal error in ../../webrtc/sdk/android/src/jni/peerconnection_jni.cc, line 835
                                                                    # last system error: 0
                                                                    # Check failed: !jni()->ExceptionCheck()
                                                                    # 
                                                                    #
                                                                    
                                                                    
                                                                    --------- beginning of crash
04-01 02:09:29.148 23980-25882/com.myproject A/libc: Fatal signal 6 (SIGABRT), code -6 in tid 25882 (IncomingVideoSt)
                                                                     
                                                                     [ 04-01 02:09:29.148   555:  555 W/         ]
                                                                     debuggerd: handling request: pid=23980 uid=10195 gid=10195 tid=25882
04-01 02:09:29.168 25987-25987/? A/DEBUG: *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
04-01 02:09:29.169 25987-25987/? A/DEBUG: Build fingerprint: 'google/sailfish/sailfish:7.1.1/NOF27B/3687361:user/release-keys'
04-01 02:09:29.169 25987-25987/? A/DEBUG: Revision: '0'
04-01 02:09:29.169 25987-25987/? A/DEBUG: ABI: 'arm'
04-01 02:09:29.169 25987-25987/? A/DEBUG: pid: 23980, tid: 25882, name: IncomingVideoSt  >>> com.myproject <<<
04-01 02:09:29.169 25987-25987/? A/DEBUG: signal 6 (SIGABRT), code -6 (SI_TKILL), fault addr --------
04-01 02:09:29.169 25987-25987/? A/DEBUG:     r0 00000000  r1 0000651a  r2 00000006  r3 00000008
04-01 02:09:29.169 25987-25987/? A/DEBUG:     r4 c0201978  r5 00000006  r6 c0201920  r7 0000010c
04-01 02:09:29.169 25987-25987/? A/DEBUG:     r8 0000000a  r9 ccf3ddc0  sl 00200001  fp 000001e0
04-01 02:09:29.169 25987-25987/? A/DEBUG:     ip 00000000  sp c02014e8  lr eb2195c7  pc eb21be30  cpsr 600f0010
04-01 02:09:29.175 25987-25987/? A/DEBUG: backtrace:
04-01 02:09:29.176 25987-25987/? A/DEBUG:     #00 pc 00049e30  /system/lib/libc.so (tgkill+12)
04-01 02:09:29.176 25987-25987/? A/DEBUG:     #01 pc 000475c3  /system/lib/libc.so (pthread_kill+34)
04-01 02:09:29.176 25987-25987/? A/DEBUG:     #02 pc 0001d635  /system/lib/libc.so (raise+10)
04-01 02:09:29.176 25987-25987/? A/DEBUG:     #03 pc 00019181  /system/lib/libc.so (__libc_android_abort+34)
04-01 02:09:29.176 25987-25987/? A/DEBUG:     #04 pc 00017048  /system/lib/libc.so (abort+4)
04-01 02:09:29.176 25987-25987/? A/DEBUG:     #05 pc 002267c3  /data/app/com.myproject-2/lib/arm/libjingle_peerconnection_so.so
04-01 02:09:29.176 25987-25987/? A/DEBUG:     #06 pc 0051c4cb  /data/app/com.myproject-2/lib/arm/libjingle_peerconnection_so.so
04-01 02:09:29.176 25987-25987/? A/DEBUG:     #07 pc 0023eccf  /data/app/com.myproject-2/lib/arm/libjingle_peerconnection_so.so
04-01 02:09:29.176 25987-25987/? A/DEBUG:     #08 pc 002da28d  /data/app/com.myproject-2/lib/arm/libjingle_peerconnection_so.so
04-01 02:09:29.176 25987-25987/? A/DEBUG:     #09 pc 0032f6d3  /data/app/com.myproject-2/lib/arm/libjingle_peerconnection_so.so
04-01 02:09:29.176 25987-25987/? A/DEBUG:     #10 pc 0039bf45  /data/app/com.myproject-2/lib/arm/libjingle_peerconnection_so.so
04-01 02:09:29.176 25987-25987/? A/DEBUG:     #11 pc 00227265  /data/app/com.myproject-2/lib/arm/libjingle_peerconnection_so.so
04-01 02:09:29.176 25987-25987/? A/DEBUG:     #12 pc 00227279  /data/app/com.myproject-2/lib/arm/libjingle_peerconnection_so.so
04-01 02:09:29.176 25987-25987/? A/DEBUG:     #13 pc 00047093  /system/lib/libc.so (_ZL15__pthread_startPv+22)
04-01 02:09:29.176 25987-25987/? A/DEBUG:     #14 pc 00019bdd  /system/lib/libc.so (__start_thread+6)
04-01 02:09:29.228 656-17081/? E/mm-camera: <CPP   ><ERROR> 2208: cpp_module_notify_add_stream: failed, CPP current version can not support rotation
04-01 02:09:30.062 1074-26055/? E/JavaBinder: !!! FAILED BINDER TRANSACTION !!!  (parcel size = 112)
04-01 02:09:30.067 1074-26055/? E/JavaBinder: !!! FAILED BINDER TRANSACTION !!!  (parcel size = 108)
04-01 02:09:30.070 1074-26055/? E/JavaBinder: !!! FAILED BINDER TRANSACTION !!!  (parcel size = 2636)
04-01 02:09:30.075 1074-26055/? E/JavaBinder: !!! FAILED BINDER TRANSACTION !!!  (parcel size = 56)
04-01 02:09:30.084 660-791/? E/OMXNodeInstance: !!! Observer died. Quickly, do something, ... anything...
04-01 02:09:30.087 656-6656/? E/Camera2Client: notifyError: Error condition 0 reported by HAL, requestId -1
@ra992

This comment has been minimized.

Copy link

commented Apr 2, 2017

Hi,

I have downloaded the application "Twilio Android Project, Spark based authentication server" from links mentioned above. But its not working as expected:

  1. The list of participants are not displayed on the web page
  2. List of participants are displayed on mobile, but when call to any selected participant, call dialog opens, but the Callee didn't get any call.

So, is there anything which I missed to configure?

Please advise.
Thanks.

@FranziscaZarrai

This comment has been minimized.

Copy link
Author

commented Apr 2, 2017

@ra992 This is not the place for that question :) if you have questions regarding that project, please open an issue on the corresponding project and we'll take a look. We'll need more details though and log outputs documenting your issue. However I will answer your questions - but for further inquiries, please open an issue on the corresponding project:

  1. There is no webpage to display participants. The webinterface you see when opening the sparky URL does nothing in the sparky project - we removed the functionality afaik.
  2. If you see the list of participants on both mobiles your configuration is likely correct. However we're using an older Twilio SDK Version on the android project you're referring to - we'll upgrade it shortly. Please provide logs for the caller and callee when opening an issue on the corresponding project.
@aaalaniz

This comment has been minimized.

Copy link
Member

commented Apr 28, 2017

Hey @FranziscaZarrai

Can you try updating to our 1.0.0 release? Please let us know if you are still experiencing crashes in the latest release and we will diagnose.

Thanks

@aaalaniz

This comment has been minimized.

Copy link
Member

commented May 10, 2017

I'm going to close this issue. @FranziscaZarrai Please open a new issue if you encounter any new problems.

Thanks!

@aaalaniz aaalaniz closed this May 10, 2017

@YashAjabiya

This comment has been minimized.

Copy link

commented May 25, 2018

Hello the given URL not found : https://github.com/coresystemsFSM/android-twilio-video-with-service

I want to develop one to one user video calling app using twilio. can you help me ?

Thanks,

@aaalaniz

This comment has been minimized.

Copy link
Member

commented May 29, 2018

Hey @YashAjabiya

I recommend following the project README. If you require further assistance please open a separate issue with details on what you need help building.

Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
5 participants
You can’t perform that action at this time.