Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Voice NDK crashes Java_com_twilio_voice_JniUtils_nativeJavaUtf16StringToStdString #572

Closed
paco-spr opened this issue Nov 1, 2023 · 17 comments

Comments

@paco-spr
Copy link

paco-spr commented Nov 1, 2023

Description

Voice NDK crashes com_twilio_voice_JniUtils_nativeJavaUtf16StringToStdString

Steps to Reproduce

Unknown, these issues are being surfaced to us via crashlytics.

Code

Proguard rules being used:

# Twilio Programmable Voice
-keep class com.twilio.** { *; }
-keep class tvo.webrtc.** { *; }
-dontwarn tvo.webrtc.**
-keep class com.twilio.voice.** { *; }
-keepattributes InnerClasses

Expected Behavior

The app does not crash

Actual Behavior

The app crashes

Reproduces How Often

About 1-2% of our users have experienced one of these 2 crashes.

Twilio Call SID(s)

Unknown.

Logs

Java_com_twilio_voice_JniUtils_nativeJavaUtf16StringToStdString

null pointer dereference: SIGSEGV  0x0000000000000048
#00 pc 0xae75a libtwilio_voice_android_so.so (Java_com_twilio_voice_JniUtils_nativeJavaUtf16StringToStdString) (BuildId: 3f1eaa46e50eae77373842563e1080e5e07b180a)
#01 pc 0x5aa12e libtwilio_voice_android_so.so (__udivdi3) (BuildId: 3f1eaa46e50eae77373842563e1080e5e07b180a)
#02 pc 0x5aa13a libtwilio_voice_android_so.so (__udivdi3) (BuildId: 3f1eaa46e50eae77373842563e1080e5e07b180a)
#03 pc 0x5aa12e libtwilio_voice_android_so.so (__udivdi3) (BuildId: 3f1eaa46e50eae77373842563e1080e5e07b180a)
#04 pc 0xcc4be libc.so (BuildId: f4abefa5427b93525ac009af9345d160)
#05 pc 0x690782 libtwilio_voice_android_so.so (__udivdi3) (BuildId: 3f1eaa46e50eae77373842563e1080e5e07b180a)
#06 pc 0x663c5a libtwilio_voice_android_so.so (__udivdi3) (BuildId: 3f1eaa46e50eae77373842563e1080e5e07b180a)
#07 pc 0x69075a libtwilio_voice_android_so.so (__udivdi3) (BuildId: 3f1eaa46e50eae77373842563e1080e5e07b180a)
#08 pc 0xaea73 libtwilio_voice_android_so.so (Java_com_twilio_voice_JniUtils_nativeJavaUtf16StringToStdString) (BuildId: 3f1eaa46e50eae77373842563e1080e5e07b180a)
#09 pc 0x5aa283 libtwilio_voice_android_so.so (__udivdi3) (BuildId: 3f1eaa46e50eae77373842563e1080e5e07b180a)
#10 pc 0x5aa2fa libtwilio_voice_android_so.so (__udivdi3) (BuildId: 3f1eaa46e50eae77373842563e1080e5e07b180a)
#11 pc 0x5aa283 libtwilio_voice_android_so.so (__udivdi3) (BuildId: 3f1eaa46e50eae77373842563e1080e5e07b180a)
#12 pc 0xffffe libharfbuzz_ng.so (BuildId: 97ac946b99d008fd0230b003ed720669)
#13 pc 0xcc4be libc.so (BuildId: f4abefa5427b93525ac009af9345d160)
#14 pc 0x663c46 libtwilio_voice_android_so.so (__udivdi3) (BuildId: 3f1eaa46e50eae77373842563e1080e5e07b180a)
#15 pc 0x8b105 libtwilio_voice_android_so.so (BuildId: 3f1eaa46e50eae77373842563e1080e5e07b180a)
#16 pc 0xaec97 libtwilio_voice_android_so.so (Java_com_twilio_voice_JniUtils_nativeJavaUtf16StringToStdString) (BuildId: 3f1eaa46e50eae77373842563e1080e5e07b180a)
#17 pc 0xa29dd libtwilio_voice_android_so.so (Java_com_twilio_voice_JniUtils_nativeJavaUtf16StringToStdString) (BuildId: 3f1eaa46e50eae77373842563e1080e5e07b180a)
#18 pc 0x504087 libtwilio_voice_android_so.so (std::__ndk1::__shared_weak_count::__add_weak()) (BuildId: 3f1eaa46e50eae77373842563e1080e5e07b180a)
#19 pc 0x9824d libtwilio_voice_android_so.so (Java_com_twilio_voice_JniUtils_nativeJavaUtf16StringToStdString) (BuildId: 3f1eaa46e50eae77373842563e1080e5e07b180a)
#20 pc 0xb0e67 libtwilio_voice_android_so.so (Java_com_twilio_voice_JniUtils_nativeJavaUtf16StringToStdString) (BuildId: 3f1eaa46e50eae77373842563e1080e5e07b180a)
#21 pc 0x5aac86 libtwilio_voice_android_so.so (__udivdi3) (BuildId: 3f1eaa46e50eae77373842563e1080e5e07b180a)
#22 pc 0x5aac90 libtwilio_voice_android_so.so (__udivdi3) (BuildId: 3f1eaa46e50eae77373842563e1080e5e07b180a)
#23 pc 0x5aac86 libtwilio_voice_android_so.so (__udivdi3) (BuildId: 3f1eaa46e50eae77373842563e1080e5e07b180a)
#24 pc 0xcc4be libc.so (BuildId: f4abefa5427b93525ac009af9345d160)
#25 pc 0xb1051 libtwilio_voice_android_so.so (Java_com_twilio_voice_JniUtils_nativeJavaUtf16StringToStdString) (BuildId: 3f1eaa46e50eae77373842563e1080e5e07b180a)
#26 pc 0x504087 libtwilio_voice_android_so.so (std::__ndk1::__shared_weak_count::__add_weak()) (BuildId: 3f1eaa46e50eae77373842563e1080e5e07b180a)
#27 pc 0x98561 libtwilio_voice_android_so.so (Java_com_twilio_voice_JniUtils_nativeJavaUtf16StringToStdString) (BuildId: 3f1eaa46e50eae77373842563e1080e5e07b180a)
#28 pc 0xc27ab libtwilio_voice_android_so.so (Java_com_twilio_voice_JniUtils_nativeJavaUtf16StringToStdString) (BuildId: 3f1eaa46e50eae77373842563e1080e5e07b180a)
#29 pc 0x514525 libtwilio_voice_android_so.so (__gnu_ldivmod_helper) (BuildId: 3f1eaa46e50eae77373842563e1080e5e07b180a)
#30 pc 0x22a223 libtwilio_voice_android_so.so (Java_com_twilio_voice_JniUtils_nativeJavaUtf16StringToStdString) (BuildId: 3f1eaa46e50eae77373842563e1080e5e07b180a)
#31 pc 0x2295c7 libtwilio_voice_android_so.so (Java_com_twilio_voice_JniUtils_nativeJavaUtf16StringToStdString) (BuildId: 3f1eaa46e50eae77373842563e1080e5e07b180a)
#32 pc 0x229117 libtwilio_voice_android_so.so (Java_com_twilio_voice_JniUtils_nativeJavaUtf16StringToStdString) (BuildId: 3f1eaa46e50eae77373842563e1080e5e07b180a)
#33 pc 0x229117 libtwilio_voice_android_so.so (Java_com_twilio_voice_JniUtils_nativeJavaUtf16StringToStdString) (BuildId: 3f1eaa46e50eae77373842563e1080e5e07b180a)
#34 pc 0x5a47eb libtwilio_voice_android_so.so (__udivdi3) (BuildId: 3f1eaa46e50eae77373842563e1080e5e07b180a)
#35 pc 0x228bcb libtwilio_voice_android_so.so (Java_com_twilio_voice_JniUtils_nativeJavaUtf16StringToStdString) (BuildId: 3f1eaa46e50eae77373842563e1080e5e07b180a)
#36 pc 0x5cc068 libtwilio_voice_android_so.so (__udivdi3) (BuildId: 3f1eaa46e50eae77373842563e1080e5e07b180a)
#37 pc 0x5cbf4a libtwilio_voice_android_so.so (__udivdi3) (BuildId: 3f1eaa46e50eae77373842563e1080e5e07b180a)
#38 pc 0x229879 libtwilio_voice_android_so.so (Java_com_twilio_voice_JniUtils_nativeJavaUtf16StringToStdString) (BuildId: 3f1eaa46e50eae77373842563e1080e5e07b180a)
#39 pc 0xade0b libc.so (BuildId: f4abefa5427b93525ac009af9345d160)
#40 pc 0xadde1 libc.so (BuildId: f4abefa5427b93525ac009af9345d160)
#41 pc 0x63e73 libc.so (BuildId: f4abefa5427b93525ac009af9345d160)
#42 pc 0xadde1 libc.so (BuildId: f4abefa5427b93525ac009af9345d160)
#43 pc 0x229851 libtwilio_voice_android_so.so (Java_com_twilio_voice_JniUtils_nativeJavaUtf16StringToStdString) (BuildId: 3f1eaa46e50eae77373842563e1080e5e07b180a)

Java_com_twilio_voice_JniUtils_nativeJavaUtf16StringToStdString

Crashed: Thread: SIGABRT  0x0000000000000000
#00 pc 0x89bcc libc.so (BuildId: 4ba2121e6580a7f88bce586ed966851c)
#01 pc 0x89b9c libc.so (BuildId: 4ba2121e6580a7f88bce586ed966851c)
#02 pc 0xed044 libc.so (BuildId: 4ba2121e6580a7f88bce586ed966851c)
#03 pc 0xec640 libc.so (BuildId: 4ba2121e6580a7f88bce586ed966851c)
#04 pc 0xec498 libc.so (BuildId: 4ba2121e6580a7f88bce586ed966851c)
#05 pc 0x3a64e0 libtwilio_voice_android_so.so (Java_com_twilio_voice_JniUtils_nativeJavaUtf16StringToStdString) (BuildId: 813acfa9a0fd2a25ad616ccc3794e8b8df49b8fd)
#06 pc 0x159000 libtwilio_voice_android_so.so (Java_com_twilio_voice_JniUtils_nativeJavaUtf16StringToStdString) (BuildId: 813acfa9a0fd2a25ad616ccc3794e8b8df49b8fd)
#07 pc 0x159444 libtwilio_voice_android_so.so (Java_com_twilio_voice_JniUtils_nativeJavaUtf16StringToStdString) (BuildId: 813acfa9a0fd2a25ad616ccc3794e8b8df49b8fd)
#08 pc 0x1596b8 libtwilio_voice_android_so.so (Java_com_twilio_voice_JniUtils_nativeJavaUtf16StringToStdString) (BuildId: 813acfa9a0fd2a25ad616ccc3794e8b8df49b8fd)
#09 pc 0x15e2e4 libtwilio_voice_android_so.so (Java_com_twilio_voice_JniUtils_nativeJavaUtf16StringToStdString) (BuildId: 813acfa9a0fd2a25ad616ccc3794e8b8df49b8fd)
#10 pc 0x15e578 libtwilio_voice_android_so.so (Java_com_twilio_voice_JniUtils_nativeJavaUtf16StringToStdString) (BuildId: 813acfa9a0fd2a25ad616ccc3794e8b8df49b8fd)
#11 pc 0x17fa44 libtwilio_voice_android_so.so (Java_com_twilio_voice_JniUtils_nativeJavaUtf16StringToStdString) (BuildId: 813acfa9a0fd2a25ad616ccc3794e8b8df49b8fd)
#12 pc 0x3a6af0 libtwilio_voice_android_so.so (Java_com_twilio_voice_JniUtils_nativeJavaUtf16StringToStdString) (BuildId: 813acfa9a0fd2a25ad616ccc3794e8b8df49b8fd)
#13 pc 0x3a53f4 libtwilio_voice_android_so.so (Java_com_twilio_voice_JniUtils_nativeJavaUtf16StringToStdString) (BuildId: 813acfa9a0fd2a25ad616ccc3794e8b8df49b8fd)
#14 pc 0x3a427c libtwilio_voice_android_so.so (Java_com_twilio_voice_JniUtils_nativeJavaUtf16StringToStdString) (BuildId: 813acfa9a0fd2a25ad616ccc3794e8b8df49b8fd)
#15 pc 0x3a5990 libtwilio_voice_android_so.so (Java_com_twilio_voice_JniUtils_nativeJavaUtf16StringToStdString) (BuildId: 813acfa9a0fd2a25ad616ccc3794e8b8df49b8fd)
#16 pc 0xeb968 libc.so (BuildId: 4ba2121e6580a7f88bce586ed966851c)
#17 pc 0x8bb88 libc.so (BuildId: 4ba2121e6580a7f88bce586ed966851c)

Versions

Kotlin: 1.9.0
Android gradle plugin: 8.1.1
com.twilio:audioswitch: 1.1.8

Voice Android SDK

com.twilio:voice-android:6.3.2

OS Version

Various but primarily: 14, 13, 12, 10

Device Model

Various: Samsung, Xiaomi, Google, etc.

@paco-spr paco-spr changed the title Voice NDK crashes Voice NDK crashes Java_com_twilio_voice_JniUtils_nativeJavaUtf16StringToStdString Nov 1, 2023
@afalls-twilio
Copy link
Contributor

afalls-twilio commented Nov 15, 2023

@paco-spr Thanks for the report, do you have by any chance, any knowledge if this is only happening on 32 bit devices or builds? (I ask because of the call to software division, __udivdi3)

@paco-spr
Copy link
Author

paco-spr commented Dec 1, 2023

@paco-spr Thanks for the report, do you have by any chance, any knowledge if this is only happening on 32 bit devices or builds? (I ask because of the call to software division, __udivdi3)

Sorry for the delay in getting back to you.

It looks like firebase lists a couple different variants for this issue. The variant containing the __udivdi3 call happens significantly less than this variant:

null pointer dereference: SIGSEGV  0x0000000000000092
#00 pc 0x158fec libtwilio_voice_android_so.so (Java_com_twilio_voice_JniUtils_nativeJavaUtf16StringToStdString) (BuildId: 813acfa9a0fd2a25ad616ccc3794e8b8df49b8fd)
#01 pc 0x158fd4 libtwilio_voice_android_so.so (Java_com_twilio_voice_JniUtils_nativeJavaUtf16StringToStdString) (BuildId: 813acfa9a0fd2a25ad616ccc3794e8b8df49b8fd)
#02 pc 0x159444 libtwilio_voice_android_so.so (Java_com_twilio_voice_JniUtils_nativeJavaUtf16StringToStdString) (BuildId: 813acfa9a0fd2a25ad616ccc3794e8b8df49b8fd)
#03 pc 0x1596b8 libtwilio_voice_android_so.so (Java_com_twilio_voice_JniUtils_nativeJavaUtf16StringToStdString) (BuildId: 813acfa9a0fd2a25ad616ccc3794e8b8df49b8fd)
#04 pc 0x15e2e4 libtwilio_voice_android_so.so (Java_com_twilio_voice_JniUtils_nativeJavaUtf16StringToStdString) (BuildId: 813acfa9a0fd2a25ad616ccc3794e8b8df49b8fd)
#05 pc 0x15e578 libtwilio_voice_android_so.so (Java_com_twilio_voice_JniUtils_nativeJavaUtf16StringToStdString) (BuildId: 813acfa9a0fd2a25ad616ccc3794e8b8df49b8fd)
#06 pc 0x17fa44 libtwilio_voice_android_so.so (Java_com_twilio_voice_JniUtils_nativeJavaUtf16StringToStdString) (BuildId: 813acfa9a0fd2a25ad616ccc3794e8b8df49b8fd)
#07 pc 0x3a6af0 libtwilio_voice_android_so.so (Java_com_twilio_voice_JniUtils_nativeJavaUtf16StringToStdString) (BuildId: 813acfa9a0fd2a25ad616ccc3794e8b8df49b8fd)
#08 pc 0x3a53f4 libtwilio_voice_android_so.so (Java_com_twilio_voice_JniUtils_nativeJavaUtf16StringToStdString) (BuildId: 813acfa9a0fd2a25ad616ccc3794e8b8df49b8fd)
#09 pc 0x3a427c libtwilio_voice_android_so.so (Java_com_twilio_voice_JniUtils_nativeJavaUtf16StringToStdString) (BuildId: 813acfa9a0fd2a25ad616ccc3794e8b8df49b8fd)
#10 pc 0x3a5990 libtwilio_voice_android_so.so (Java_com_twilio_voice_JniUtils_nativeJavaUtf16StringToStdString) (BuildId: 813acfa9a0fd2a25ad616ccc3794e8b8df49b8fd)
#11 pc 0xbb6e8 libc.so (BuildId: 5111da12b6134a422c2730af0cf237cb)
#12 pc 0x54fdc libc.so (BuildId: 5111da12b6134a422c2730af0cf237cb)

However, I've checked both variants, and they both seem to happen on 64 bit devices such as Galaxy A13, Redmi A1+, Galaxy S22 Ultra.

@jamesbluecrow
Copy link

Hi @afalls-twilio , any update around this?

Looking at all our error logs we are seeing around 3k crashes per month around the native voice library.

Below I'm pasting the most common traces that we are seeing:

Java_com_twilio_voice_JniUtils_nativeJavaUtf16StringToStdString
null pointer dereference: SIGSEGV  0x0000000000000000
#00 pc 0x3bd8d0 libtwilio_voice_android_so.so (Java_com_twilio_voice_JniUtils_nativeJavaUtf16StringToStdString) (BuildId: 753c8c664d7e4c562d64143915c5fd60f776e52e)
_Unwind_GetTextRelBase
Crashed: Thread: SIGSEGV  0x000492efe99abc00
#00 pc 0x85b4e0 libtwilio_voice_android_so.so (_Unwind_GetTextRelBase) (BuildId: 813acfa9a0fd2a25ad616ccc3794e8b8df49b8fd)
#01 pc 0x85b4c8 libtwilio_voice_android_so.so (_Unwind_GetTextRelBase) (BuildId: 813acfa9a0fd2a25ad616ccc3794e8b8df49b8fd)
#02 pc 0x85c8f4 libtwilio_voice_android_so.so (_Unwind_Backtrace) (BuildId: 813acfa9a0fd2a25ad616ccc3794e8b8df49b8fd)
#03 pc 0x39e014 libtwilio_voice_android_so.so (Java_com_twilio_voice_JniUtils_nativeJavaUtf16StringToStdString) (BuildId: 813acfa9a0fd2a25ad616ccc3794e8b8df49b8fd)
Java_com_twilio_voice_Call_nativeDisconnect
null pointer dereference: SIGSEGV  0x0000000000000000
#00 pc 0x38100c libtwilio_voice_android_so.so (BuildId: 753c8c664d7e4c562d64143915c5fd60f776e52e)
#01 pc 0x389a04 libtwilio_voice_android_so.so (Java_com_twilio_voice_Call_nativeDisconnect) (BuildId: 753c8c664d7e4c562d64143915c5fd60f776e52e)

Any recommendation or tip as to how to avoid them or fix them (on our end if possible) would be much appreciated it.

@afalls-twilio
Copy link
Contributor

afalls-twilio commented Jan 23, 2024

jamesbluecrow what is your build setup?

or how is the application installed on the user's system? is it a preloaded app, or are their other versions twilio voice somehow on the same application folder?

The other thing, have you been able to reproduce this with the quickstart?

Is your application exposing this C API (Java_com_twilio_voice_JniUtils_nativeJavaUtf16StringToStdString)? It is a JNI binding method that the JVM will bind to the java class com.twilio.voice.JniUtils which does not exist as of the 6.3.2 (the stated release) release. So unless it was exposed some other way, its not exposed to the JAVA layer and is not linked/bound to any java class... its dead legacy code (not called by anyone).

@jamesbluecrow
Copy link

jamesbluecrow commented Jan 24, 2024

jamesbluecrow what is your build setup?

or how is the application installed on the user's system? is it a preloaded app, or are their other versions twilio voice somehow on the same application folder?

The other thing, have you been able to reproduce this with the quickstart?

Is your application exposing this C API (Java_com_twilio_voice_JniUtils_nativeJavaUtf16StringToStdString)? It is a JNI binding method that the JVM will bind to the java class com.twilio.voice.JniUtils which does not exist as of the 6.3.2 (the stated release) release. So unless it was exposed some other way, its not exposed to the JAVA layer and is not linked/bound to any java class... its dead legacy code (not called by anyone).

The users are installing the app thorough google play.

It is not a preloaded app and there is no other versions of twilio voice:

We have this in our gradle (copied from the quickstart):

    splits {
        abi {
            isEnable = true
            reset()
            // Specify each architecture currently supported by the Twilio Voice SDK
            include("armeabi-v7a", "arm64-v8a", "x86", "x86_64")
            // Specify that we do not want an additional universal SDK
            isUniversalApk = false
        }
    }

dependencies {
    // com.twilio:voice-android:6.4.0
    implementation(libs.twilio.voice.android) 
    // com.twilio:audioswitch:1.1.8
    implementation(libs.twilio.audio.switch)
}

I haven't been able to reproduce this, neither with the quickstart, or in our app in debug or even in our release, but we are definitely seeing some users affected by this, we have around 3k crashes monthly coming from the native Twilio SDK.

We only started noticing this when we enabled crashlytics ndk reports so they were likely happening before but we hadn't noticed.

We are seeing this happening in different versions of Android 8, 9, 10, 11, 12, 13 but it id not happening to all users on those versions.

We also see a variety of manufacturers & models (Samsung, Xiaomi, Motorola, etc) so it is not related to a specific manufacturer/model either.

Proguard is just configured as as the quickstart project

# Twilio Programmable Voice
-keep class com.twilio.** { *; }
-keep class tvo.webrtc.** { *; }
-dontwarn tvo.webrtc.**
-keep class com.twilio.voice.** { *; }
-keepattributes InnerClasses
# needed with AGP 8.x
-dontwarn android.content.pm.PackageManager$ApplicationInfoFlags
-dontwarn android.content.pm.PackageManager$PackageInfoFlags

...

and the most common crashes come from one of these 3 functions calls:

Java_com_twilio_voice_JniUtils_nativeJavaUtf16StringToStdString

null pointer dereference: SIGSEGV  0x0000000000000048
#00 pc 0xae75a libtwilio_voice_android_so.so (Java_com_twilio_voice_JniUtils_nativeJavaUtf16StringToStdString) (BuildId: 3f1eaa46e50eae77373842563e1080e5e07b180a)
#01 pc 0x5aa12e libtwilio_voice_android_so.so (__udivdi3) (BuildId: 3f1eaa46e50eae77373842563e1080e5e07b180a)
#02 pc 0x5aa13a libtwilio_voice_android_so.so (__udivdi3) (BuildId: 3f1eaa46e50eae77373842563e1080e5e07b180a)
#03 pc 0x5aa12e libtwilio_voice_android_so.so (__udivdi3) (BuildId: 3f1eaa46e50eae77373842563e1080e5e07b180a)
#04 pc 0x690782 libtwilio_voice_android_so.so (__udivdi3) (BuildId: 3f1eaa46e50eae77373842563e1080e5e07b180a)
#05 pc 0x9fd53 libtwilio_voice_android_so.so (Java_com_twilio_voice_JniUtils_nativeJavaUtf16StringToStdString) (BuildId: 3f1eaa46e50eae77373842563e1080e5e07b180a)
#06 pc 0xcc4be [libc.so](http://libc.so/) (BuildId: ab9eb3f965a77a04086244c6ba4e1c11)
#07 pc 0x690782 libtwilio_voice_android_so.so (__udivdi3) (BuildId: 3f1eaa46e50eae77373842563e1080e5e07b180a)
#08 pc 0x663c5a libtwilio_voice_android_so.so (__udivdi3) (BuildId: 3f1eaa46e50eae77373842563e1080e5e07b180a)
#09 pc 0x69075a libtwilio_voice_android_so.so (__udivdi3) (BuildId: 3f1eaa46e50eae77373842563e1080e5e07b180a)
#10 pc 0xaea73 libtwilio_voice_android_so.so (Java_com_twilio_voice_JniUtils_nativeJavaUtf16StringToStdString) (BuildId: 3f1eaa46e50eae77373842563e1080e5e07b180a)
#11 pc 0x5aa283 libtwilio_voice_android_so.so (__udivdi3) (BuildId: 3f1eaa46e50eae77373842563e1080e5e07b180a)
#12 pc 0x5aa2fa libtwilio_voice_android_so.so (__udivdi3) (BuildId: 3f1eaa46e50eae77373842563e1080e5e07b180a)
#13 pc 0x5aa283 libtwilio_voice_android_so.so (__udivdi3) (BuildId: 3f1eaa46e50eae77373842563e1080e5e07b180a)
#14 pc 0xcc4be [libc.so](http://libc.so/) (BuildId: ab9eb3f965a77a04086244c6ba4e1c11)
#15 pc 0x663c46 libtwilio_voice_android_so.so (__udivdi3) (BuildId: 3f1eaa46e50eae77373842563e1080e5e07b180a)
#16 pc 0x8b105 libtwilio_voice_android_so.so (BuildId: 3f1eaa46e50eae77373842563e1080e5e07b180a)
#17 pc 0xaec97 libtwilio_voice_android_so.so (Java_com_twilio_voice_JniUtils_nativeJavaUtf16StringToStdString) (BuildId: 3f1eaa46e50eae77373842563e1080e5e07b180a)
#18 pc 0xa29dd libtwilio_voice_android_so.so (Java_com_twilio_voice_JniUtils_nativeJavaUtf16StringToStdString) (BuildId: 3f1eaa46e50eae77373842563e1080e5e07b180a)
#19 pc 0x504087 libtwilio_voice_android_so.so (std::__ndk1::__shared_weak_count::__add_weak()) (BuildId: 3f1eaa46e50eae77373842563e1080e5e07b180a)
#20 pc 0x9824d libtwilio_voice_android_so.so (Java_com_twilio_voice_JniUtils_nativeJavaUtf16StringToStdString) (BuildId: 3f1eaa46e50eae77373842563e1080e5e07b180a)
#21 pc 0xb0e67 libtwilio_voice_android_so.so (Java_com_twilio_voice_JniUtils_nativeJavaUtf16StringToStdString) (BuildId: 3f1eaa46e50eae77373842563e1080e5e07b180a)
#22 pc 0x5aac86 libtwilio_voice_android_so.so (__udivdi3) (BuildId: 3f1eaa46e50eae77373842563e1080e5e07b180a)
#23 pc 0x5aac90 libtwilio_voice_android_so.so (__udivdi3) (BuildId: 3f1eaa46e50eae77373842563e1080e5e07b180a)
#24 pc 0x5aac86 libtwilio_voice_android_so.so (__udivdi3) (BuildId: 3f1eaa46e50eae77373842563e1080e5e07b180a)
#25 pc 0xcc4be [libc.so](http://libc.so/) (BuildId: ab9eb3f965a77a04086244c6ba4e1c11)
#26 pc 0xb1051 libtwilio_voice_android_so.so (Java_com_twilio_voice_JniUtils_nativeJavaUtf16StringToStdString) (BuildId: 3f1eaa46e50eae77373842563e1080e5e07b180a)
#27 pc 0x504087 libtwilio_voice_android_so.so (std::__ndk1::__shared_weak_count::__add_weak()) (BuildId: 3f1eaa46e50eae77373842563e1080e5e07b180a)
#28 pc 0x98561 libtwilio_voice_android_so.so (Java_com_twilio_voice_JniUtils_nativeJavaUtf16StringToStdString) (BuildId: 3f1eaa46e50eae77373842563e1080e5e07b180a)
#29 pc 0xc27ab libtwilio_voice_android_so.so (Java_com_twilio_voice_JniUtils_nativeJavaUtf16StringToStdString) (BuildId: 3f1eaa46e50eae77373842563e1080e5e07b180a)
#30 pc 0x514525 libtwilio_voice_android_so.so (__gnu_ldivmod_helper) (BuildId: 3f1eaa46e50eae77373842563e1080e5e07b180a)
#31 pc 0x22a223 libtwilio_voice_android_so.so (Java_com_twilio_voice_JniUtils_nativeJavaUtf16StringToStdString) (BuildId: 3f1eaa46e50eae77373842563e1080e5e07b180a)
#32 pc 0x2295c7 libtwilio_voice_android_so.so (Java_com_twilio_voice_JniUtils_nativeJavaUtf16StringToStdString) (BuildId: 3f1eaa46e50eae77373842563e1080e5e07b180a)
#33 pc 0x229117 libtwilio_voice_android_so.so (Java_com_twilio_voice_JniUtils_nativeJavaUtf16StringToStdString) (BuildId: 3f1eaa46e50eae77373842563e1080e5e07b180a)
#34 pc 0x229117 libtwilio_voice_android_so.so (Java_com_twilio_voice_JniUtils_nativeJavaUtf16StringToStdString) (BuildId: 3f1eaa46e50eae77373842563e1080e5e07b180a)
#35 pc 0x5a47eb libtwilio_voice_android_so.so (__udivdi3) (BuildId: 3f1eaa46e50eae77373842563e1080e5e07b180a)
#36 pc 0x228bcb libtwilio_voice_android_so.so (Java_com_twilio_voice_JniUtils_nativeJavaUtf16StringToStdString) (BuildId: 3f1eaa46e50eae77373842563e1080e5e07b180a)
#37 pc 0x5cc068 libtwilio_voice_android_so.so (__udivdi3) (BuildId: 3f1eaa46e50eae77373842563e1080e5e07b180a)
#38 pc 0x5cbf4a libtwilio_voice_android_so.so (__udivdi3) (BuildId: 3f1eaa46e50eae77373842563e1080e5e07b180a)
#39 pc 0x229879 libtwilio_voice_android_so.so (Java_com_twilio_voice_JniUtils_nativeJavaUtf16StringToStdString) (BuildId: 3f1eaa46e50eae77373842563e1080e5e07b180a)
#40 pc 0xade0b [libc.so](http://libc.so/) (BuildId: ab9eb3f965a77a04086244c6ba4e1c11)
#41 pc 0xadde1 [libc.so](http://libc.so/) (BuildId: ab9eb3f965a77a04086244c6ba4e1c11)
#42 pc 0x63e73 [libc.so](http://libc.so/) (BuildId: ab9eb3f965a77a04086244c6ba4e1c11)
#43 pc 0xadde1 [libc.so](http://libc.so/) (BuildId: ab9eb3f965a77a04086244c6ba4e1c11)
#44 pc 0x229851 libtwilio_voice_android_so.so (Java_com_twilio_voice_JniUtils_nativeJavaUtf16StringToStdString) (BuildId: 3f1eaa46e50eae77373842563e1080e5e07b180a)
Java_com_twilio_voice_Call_nativeDisconnect

null pointer dereference: SIGSEGV  0x0000000000000009
#00 pc 0x38100c libtwilio_voice_android_so.so (BuildId: 753c8c664d7e4c562d64143915c5fd60f776e52e)
#01 pc 0x389a04 libtwilio_voice_android_so.so (Java_com_twilio_voice_Call_nativeDisconnect) (BuildId: 753c8c664d7e4c562d64143915c5fd60f776e52e)

I can see in the logs that it is usually after a call has ended (but I can't confirm that is the case 100% of the time).

We are honestly a bit at a lost with this and not sure how to proceed/fix it.

@afalls-twilio
Copy link
Contributor

afalls-twilio commented Jan 26, 2024

yeah you and me both...
what is weird is that the method Java_com_twilio_voice_JniUtils_nativeJavaUtf16StringToStdString isn't even exposed, its dead code. I know you said that they were using 6.3.2, but is it possible that these crashes happening on old versions of our SDK?

I'll throw in a ticket on our end to remove that dead function.. as for clients in the field... I have no idea at this point in time.. I have not been able to repo on our end.

@jamesbluecrow
Copy link

Definitely happening in newer versions. We have been using 6.4.0 for the last couple of versions of our app and still see crashes in Java_com_twilio_voice_JniUtils_nativeJavaUtf16StringToStdString.

That code must somehow be used as we obviously don't interact with the native code itself. Maybe it is some kind of internal thread of the native sdk calling itself?

The second and third biggest sources of crashes coming from the native sdk are in case they are related:

_Unwind_GetTextRelBase and Java_com_twilio_voice_Call_nativeDisconnect.

We will keep an eye on when a new version is published and we will see what happens.

Ps. Many thanks for the effort you are putting to look into this 🙏 .

Ps2. Proportionally with the amount of use we do of the SDK the crashes are small, but at the moment the Twilio SDK is the biggest source of crashes in our app which is something we need to address obviously

@YvesDelerm
Copy link

Hello, just my two cents,
We have had the same issue with the same stacktrace and similar frequence (1-2% of users) : Java_com_twilio_voice_JniUtils_nativeJavaUtf16StringToStdString
It was occurring on calls lasting more than 10 minutes, but it was fixed in Twilio SDK 6.3.2. So it's not exactly the same issue but in case it could help...

We also reduced the number of crashes by making sure that all calls to Twilio SDK are wrapped within the same thread.

@afalls-twilio
Copy link
Contributor

afalls-twilio commented Feb 1, 2024

@YvesDelerm yes, all the Twilio api calls must come from the same thread, its not thread safe.. As for Java_com_twilio_voice_JniUtils_nativeJavaUtf16StringToStdString, we recently removed it entirely (not yet released), its was not being called internally either, it was a vestige from the past that was just dead. It is a complete mystery how it is ever being invoked....

@jamesbluecrow
Copy link

@YvesDelerm yes, all the Twilio api calls must come from the same thread, its not thread safe.. As for Java_com_twilio_voice_JniUtils_nativeJavaUtf16StringToStdString, we recently removed it entirely (not yet released), its was not being called internally either, it was a vestige from the past that was just dead. It is a complete mystery how it is ever being invoked....

Ok cool.

  1. Please let us know when the release that removes Java_com_twilio_voice_JniUtils_nativeJavaUtf16StringToStdString is available, we will keep an eye on the crashes after we update to it.
  2. We will revisit our implementation and guarantee that we always interact with the SDK in the same thread (I think we do, but we will check)
  3. Did you have time to look at the other two crashes I mentioned? _Unwind_GetTextRelBase and Java_com_twilio_voice_Call_nativeDisconnect? They happen way less often, but we we still see around 200 crashes per month for each of them.

@afalls-twilio
Copy link
Contributor

afalls-twilio commented Feb 5, 2024

As for _Unwind_GetTextRelBase, thats a libc++ method that gets called when a c++ exception is triggered, again its very weird that there was crash there and for Java_com_twilio_voice_Call_nativeDisconnect while I was unable to repo it, that method is what is gets called when the SDK is told to disconnect and in the past, I have seen crashes there when users attempt to call it or another sdk method from a different thread than the thread they have made all other SDK calls from. This is especially true when users call the stats functions from a different thread than the thread they are calling disconnect from. Another common error is that they make calls SDK calls from a thread that does not have a handler... ie if you are not making SDK calls from the Main thread, make sure the thread you are invoking the sdk from is a HandlerThread, otherwise the call backs will be routed to the main thread (and any subsequent SDK method invocations from those callbacks/notify methods will have been made from a different thread).

@jamesbluecrow
Copy link

As for _Unwind_GetTextRelBase, thats a libc++ method that gets called when a c++ exception is triggered, again its very weird that there was crash there and for Java_com_twilio_voice_Call_nativeDisconnect while I was unable to repo it, that method is what is gets called when the SDK is told to disconnect and in the past, I have seen crashes there when users attempt to call it or another sdk method from a different thread than the thread they have made all other SDK calls from. This is especially true when users call the stats functions from a different thread than the thread they are calling disconnect from. Another common error is that they make calls SDK calls from a thread that does not have a handler... ie if you are not making SDK calls from the Main thread, make sure the thread you are invoking the sdk from is a HandlerThread, otherwise the call backs will be routed to the main thread (and any subsequent SDK method invocations from those callbacks/notify methods will have been made from a different thread).

Perfect, we are revisiting how we call the SDK to make sure that we always use the Main thread.

Thanks for the response and the tips 🙏

@afalls-twilio
Copy link
Contributor

FYI, the release made yesterday (6.4.1) contains the removal of the unused method Java_com_twilio_voice_JniUtils_nativeJavaUtf16StringToStdString

@jamesbluecrow
Copy link

Great, thanks for letting me know.

We will update to the latest version and will let you know in a week or so when we release if we stop seeing crashes.

@kbagchiGWC
Copy link
Contributor

Hi @jamesbluecrow - did you get a chance to validate Voice Android 6.4.1 release?

@jamesbluecrow
Copy link

Hey @kbagchiGWC @afalls-twilio!

Yes, I can confirm that in com.twilio:voice-android:6.4.1 the crash seems gone to be gone, I haven't seen a single instance of it and we have a fair amount of users running the latest version.

I think this issue can now be closed.

Thank you for your help folks 🙏

@kbagchiGWC
Copy link
Contributor

@jamesbluecrow Thanks for validation the behavior. I will close the ticket as completed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

5 participants