SIGSEGV crash immediately after entering AchievementManager#InternalFetchAllCallback

[FrankNine]

Hello,

We were logging our users into Google Play Service on start up before, and we switched to sign-in button approach to comply item 1.1.2 on Google Play Service Guidelines:
https://developers.google.com/games/services/checklist#1_sign-in

But since then, there is a chance our app would crash when users click on sign-in button and triggered a silent sign-in.(User signed-in without Google Play Service overlay pop-up) The app was using 0.9.32 with Google Play Service 8.4, and we tried upgrading to 0.9.34 with Google Play Service 9.0.1 but the same thing still happens.

The logcat is attached here (Bundle ID is removed):
06-01 12:20:03.786: I/Unity(8726): [Play Games Plugin DLL] 06/01/16 12:20:03 +08:00 DEBUG: Entering internal callback for AchievementManager#InternalFetchAllCallback
06-01 12:20:03.786: I/Unity(8726):
06-01 12:20:03.786: I/Unity(8726): (Filename: ./artifacts/generated/Android/runtime/UnityEngineDebugBindings.gen.cpp Line: 40)
06-01 12:20:03.791: I/Unity(8726): [Play Games Plugin DLL] 06/01/16 12:20:03 +08:00 DEBUG: Populating Achievements, status = VALID
06-01 12:20:03.791: I/Unity(8726):
06-01 12:20:03.791: I/Unity(8726): (Filename: ./artifacts/generated/Android/runtime/UnityEngineDebugBindings.gen.cpp Line: 40)
06-01 12:20:03.791: I/Unity(8726): --------- beginning of crash
06-01 12:20:03.805: A/libc(8726): Fatal signal 11 (SIGSEGV), code 1, fault addr 0xd19c0000 in tid 10350 (callback_queue)
06-01 12:20:03.831: I/Unity(8726): Makoto-Manager: Timeout, choose to abort
06-01 12:20:03.831: I/Unity(8726):
06-01 12:20:03.831: I/Unity(8726): (Filename: ./artifacts/generated/Android/runtime/UnityEngineDebugBindings.gen.cpp Line: 40)
06-01 12:20:03.861: D/clmlib(628): Got activities:0x0000000E
06-01 12:20:03.898: I/SELinux(628): SELinux: Loaded file_contexts contexts from /file_contexts.
06-01 12:20:03.904: A/DEBUG(628): *** *** *** *** *** *** *** *** *** *** *** *** *** *** ** ***
06-01 12:20:03.905: A/DEBUG(628): UUID: df0cb7ff-6a71-486f-af0b-54c8fca5ad9b
06-01 12:20:03.905: A/DEBUG(628): Build fingerprint: 'Sony/E6853/E6853:6.0/32.1.A.1.185/3574277109:user/release-keys'
06-01 12:20:03.905: A/DEBUG(628): Revision: '0'
06-01 12:20:03.905: A/DEBUG(628): ABI: 'arm'
06-01 12:20:03.905: A/DEBUG(628): pid: 8726, tid: 10350, name: callback_queue >>> com..<<<
06-01 12:20:03.905: A/DEBUG(628): signal 11 (SIGSEGV), code 1 (SEGV_MAPERR), fault addr 0xd19c0000
06-01 12:20:03.953: A/DEBUG(628): Abort message: 'PRODUCTION: Adjust is running in Production mode. Use this setting only for the build that you want to publish. Set the environment to sandbox if you want to test your app!'
06-01 12:20:03.953: A/DEBUG(628): r0 c4a2866c r1 c4a2866c r2 d19c0000 r3 0036a920
06-01 12:20:03.953: A/DEBUG(628): r4 c4a2866c r5 d19c0000 r6 d196c754 r7 fca02400
06-01 12:20:03.953: A/DEBUG(628): r8 fffdb1a8 r9 c4a28728 sl 00000000 fp c4a286b8
06-01 12:20:03.953: A/DEBUG(628): ip db96b9f8 sp c4a28648 lr db5ff190 pc db5ff164 cpsr 200d0010
06-01 12:20:03.961: A/DEBUG(628): backtrace:
06-01 12:20:03.961: A/DEBUG(628): #00 pc 00cde164 /data/app/com..-1/lib/arm/libil2cpp.so (il2cpp::utils::StringUtils::Utf16ToUtf8(unsigned short const_, int)+132)
06-01 12:20:03.961: A/DEBUG(628): #1 pc 00cf21f4 /data/app/com..-1/lib/arm/libil2cpp.so (il2cpp::vm::PlatformInvoke::MarshalStringBuilder(Il2CppStringBuilder_)+56)
06-01 12:20:03.961: A/DEBUG(628): #2 pc 002c1308 /data/app/com..-1/lib/arm/libil2cpp.so
06-01 12:20:03.961: A/DEBUG(628): #3 pc 0047552c /data/app/com..-1/lib/arm/libil2cpp.so
06-01 12:20:03.961: A/DEBUG(628): #4 pc 0046f828 /data/app/com..-1/lib/arm/libil2cpp.so
06-01 12:20:03.961: A/DEBUG(628): #5 pc 00475260 /data/app/com..-1/lib/arm/libil2cpp.so
06-01 12:20:03.961: A/DEBUG(628): #6 pc 0037300c /data/app/com..-1/lib/arm/libil2cpp.so
06-01 12:20:03.961: A/DEBUG(628): #7 pc 0057c908 /data/app/com..-1/lib/arm/libil2cpp.so
06-01 12:20:03.961: A/DEBUG(628): #8 pc 0046acb0 /data/app/com..-1/lib/arm/libil2cpp.so
06-01 12:20:03.961: A/DEBUG(628): #9 pc 0046aaac /data/app/com..-1/lib/arm/libil2cpp.so
06-01 12:20:03.961: A/DEBUG(628): #10 pc 0046a9ac /data/app/com..-1/lib/arm/libil2cpp.so
06-01 12:20:03.962: A/DEBUG(628): #11 pc 000dd1cf /data/app/com..-1/lib/arm/libgpg.so
06-01 12:20:03.962: A/DEBUG(628): #12 pc 0005bae5 /data/app/com..-1/lib/arm/libgpg.so
06-01 12:20:03.962: A/DEBUG(628): #13 pc 000b3d3f /data/app/com..-1/lib/arm/libgpg.so
06-01 12:20:03.962: A/DEBUG(628): #14 pc 0006468d /data/app/com..-1/lib/arm/libgpg.so
06-01 12:20:03.962: A/DEBUG(628): #15 pc 0003fa7b /system/lib/libc.so (__pthread_start(void)+30)
06-01 12:20:03.962: A/DEBUG(628): #16 pc 0001a055 /system/lib/libc.so (__start_thread+6)
06-01 12:20:04.431: I/AudioFlinger(632): BUFFER TIMEOUT: remove(4096) from active list on thread 0xf1600000
06-01 12:20:04.603: A/DEBUG(628): Tombstone written to: /data/tombstones/tombstone_06
06-01 12:20:04.603: E/DEBUG(628): AM write failed: Broken pipe
06-01 12:20:04.603: D/(628): private_mode
06-01 12:20:04.603: I/BootReceiver(1743): Copying /data/tombstones/tombstone_06 to DropBox (SYSTEM_TOMBSTONE)

[claywilkinson]

The welcome alert is not shown if the player appears to be going in and out of the game relatively frequently. This is by design as it was found to be distracting to players.

I don't see anything Google related in the crash - the only thing that looks suspicious is this message:

Abort message: 'PRODUCTION: Adjust is running in Production mode. Use this setting only for the build that you want to publish. Set the environment to sandbox if you want to test your app!'

Is that normal?

If it is easily reproducible, can you capture a bugreport? This is done using adb bugreport > bugreport.txt.

[FrankNine]

Hello,

Thank you very much for your reply. The bugreport is recorded here.
bugreport.txt

It is normal for Adjust to display that message, but the timing looks weird. I will try to reach people at Adjust, too.

[timr-github]

Hi

We are also seeing instances of the same crash in our games, and can provide a symbolicated call stack for the calls prior to the call to MarshalStringBuilder:

libil2cpp : 00322324 : 000000e5 .hidden Achievement_Achievement_RevealedIconUrl_m3107455841
libil2cpp : 0036d0a0 : 00000075 .hidden NativeAchievement_U3CgetRevealedImageUrlU3Em__61D_m189762143
libil2cpp : 003797ba : 0000010d .hidden OutStringMethod_Invoke_m853968476
libil2cpp : 00374e3f : 0000017a .hidden PInvokeUtilities_OutParamsToString_m2028749481
libil2cpp : 0036c9d1 : 000000b5 .hidden NativeAchievement_getRevealedImageUrl_m865604097
libil2cpp : 0036cccc : 00000401 .hidden NativeAchievement_AsAchievement_m1237034209
libil2cpp : 00344658 : 00000826 .hidden NativeClient_PopulateAchievements_m1753150098
libil2cpp : 00348d30 : 00000046 .hidden U3CHandleAuthTransitionU3Ec__AnonStorey26E_U3CU3Em__5A1_m2976330208
libil2cpp : 00a6ec86 : 000000a4 .hidden Action_1_Invoke_m101203496_gshared
libil2cpp : 00a5e47c : 0000015c .hidden U3CToIntPtrU3Ec__AnonStorey2C1_1_U3CU3Em__621_m2406581467_gshared
libil2cpp : 00a6eb05 : 000000aa .hidden Action_1_Invoke_m1563799394_gshared
libil2cpp : 003635a3 : 000003c4 .hidden Callbacks_PerformInternalCallback_m3646632285
libil2cpp : 00363422 : 0000009b .hidden AchievementManager_InternalFetchAllCallback_m2450256647
libil2cpp : 00363835 : 00000069 .hidden ReversePInvokeWrapper_AchievementManager_InternalFetchAllCallback_m2450256647
libgpg : 0014192c :
libil2cpp : 003637ee : 00000069 .hidden ReversePInvokeWrapper_AchievementManager_InternalFetchAllCallback_m2450256647
libgpg : 001371a4 :

We may be able to provide some further information on this next week.

Thanks

[davidbeps]

Just to follow up on Tim's post.

We spent a bit of time instrumenting the code with extra debug.logs to find out more information.

Hopefully some of this information will be of use.

(1) These crashes only occur when the app is built with Unity IL2CPP scripting backend. I've not seen a crash when built with a Mono backend.

(2) Some devices seem to exhibit the crash more often than others, mostly x86 architectures. For example the Lenovo TAB S8-50F tablet, Android 5.0.1.

(3) It doesn't crash on every invocation of Social.localUser.Authenticate(). Strangely, when it does crash the Welcome/avatar toast banner is usually being shown. Generally when the toast is not shown I do not see the crash.

(4) The crash always occurs when extracting achievement text data in the NativeAchievement.AsAchievement() method, when called from NativeClient.PopulateAchievements(). It's not always crashing on the same text field, but mostly in getRevealedImageUrl(). I've seen it crash inside Name() and Description().

(5) It never crashes processing the same achievement - it seems to be a random achievement when it goes.

(6) We are currently building with the Google Play Game Service Plugin v0.9.34 and on Unity 5.3.5p8.

[jordiboni]

Hi @davidbeps , have you reported a bug to Unity with a repro project?

[timr-github]

To follow-up :

We reported the issue to Unity. A repro project isn't easy to make as the issue is very sporadic both in terms of frequency and device.

The issue is still occuring in live apps as described, as-of today.

Thanks

[claywilkinson]

Have you tried with 5.4 ? According to the Release Notes, IL2CPP was experimental for Android before 5.4

[timr-github]

Hi

We should be able to test with 5.4 soon - we had to wait for some other fixes before we could switch to it. I'll report back as soon as possible.

Thanks

[davidbeps]

Ok, we've managed to test with 5.4.0p1 but unfortunately we get the same crash again, in the same places when populating string data for achievements. Again not 100% and only more repeatable on certain devices like the Lenovo TAB S8-50F tablet, Android 5.0.1.

By adding debug logs it seems like it's failing native side, or the bit that marshals/unmarshals the StringBuilder between C# and native code. It's during this phase there is a probability it will crash on some devices.

[davidbeps]

After some further investigation, we have discovered that the use of a StringBuilder object to exchange string data between native code and C# might be contributing to the random sign-in crashes (when the Android app is built with IL2CPP scripting backend).

As an experiment we replaced the use of StringBuilder with a char[] array as a means of marshalling string data between native code and C# and did not experience any sign-in crashes.

Taking the marshalling of an achievement's description as an example, we made the following changes:-

(1) In GooglePlayGames\Platforms\Native\CWrapper\Achievement.cs,

Replace usage of out_arg from StringBuilder to char[]

    [DllImport(SymbolLocation.NativeSymbolLocation)]
    internal static extern /* from(size_t) */ UIntPtr Achievement_Description(
        HandleRef self,
     /* from(char *) */StringBuilder out_arg,
     /* from(size_t) */UIntPtr out_size);

to become

    [DllImport(SymbolLocation.NativeSymbolLocation)]
    internal static extern /* from(size_t) */ UIntPtr Achievement_Description(
        HandleRef self,
     [In, Out] /* from(char *) */ char[] out_arg,
     /* from(size_t) */UIntPtr out_size);

(2) In GooglePlayGames\Platforms\Native\PInvoke\PInvokeUtilities.cs,

Change OutParamsToString() method to use a char[] instead of a StringBuilder when invoking the native code.

From (original)

    internal delegate UIntPtr OutStringMethod(StringBuilder out_string,UIntPtr out_size);

    internal static String OutParamsToString(OutStringMethod outStringMethod)
    {
        UIntPtr requiredSize = outStringMethod(null, UIntPtr.Zero);
        if (requiredSize.Equals(UIntPtr.Zero))
        {
            return null;
        }

        StringBuilder sizedBuilder = new StringBuilder((int)requiredSize.ToUInt32());
        outStringMethod(sizedBuilder, requiredSize);
        return sizedBuilder.ToString();
    }

to become (based on the implementation of existing OutParamsToArray generic method in PInvokeUtilities.cs)

    internal delegate UIntPtr OutStringMethod([In, Out] char[] out_bytes,UIntPtr out_size);

    internal static string OutParamsToString(OutStringMethod outMethod)
    {
        UIntPtr requiredSize = outMethod(null, UIntPtr.Zero);
        if (requiredSize.Equals(UIntPtr.Zero))
        {
            return string.Empty;
        }

        string str = null;
        try
        {
            char[] array = new char[requiredSize.ToUInt32()];
            outMethod(array, requiredSize);
            str = new string(array, 0, (int)requiredSize.ToUInt32() - 1);
        }
        catch (Exception e)
        {
            UnityEngine.Debug.Log("[GPG] Exception creating string from char array: " + e);
            str = string.Empty;
        }

        return str;
    }

The above changes were also applied to all achievement and player data fields that used StringBuilder to marshal data between native and C# code.
When we pushed these code change to our clients we went from hundreds of crashes a day to 0. I'm hoping the above information can provide further insight into the true cause of the problem and lead to an official fix.

[claywilkinson]

Thanks for the details! I tried IL2CPP for the sample apps, and everything seems to work Unity 5.4

[nhatnd]

I have the same problem with @davidbeps and after following his instruction, the bug has gone. But there is another problem: When I build the app using IL2CPP (for production version), the string response from plugin is correct, but if I build the app using Mono2x (for test version - to reduce build time), the string responses is incorrect.

This is the string I received with the same google account:
IL2CPP:
[Play Games Plugin DLL] 04/20/17 10:01:15 +07:00 DEBUG: Found User: [Player: 'Beelzeebub91' (id g00640087066468541325)]
Mono2x:
[Play Games Plugin DLL] 04/20/17 10:12:29 +07:00 DEBUG: Found User: [Player: '敂汥敺扥扵ㄹ

I have no idea where the Chines character comes from.