[mono] App hangs at startup when using `dotnet publish -t:Run` and different combination of RIDs

[tipa]

Android framework version

net10.0-android (Preview)

Affected platform version

.net10-rc2

Description

App crashes when accessing crypto-related APIs.
From what I can recall, this issue started with .net10-rc2 - it still worked fine on .net10-rc1 or .net9

Steps to Reproduce

• dotnet new android
• Add this to csproj: <RuntimeIdentifiers>android-arm;android-arm64;android-x64</RuntimeIdentifiers>
• Add the following code:

using var ecdsa = ECDsa.Create();
ecdsa.ImportPkcs8PrivateKey(Convert.FromBase64String("MIGHAgEAMBMGByqGSM49AgEGCCqGSM49AwEHBG0wawIBAQQgSpP55ELdXswj9JRZAPRwtTfS4CNRqpKIs+28rNHiPAqhRANCAASs8nLES7b+goKslppNVOurf0MonZdw3pb6TxS8Z/5j+UNY1sWK1ChxpuwNS9I3R50cfdQo/lA9PPhw6XIg8ytd\r\n"), out _);

• dotnet publish -t:Run (or dotnet build -c:Release -t:Run)
• crash (when running on arm64 device, in my case Pixel 7)

Did you find any workaround?

• Removing the android-arm rid
• Putting the android-arm rid last in the list: <RuntimeIdentifiers>android-arm64;android-x64;android-arm</RuntimeIdentifiers>

Relevant log output

[grendello]

I tested the sample and in my findings (using .NET for Android main) I can repro the issue with android-arm first in the list of RIDs and using dotnet publish -t:Run, but it works for me when I use dotnet build -t:Run. Putting android-arm last in the list of RIDs makes it work with publish, thus suggesting a packaging issue when using it.

The difference between the two appears to be that the latter doesn't AOT and it uses individual assemblies, while publish performs AOT and uses assembly stores.

I don't see a hard crash either, the application exits cleanly in the "not working" case, albeit the puzzling part is that the reportedly missing symbol is actually there:

10-17 11:25:18.062 27900 27900 F monodroid-assembly: Failed to load symbol 'AndroidCryptoNative_EcKeyCreateByKeyParameters' from shared library 'libSystem.Security.Cryptography.Native.Android'
10-17 11:25:18.110  1444  4298 I ActivityManager: Process com.companyname.test_jwt (pid 27900) has died: fg  TOP 
10-17 11:25:18.110  1444  1712 I libprocessgroup: Removed cgroup /sys/fs/cgroup/apps/uid_10361/pid_27900
10-17 11:25:18.111   913   913 I Zygote  : Process 27900 exited cleanly (0)

The above makes me think that some code calls exit(0) which causes Android to restart the app, and thus get stuck in a restart loop. I will investigate the "missing symbol" issue in the dotnet publish case, @jonathanpeppers can you take a look into the packaging part of publish?

[grendello]

dotnet publish appears to be quite broken. I tested replacing android-arm as the first RID with android-x86, and this resulted in these build errors (with <RuntimeIdentifiers>android-x86;android-arm64;android-x64</RuntimeIdentifiers> in the .csproj):

libSystem.IO.Compression.Native.so : error XA4301: Cannot determine ABI of native library 'libSystem.IO.Compression.Native.so'. Move this file to a directory with a valid Android ABI name such as 'libs/armeabi-v7a/'
libSystem.IO.Compression.Native.so : error XA4301: Cannot determine ABI of native library 'libSystem.IO.Compression.Native.so'. Move this file to a directory with a valid Android ABI name such as 'libs/armeabi-v7a/'
libSystem.Security.Cryptography.Native.Android.so : error XA4301: Cannot determine ABI of native library 'libSystem.Security.Cryptography.Native.Android.so'. Move this file to a directory with a valid Android ABI name such as 'libs/armeabi-v7a/
libSystem.IO.Compression.Native.so : error XA4301: Cannot determine ABI of native library 'libSystem.IO.Compression.Native.so'. Move this file to a directory with a valid Android ABI name such as 'libs/armeabi-v7a/'
libSystem.Security.Cryptography.Native.Android.so : error XA4301: Cannot determine ABI of native library 'libSystem.Security.Cryptography.Native.Android.so'. Move this file to a directory with a valid Android ABI name such as 'libs/armeabi-v7a/'

Furthermore, when using android-arm as the first RID, I see that the JNI preload indexes and data (generated here) are broken for all the RIDs except for android-arm, and that's what causes the issue.
Correct data for android-arm (from obj/Release/net10.0-android/android/environment.armeabi-v7a.ll), showing only the relevant indexes of the dso_cache array (index number is shown after each entry):

; Indices into dso_cache[] of DSO libraries to preload because of JNI use
@dso_jni_preloads_idx = dso_local local_unnamed_addr constant [4 x i32] [
        i32 15, ; libSystem.Security.Cryptography.Native.Android.so
        i32 0, ; libSystem.Security.Cryptography.Native.Android
        i32 7, ; System.Security.Cryptography.Native.Android.so
        i32 8 ; System.Security.Cryptography.Native.Android
], align 4

; DSO cache entries
@dso_cache = dso_local local_unnamed_addr global [28 x %struct.DSOCacheEntry] [
        %struct.DSOCacheEntry {
                i64 u0x00000000093625cd, ; from name: libSystem.Security.Cryptography.Native.Android
                i64 u0x000000007d3da8be, ; uint64_t real_name_hash
                i1 false, ; bool ignore
                i1 true, ; bool is_jni_library
                ptr @.DSOCacheEntry.1_name, ; name: libSystem.Security.Cryptography.Native.Android.so
                ptr null; void* handle
        }, ; 0
; ...
        %struct.DSOCacheEntry {
                i64 u0x0000000033e41c10, ; from name: System.Security.Cryptography.Native.Android.so
                i64 u0x000000007d3da8be, ; uint64_t real_name_hash
                i1 false, ; bool ignore
                i1 true, ; bool is_jni_library
                ptr @.DSOCacheEntry.1_name, ; name: libSystem.Security.Cryptography.Native.Android.so
                ptr null; void* handle
        }, ; 7
        %struct.DSOCacheEntry {
                i64 u0x000000005360f89d, ; from name: System.Security.Cryptography.Native.Android
                i64 u0x000000007d3da8be, ; uint64_t real_name_hash
                i1 false, ; bool ignore
                i1 true, ; bool is_jni_library
                ptr @.DSOCacheEntry.1_name, ; name: libSystem.Security.Cryptography.Native.Android.so
                ptr null; void* handle
        }, ; 8
; ...
        %struct.DSOCacheEntry {
                i64 u0x000000007d3da8be, ; from name: libSystem.Security.Cryptography.Native.Android.so
                i64 u0x000000007d3da8be, ; uint64_t real_name_hash
                i1 false, ; bool ignore
                i1 true, ; bool is_jni_library
                ptr @.DSOCacheEntry.1_name, ; name: libSystem.Security.Cryptography.Native.Android.so
                ptr null; void* handle
        }, ; 15

Note that the name: fields in dso_cache match those in dso_jni_preloads_idx.

Now the same data for android-arm64:

; Indices into dso_cache[] of DSO libraries to preload because of JNI use
@dso_jni_preloads_idx = dso_local local_unnamed_addr constant [8 x i32] [
        i32 15, ; libSystem.Security.Cryptography.Native.Android.so
        i32 0, ; libSystem.Security.Cryptography.Native.Android
        i32 7, ; System.Security.Cryptography.Native.Android.so
        i32 8, ; System.Security.Cryptography.Native.Android
        i32 10, ; Invalid index 4
        i32 0, ; Invalid index 5
        i32 1, ; Invalid index 6
        i32 14 ; Invalid index 7
], align 4

and the same for android-x64:

; Indices into dso_cache[] of DSO libraries to preload because of JNI use
@dso_jni_preloads_idx = dso_local local_unnamed_addr constant [12 x i32] [
        i32 15, ; libSystem.Security.Cryptography.Native.Android.so
        i32 0, ; libSystem.Security.Cryptography.Native.Android
        i32 7, ; System.Security.Cryptography.Native.Android.so
        i32 8, ; System.Security.Cryptography.Native.Android
        i32 10, ; Invalid index 4
        i32 0, ; Invalid index 5
        i32 1, ; Invalid index 6
        i32 14, ; Invalid index 7
        i32 10, ; Invalid index 8
        i32 0, ; Invalid index 9
        i32 1, ; Invalid index 10
        i32 14 ; Invalid index 11
], align 16

Something in dotnet publish messes up our build state when processing data for different architectures. In the two latter cases above you can see that android-arm data always comes first, followed by the 2nd RID's and then the 3rd RID's which are appended to the android-arm state. To understand the data a bit better - every shared library has 4 entries in the dso_jni_preloads_idx array as well as in the dso_cache array. They are different variations of the library name as any of them may be used by the managed code and we xxhash all the names to search the dso_cache array without having to find entries in it by string.
For JNI libraries (like the android crypto one above), we preload them at startup and then set the library handle in all the alias entires of dso_cache. This is where things go wrong, because (this is for android-arm64 using the data shown above):

10-17 13:43:16.774 12902 12902 D monodroid-assembly: DSO jni preloads index stride == 4
10-17 13:43:16.774 12902 12902 D monodroid-assembly: Preloading JNI shared library: libSystem.IO.Compression.Native.so (entry's index: 15; real name hash: c3cb80650fe5a0ab; name hash: 9190f4cb761b1d3c)
10-17 13:43:16.774 12902 12902 D monodroid-assembly: monodroid_dlopen: hash match found, DSO name is 'libSystem.IO.Compression.Native.so'
10-17 13:43:16.774 12902 12902 D monodroid-assembly: monodroid_dlopen: cache entry's real name hash == 0xc3cb80650fe5a0ab; name hash == 0x9190f4cb761b1d3c
10-17 13:43:16.774 12902 12902 I monodroid-assembly: [apk] Trying to load shared library 'libSystem.IO.Compression.Native.so', offset in the apk == 98304
10-17 13:43:16.775 12902 12902 D monodroid-assembly: Shared library libSystem.IO.Compression.Native.so loaded (handle 0x30ec238ac1f60407)
10-17 13:43:16.775 12902 12902 D monodroid-assembly: Putting JNI library handle in alias entry at index 0: libSystem.Security.Cryptography.Native.Android.so
10-17 13:43:16.775 12902 12902 D monodroid-assembly: Putting JNI library handle in alias entry at index 7: libmonodroid.so
10-17 13:43:16.775 12902 12902 D monodroid-assembly: Putting JNI library handle in alias entry at index 8: libSystem.IO.Compression.Native.so

Since the android-arm64 data above has the first 4 entries mapped incorrectly, it tries to load libSystem.IO.Compression.Native.so which lives at index 14 for arm64 and index 15 for arm (entries are sorted by hash) and that explains the later error of:

10-17 13:43:16.885 12902 12902 D monodroid-assembly: monodroid_dlopen: hash match found, DSO name is 'libSystem.Security.Cryptography.Native.Android.so'
10-17 13:43:16.885 12902 12902 I monodroid-assembly: monodroid_dlsym: handle == 0x30ec238ac1f60407; name == 'AndroidCryptoNative_EcKeyCreateByKeyParameters'
10-17 13:43:16.885 12902 12902 I monodroid-assembly:   s == 0x0
10-17 13:43:16.885 12902 12902 I monodroid-assembly:   errno == 2; error == 'No such file or directory'
10-17 13:43:16.885 12902 12902 W monodroid-assembly: Symbol 'AndroidCryptoNative_EcKeyCreateByKeyParameters' not found in shared library 'libSystem.Security.Cryptography.Native.Android', p/invoke may fail
10-17 13:43:16.885 12902 12902 F monodroid-assembly: Failed to load symbol 'AndroidCryptoNative_EcKeyCreateByKeyParameters' from shared library 'libSystem.Security.Cryptography.Native.Android'

The handle we pass to dlsym is actually for libSystem.IO.Compression.Native.so and not for libSystem.Security.Cryptography.Native.Android.so, so the AndroidCryptoNative_EcKeyCreateByKeyParameters symbol really doesn't exist there.

The native code works correctly, given the data it uses, but we need to figure out what causes the mixing of data from different RIDs during build with dotnet publish -t:Run but not with dotnet build -t:Run

[grendello]

The cache generation issue is a bug in my code, fix is in progress. And the libSystem.IO.Compression.Native.so : error XA4301: Cannot determine ABI of native library 'libSystem.IO.Compression.Native.so'. Move this file to a directory with a valid Android ABI name such as 'libs/armeabi-v7a/' error is a red herring caused by automatic inclusion of every file in the build directory... 🤦 :)

I still don't understand why the code works with dotnet build but breaks with dotnet publish, though. The bug I found is real, however, and needs to be fixed regardless. Stay tuned.