Users getting stuck on Splash Screen -> Fatal Exception: java.lang.UnsatisfiedLinkError

[mxa0079]

Internal track: b/159618489

Versions of flutter were we have seen this issue:

• V 1.12
• V 1.17

Steps to Reproduce

1. Create a CI pipeline on a Mac OS 10.14
2. Build the application with the following command $(FlutterToolPath)/flutter build appbundle --build-name=$(version_name) --build-number=$(build_number)
3. Use Fastlane to upload the application to the PlayStore using the command fastlane supply --track internal --aab ../build/app/outputs/bundle/release/app.aab
4. Distribute the application to a large number of users

Expected results:
All users should be able to access the application.

Actual results:
A number of users are getting stuck on the splash screen of the application.

We believe this is being caused by the following exception being reported by Crashlytics on FlutterLoader.Java

Fatal Exception: java.lang.UnsatisfiedLinkError
dalvik.system.PathClassLoader[DexPathList[[zip file "/data/app/com.sivale.android-1/base.apk", zip file "/data/app/com.sivale.android-1/split_config.armeabi_v7a.apk"],nativeLibraryDirectories=[/data/app/com.sivale.android-1/lib/arm, /data/app/com.sivale.android-1/base.apk!/lib/armeabi-v7a, /data/app/com.sivale.android-1/split_config.armeabi_v7a.apk!/lib/armeabi-v7a, /system/lib, /vendor/lib]]] couldn't find "libflutter.so"

We believe that Crashlytics is under-counting this problem because according to the official data there are 160 users affected:

but given that we have more than 1 million users, and we have many reports in our close circle, we fear that the actual number of users affected is MUCH higher

Logs

We see two different reports related to this issue on Crashlytics. One occurring on FlutterLoader.java line 120 and the other on FlutterLoader.java line 122

Full stack trace for FlutterLoader.java line 120:

Fatal Exception: java.lang.UnsatisfiedLinkError
dalvik.system.PathClassLoader[DexPathList[[zip file "/data/app/com.sivale.android-1/base.apk", zip file "/data/app/com.sivale.android-1/split_config.armeabi_v7a.apk"],nativeLibraryDirectories=[/data/app/com.sivale.android-1/lib/arm, /data/app/com.sivale.android-1/base.apk!/lib/armeabi-v7a, /data/app/com.sivale.android-1/split_config.armeabi_v7a.apk!/lib/armeabi-v7a, /system/lib, /vendor/lib]]] couldn't find "libflutter.so"
java.lang.Runtime.loadLibrary0 (Runtime.java:984)
java.lang.System.loadLibrary (System.java:1562)
io.flutter.embedding.engine.loader.FlutterLoader.startInitialization (FlutterLoader.java:120)
io.flutter.embedding.engine.loader.FlutterLoader.startInitialization (FlutterLoader.java:88)
io.flutter.view.FlutterMain.startInitialization (FlutterMain.java:48)
io.flutter.app.FlutterApplication.onCreate (FlutterApplication.java:24)
android.app.Instrumentation.callApplicationOnCreate (Instrumentation.java:1025)
android.app.ActivityThread.handleBindApplication (ActivityThread.java:5616)
android.app.ActivityThread.-wrap2 (ActivityThread.java)
android.app.ActivityThread$H.handleMessage (ActivityThread.java:1614)
android.os.Handler.dispatchMessage (Handler.java:102)
android.os.Looper.loop (Looper.java:154)
android.app.ActivityThread.main (ActivityThread.java:6351)
java.lang.reflect.Method.invoke (Method.java)
com.android.internal.os.ZygoteInit$MethodAndArgsCaller.run (ZygoteInit.java:896)
com.android.internal.os.ZygoteInit.main (ZygoteInit.java:786)

Full stack trace for FlutterLoader.java line 122:

Fatal Exception: java.lang.UnsatisfiedLinkError
Couldn't load flutter from loader dalvik.system.PathClassLoader[DexPathList[[zip file "/data/app/com.sivale.android-2.apk"],nativeLibraryDirectories=[/data/app-lib/com.sivale.android-2, /vendor/lib, /system/lib]]]: findLibrary returned null
java.lang.Runtime.loadLibrary (Runtime.java:358)
java.lang.System.loadLibrary (System.java:526)
io.flutter.embedding.engine.loader.FlutterLoader.startInitialization (FlutterLoader.java:122)
io.flutter.embedding.engine.loader.FlutterLoader.startInitialization (FlutterLoader.java:93)
io.flutter.view.FlutterMain.startInitialization (FlutterMain.java:45)
io.flutter.app.FlutterApplication.onCreate (FlutterApplication.java:22)
android.app.Instrumentation.callApplicationOnCreate (Instrumentation.java:1013)
android.app.ActivityThread.handleBindApplication (ActivityThread.java:4624)
android.app.ActivityThread.access$1600 (ActivityThread.java:166)
android.app.ActivityThread$H.handleMessage (ActivityThread.java:1341)
android.os.Handler.dispatchMessage (Handler.java:102)
android.os.Looper.loop (Looper.java:146)
android.app.ActivityThread.main (ActivityThread.java:5511)
java.lang.reflect.Method.invokeNative (Method.java)
java.lang.reflect.Method.invoke (Method.java:515)
com.android.internal.os.ZygoteInit$MethodAndArgsCaller.run (ZygoteInit.java:1283)
com.android.internal.os.ZygoteInit.main (ZygoteInit.java:1099)
dalvik.system.NativeStart.main (NativeStart.java)

Details of the machine running the CI Build: https://github.com/microsoft/azure-pipelines-image-generation/blob/master/images/macos/macos-10.14-Readme.md

Note: we have seen what it appears to be a sharp increase in the occurrence of this issue since migrating to 1.17 specially in the exception thrown on line 120:

Here are the stats for the exception thrown on line 122:

[mxa0079]

This issue is a fork of the following issue> #37566

[zanderso]

@chinmaygarde from #37566

So my current best hypothesis is the following: As long as you are not running into the problem specified by the original reporter, these issues are coming from app bundle installs on devices without Google Play Services, or, incomplete sideloads. I am trying to verify this hypothesis now.

/cc @blasten for app bundle advice.

[blasten]

@mxa0079 A few questions:

V 1.13 never made it to stable, did you intend to use a dev Flutter version to build your app?

Have you set up special split rules in build.gradle?

[mxa0079]

Apologies @blasten - I meant V 1.12

I'll check with the team regarding the build.gradle.

[chinmaygarde]

@mxa0079: I spent a lot more time on this and while I still think incomplete app bundle installs are the underlying cause, I haven't able to induce this error. We'll wait to see if @blasten has any ideas but in the meantime, I need some help with more information:

• Using bundletool, expand the aab you have distributed and you know is causing issues: bundletool build-apks --bundle app.aab --output out.apks.
• out.apks will be a ZIP archive, unzip it.
• After unzipping, in the splits folder, there will be a base-armeabi_v7a.apk APK. Unzip that.
• In that unzipped, folder, there should be a lib/armeabi-v7a folder.
• Is there libflutter.so in that folder?

This is just a sanity check to ensure that the .so is in the APK.

Also, you mentioned in the previous issue that you had reports from production from users with this issue. Would it be possible to reach out to even one of them for more information? Specifically, the following information would be useful:

• How was the APK installed? From the store, or via a sideload. If it was a sideload, how was this managed?
• Does the device have Google Play Services? I believe you can have the store installed but not Play Services.
  • @blasten I am actually not sure about this, can you confirm? This is just from my understanding reading the documentation.
• Does the user have a custom ROM installed? If so, which one?
• Is the device rooted?
• If the user is willing, would they be willing to run a non-appbundle APK on their device to see if that runs?

Not sure if reaching out to a user is possible but just wanted to jot down information that would be super useful in this investigation.

[hectorvillag]

Hi @chinmaygarde
This is the information we have

• They have only installed it from the store
• We are investigating if they had Google Play Services
• As far as we know, none had a custom rom, but we will investigate
• We also do not know if they were rooted although it is unlikely, we will also investigate it
• We shared a locally generated apk to a user and they managed to enter the application, we havent been able to try out with more users

[mxa0079]

Also, I'd like to point out that we have observed cases were users were running into this issue, installed an update on the app and the issue got solved. We have also had cases were users were not running into this issue, installed an update and then ran into the issue. This makes me think that something is going wrong during the installation process.

Regarding the sanity check, we will be glad to ensure that libflutter.so is in the lib/armeabi-v7a. However, I could almost bet that it will be there otherwise this issue would be much widespread and it would be easy to reproduce. Cheers, Mau

[blasten]

@chinmaygarde spot on. That's my understanding too.

@mxa0079 Would you be able to share the application id? My email is egarciad@google.com.

[hectorvillag]

Hi @blasten
We have the following rules in our build.gradle

   bundle {
       language {
           enableSplit = false
       }
       density {
           enableSplit = false
       }
       abi {
           enableSplit = true
       }
   }

[blasten]

Thanks. Would you be able to inspect the AAB files
using the bundle tool as @chinmaygarde suggested? Otherwise, if possible, feel free to send it over email.

[hectorvillag]

Sure, we inspected the ABB file, and yes there is the libflutter.so file (files: base-x86_64.apk, base-armeabi_v7a.apk, base-arm64_v8a.apk)

[perepechin]

I found a similar issue in Crashlytics :

Fatal Exception: java.lang.UnsatisfiedLinkError
dalvik.system.PathClassLoader[DexPathList[[zip file "/data/app/com.perepechin.workouts-1/base.apk"],nativeLibraryDirectories=[/data/app/com.perepechin.workouts-1/lib/x86, /data/app/com.perepechin.workouts-1/base.apk!/lib/x86, /vendor/lib, /system/lib]]] couldn't find "libflutter.so"

which occurred on the Nexus 5S (x86)

In my case I am using bundles, and I see that if I build the app using this command:
flutter build appbundle --release

in the generated bundle, I have 4 folders in the lib folder inside that bundle:
arm64-v8a
armeabi-v7a
x86
x86_64

in in the x86 folder, in my case, I have only one file:
libsqlite3.so

so I don't see libflutter.so in this x86 folder

Then I found that if I build apk with this command:
flutter build apk --release --split-per-abi --target-platform android-arm,android-arm64,android-x64
in the generated APK files I don't have x86 at all, because I didn't include android-x86 as option to the target-platform. So I have only APK files with arm64-v8a, armeabi-v7a and x86_64 and in that case I can install the app from the Play Store on my x86 emulator, and I can run the app without any issues and I don't have this exception anymore.

at this point I am not sure how to create an app bundle with option to exclude x86 completelty from the generated bundle, and have only x86_64, arm64-v8a and armeabi-v7a.

so currently creating apk files via this command
flutter build apk --release --split-per-abi --target-platform android-arm,android-arm64,android-x64
is a workaround for me

I hope that this workaround will help someone as well.

[chinmaygarde]

@perepechin You are running into this issue instead. The solution is to to specify the architectures up-front as you said, or, use the abiFilters directive in the Gradle file.

[blasten]

@mxa0079 has there been any progress on this issue?

[mxa0079]

Hello @blasten,

We are currently working and exchanging logs with the App Bundle team. I will update the issue once we have relevant results to share.

Thanks,

Mau

[chinmaygarde]

Removing the "waiting for customer response" tag so it doesn't get auto-closed. Looks like this is assigned internally.

[faslurrajah]

I found a similar issue in Crashlytics :

Fatal Exception: java.lang.UnsatisfiedLinkError
dalvik.system.PathClassLoader[DexPathList[[zip file "/data/app/com.perepechin.workouts-1/base.apk"],nativeLibraryDirectories=[/data/app/com.perepechin.workouts-1/lib/x86, /data/app/com.perepechin.workouts-1/base.apk!/lib/x86, /vendor/lib, /system/lib]]] couldn't find "libflutter.so"

which occurred on the Nexus 5S (x86)

In my case I am using bundles, and I see that if I build the app using this command:
flutter build appbundle --release

in the generated bundle, I have 4 folders in the lib folder inside that bundle:
arm64-v8a
armeabi-v7a
x86
x86_64

in in the x86 folder, in my case, I have only one file:
libsqlite3.so

so I don't see libflutter.so in this x86 folder

Then I found that if I build apk with this command:
flutter build apk --release --split-per-abi --target-platform android-arm,android-arm64,android-x64
in the generated APK files I don't have x86 at all, because I didn't include android-x86 as option to the target-platform. So I have only APK files with arm64-v8a, armeabi-v7a and x86_64 and in that case I can install the app from the Play Store on my x86 emulator, and I can run the app without any issues and I don't have this exception anymore.

at this point I am not sure how to create an app bundle with option to exclude x86 completelty from the generated bundle, and have only x86_64, arm64-v8a and armeabi-v7a.

so currently creating apk files via this command
flutter build apk --release --split-per-abi --target-platform android-arm,android-arm64,android-x64
is a workaround for me

I hope that this workaround will help someone as well.

Hi, If we exclude x86 ,then how can we install it in x86 devices?

[chinmaygarde]

Hi, If we exclude x86 ,then how can we install it in x86 devices?

You cannot.

[Hixie]

@mxa0079 any updates?

[chinmaygarde]

There have been no progress updates on the internal issue either. I'll reach out separately.

[chinmaygarde]

The internal issue update indicates that there is still some trouble figuring out a reproducible test case for this.

[xster]

#59501 (comment) is the correct answer. We don't support running on Android in x86(-32). See https://flutter.dev/docs/resources/faq#what-devices-and-os-versions-does-flutter-run-on.

If you bring in 32-bit x86 native .so into your app via gradle, there will be undefined behaviors. Some devices will do arm emulation from the armv7 .so but will often turn emulation off as soon as one x86 .so appears (at which point it will fail to find a x86/libflutter.so and crash).

[chinmaygarde]

@xster This is a different issue than the one I linked. See the followup to my original comment in the linked thread.

[xster]

We're linking the same thing no? (with 1Δ in indirection)

[chinmaygarde]

Yes. The first link describes the original issue, the followup explains why this one is slightly different.

[xster]

@mxa0079 please reopen if I didn't follow the issue correctly. Otherwise I'm going to tentatively close this issue.

The remaining UX work will be at #61122.

To summarize, Flutter doesn't support x86. Trying to run on an x86 device will possibly produce the UnsatisfiedLinkError. It's possible that some devices doesn't experience it on a default Flutter project if the OEM device supports ARM emulation but the emulation will likely be turned off if there are other native x86 .so files in your lib/ directory of the APK.

If you must support x86, the best recourse which may or may not work will be to use abiFilters to remove all x86 .so files and see if the crash rate goes down by re-enabling ARM emulation on some devices.

[xster]

Oops, @chinmaygarde just pointed out that I crossed threads in my last comment which was incorrect. I incorrectly responded to #59501 (comment) which despite logs similarities is an entirely different issue.

The original question was on (based on our analysis) Play Store not fulfilling all the native libraries on the way down as uploaded initially in the AAB. The unrelated comment I was responding to was on us not supporting x86.

We're leaving this closed however since the original issue was not actionable by Flutter either and is likely an issue in the Play Store b/159618489.

[MichaelM97]

We're leaving this closed however since the original issue was not actionable by Flutter either and is likely an issue in the Play Store b/159618489.

@xster What should we do as a workaround? Only upload the APK's created using flutter build apk --split-per-abi to the Playstore?

As for me if I use apktool to decompile the produced APK's I can see all of them have libflutter.so and libapp.so in the lib->{ABI} folder.

Also, is there a public Google issue we can track for this issue with the Playstore?

[thechinkysight]

Can anybody tell me whats the solution?

[ycv005]

@mxa0079 any update ?

[deadsoul44]

Since this issue is closed, nobody will care about these comments.

[github-actions]

This thread has been automatically locked since there has not been any recent activity after it was closed. If you are still experiencing a similar issue, please open a new bug, including the output of flutter doctor -v and a minimal reproduction of the issue.