Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Crash on startup on Android armv7 (Flutter 1.22.3) #80043

Closed
julianzur opened this issue Apr 8, 2021 · 13 comments
Closed

Crash on startup on Android armv7 (Flutter 1.22.3) #80043

julianzur opened this issue Apr 8, 2021 · 13 comments
Assignees
Labels
a: production Issues experienced in live production apps a: release Challenges faced when attempting to productionize an app c: fatal crash Crashes that terminate the process customer: vroom engine flutter/engine repository. See also e: labels. P0 Critical issues such as a build break or regression platform-android Android applications specifically

Comments

@julianzur
Copy link

julianzur commented Apr 8, 2021

Hi flutter community,

our app is currently crashing on armv7 devices with release and profile builds. It's running fine on all the other architectures and also with debug builds. We're using Flutter 1.22.3 to build the app.

First we saw the number of crashes increasing in the Google Play Console:

signal 4 (SIGILL), code 1 (ILL_ILLOPC)
split_config.armeabi_v7a.apk (offset 0x3fa8000)
*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
pid: 0, tid: 0 >>> xx.xxx <<<

backtrace:
  #00  pc 00000000002bca1c  /data/app/xx.xxx-1/split_config.armeabi_v7a.apk (offset 0x3fa8000)
  #00  pc 0000000002161ba4  /data/app/xx.xxx-1/split_config.armeabi_v7a.apk (offset 0x59c000)

Unfortunately, we're still building with gradle 3.5.3, so we can't set the android.defaultConfig.ndk.debugSymbolLevel for now.

But luckily we're able to reproduce the problem on the Android armv7 emulator (Android 7.1.1) where we can get much more detailed information about the crash from the tombstones:

*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
Build fingerprint: 'Android/sdk_google_phone_armv7/generic:7.1.1/NYC/6695155:userdebug/test-keys'
Revision: '0'
ABI: 'arm'
pid: 12980, tid: 13012, name: 1.ui  >>> xx.xxx <<<
signal 4 (SIGILL), code 1 (ILL_ILLOPC), fault addr 0x98efa394
    r0 94b80031  r1 91f81105  r2 91f810cd  r3 00000004
    r4 94b81ce9  r5 92f00021  r6 00000001  r7 92180ffc
    r8 00000000  r9 9337bfe1  sl 986b6c00  fp 98a7475c
    ip 98987000  sp 98a74750  lr 96fe9060  pc 98efa394  cpsr 20000010
    d0  3ff0000000000002  d1  0000000040e00000
    d2  0000000041300000  d3  986ebd3898a74b94
    d4  0000000000000000  d5  986ebd3898a74b94
    d6  0000000000000000  d7  00000000986b6c00
    d8  0000000000000000  d9  0000000000000000
    d10 0000000000000000  d11 0000000000000000
    d12 0000000000000000  d13 0000000000000000
    d14 0000000000000000  d15 0000000000000000
    d16 3ff0000000000000  d17 3fe6b851eb851eb8
    d18 0000000000000000  d19 0000000000000000
    d20 0000000000000000  d21 0000000000000000
    d22 0000000000000000  d23 0000000000000000
    d24 0000000000000000  d25 0000000000000000
    d26 0000000000000000  d27 0000000000000000
    d28 0000000000000000  d29 0000000000000000
    d30 0000000000000000  d31 0000000000000000
    scr 20000013

backtrace:
    #00 pc 000b2394  /data/app/xx.xxxx-1/lib/arm/libflutter.so (offset 0x131000)
    #01 pc 0212e05c  /data/app/xx.xxxx-1/lib/arm/libapp.so (offset 0xb000) (_kDartIsolateSnapshotInstructions+34746460)

@flutter-symbolizer-bot a1440ca392ca23e874a105c5f3248b495bd0e247 release arm

flutter doctor -v
[✓] Flutter (Channel stable, 1.22.3, on Mac OS X 10.15.7 19H524, locale en-DE)
    • Flutter version 1.22.3 at /Users/xxxxx/development/flutter
    • Framework revision 8874f21e79 (5 months ago), 2020-10-29 14:14:35 -0700
    • Engine revision a1440ca392
    • Dart version 2.10.3
[!] Android toolchain - develop for Android devices (Android SDK version 30.0.1)
    • Android SDK at /Users/xxxxx/Library/Android/sdk
    • Platform android-30, build-tools 30.0.1
    • Java binary at: /Applications/Android Studio.app/Contents/jre/jdk/Contents/Home/bin/java
    • Java version OpenJDK Runtime Environment (build 1.8.0_242-release-1644-b3-6915495)
    ✗ Android license status unknown.
      Run `flutter doctor --android-licenses` to accept the SDK licenses.
      See https://flutter.dev/docs/get-started/install/macos#android-setup for more details.
[✓] Xcode - develop for iOS and macOS (Xcode 12.1)
    • Xcode at /Applications/Xcode12.1.app/Contents/Developer
    • Xcode 12.1, Build version 12A7403
    • CocoaPods version 1.10.0
[!] Android Studio (version 4.1)
    • Android Studio at /Applications/Android Studio.app/Contents
    ✗ Flutter plugin not installed; this adds Flutter specific functionality.
    ✗ Dart plugin not installed; this adds Dart specific functionality.
    • Java version OpenJDK Runtime Environment (build 1.8.0_242-release-1644-b3-6915495)
[✓] IntelliJ IDEA Ultimate Edition (version 2019.3.1)
    • IntelliJ at /Applications/IntelliJ IDEA.app
    • Flutter plugin version 42.1.4
    • Dart plugin version 193.5731
[✓] VS Code (version 1.54.3)
    • VS Code at /Applications/Visual Studio Code.app/Contents
    • Flutter extension version 3.21.0
[✓] Proxy Configuration
    • HTTP_PROXY is set
    • NO_PROXY is localhost,127.0.0.1
    • NO_PROXY contains 127.0.0.1
    • NO_PROXY contains localhost
 
[!] Connected device                          

Here's some extra information:

  • we're able to build the same code again with --split-debug-info debug_info, so we also have the debug symbols locally
  • for every crash the pc value is different. This makes debugging following this guide quite difficult.

Do you guys have any ideas what I could try next to find out what's causing the crash?

@TahaTesser TahaTesser added the in triage Presently being triaged by the triage team label Apr 8, 2021
@TahaTesser
Copy link
Member

Hi @julianzur
You seem to be using an older version of Flutter

Can you please upgrade to the latest stable and try again
flutter channel stable
flutter upgrade --force
flutter doctor -v

Unfortunately, you would need to use the latest stable and supported Gradle plugin com.android.tools.build:gradle:4.1.0 for any fixes if it is a reproducible a bug on the latest stable

we're able to reproduce the problem on the Android armv7 emulator

If the problem persists, can you please provide complete reproducible steps and updated flutter doctor -v
Thank you

@TahaTesser TahaTesser added the waiting for customer response The Flutter team cannot make further progress on this issue until the original reporter responds label Apr 8, 2021
@julianzur
Copy link
Author

julianzur commented Apr 8, 2021

Hi @TahaTesser
thanks for your quick reply! I can totally see that you encourage us to update to the latest versions of flutter and gradle. We're actively working on updating to the latest versions for the upcoming release of our app - but as you can imagine this is not a trivial task.
Until we have been able to lift our codebase fully to the latest and greatest: is there any help you could provide that would help us pin down the root cause of the crash?

@no-response no-response bot removed the waiting for customer response The Flutter team cannot make further progress on this issue until the original reporter responds label Apr 8, 2021
@TahaTesser TahaTesser added a: release Challenges faced when attempting to productionize an app platform-android Android applications specifically c: crash Stack traces logged to the console will need additional triage This issue or PR needs attention before being routed to secondary triage a: production Issues experienced in live production apps engine flutter/engine repository. See also e: labels. and removed in triage Presently being triaged by the triage team labels Apr 9, 2021
@mraleph
Copy link
Member

mraleph commented Apr 9, 2021

If you can reproduce it then you best bet is to run it under gdb and see where exactly it crashes. I have symbolised the second stack from your issue and it does not make much sense:

crash from #80043 (comment) symbolized using symbols for a1440ca392ca23e874a105c5f3248b495bd0e247 android-arm-release

#00 000b2394 <...>/lib/arm/libflutter.so (offset 0x131000)
                                         SkShaderBase::ApplyMatrix(skvm::Builder*, SkMatrix const&, skvm::Coord, skvm::Uniforms*)
                                         third_party/skia/src/shaders/SkShader.cpp:250:11
#01 0212e05c <...>/lib/arm/libapp.so (offset 0xb000) (_kDartIsolateSnapshotInstructions+34746460)

@julianzur
Copy link
Author

Thanks @mraleph. We'll try again with gdb and come back to you.

@chinmaygarde
Copy link
Member

As @mraleph said, the top two frame don't make much sense and without a repro case, this issue isn't currently actionable.

@chinmaygarde chinmaygarde added P2 Important issues not at the top of the work list waiting for customer response The Flutter team cannot make further progress on this issue until the original reporter responds c: fatal crash Crashes that terminate the process and removed c: crash Stack traces logged to the console labels Apr 12, 2021
@csells csells added P2 customer: vroom and removed P2 Important issues not at the top of the work list labels Apr 12, 2021
@julianzur
Copy link
Author

julianzur commented Apr 15, 2021

Here's a quick update: we've identified the commit that breaks armv7 for us. It introduces the emojis 0.9.3 package to our project. I'm currently working on a small reproducible case. But so far the minimal code I wrote does not seem to reproduce the crash. We'll keep on investigating in which scenario the emojis library causes harm to our code (or if another change in that PR broke our code).

@no-response no-response bot removed the waiting for customer response The Flutter team cannot make further progress on this issue until the original reporter responds label Apr 15, 2021
@TahaTesser TahaTesser removed the will need additional triage This issue or PR needs attention before being routed to secondary triage label Apr 15, 2021
@TahaTesser
Copy link
Member

@julianzur
emojis is a third-party package, please file an issue on their dedicated GitHub

For issue related to Flutter itself, please provide a minimal reproducible code sample that does not use 3rd party plugin
Thank you

@TahaTesser TahaTesser added the waiting for customer response The Flutter team cannot make further progress on this issue until the original reporter responds label Apr 15, 2021
@mraleph
Copy link
Member

mraleph commented Apr 16, 2021

Based on my analysis this is most likely caused by the AOT backend patching an out-of-range relocation for a PC-relative call when handling backwards calls (calls to already emitted functions), instead of using a forward call through a trampoline. Assigning to @mkustermann

@mraleph
Copy link
Member

mraleph commented Apr 16, 2021

One could you the following to reproduce the bug:

// /tmp/test-gen.dart
import 'dart:io';

void main(List<String> args) async {
  final out = File('/tmp/test.dart').openWrite();

  // Note: everything is written into one line because otherwise gen_snapshot
  // spends most of the time dealing with line number information
  out.write(
      ''' @pragma('vm:never-inline') void f0(List<int> a) { a.add(1); } ''');

  final N = 10000;

  for (var i = 1; i <= N; i++) {
    out.write(
        ''' @pragma('vm:never-inline') void f$i(List<int> a) { f0(a); f${i - 1}(a);''');
    for (var j = 0; j < 100; j++) {
      out.write('''f0([]);''');
    }
    out.write('''}''');
  }

  out.write('''void main() { f${N}([]); f0([]); print('ok'); }''');

  await out.flush();
  await out.close();
  print('generated input file with ${N} methods');
}
$ dart /tmp/test-gen.dart && DART_CONFIGURATION=ReleaseSIMARM pkg/vm/tool/precompiler2 /tmp/test.dart /tmp/test.dart.aot  && echo "executing" && out/ReleaseSIMARM/dart_precompiled_runtime /tmp/test.dart.aot
generated input file with 10000 methods
executing

===== CRASH =====
si_signo=Segmentation fault(11), si_code=1, si_addr=0xf8f4d7a4
version=2.14.0-edge.7515fe5fe77d7ae204233f2d936fd947acde765c (be) (Thu Apr 15 10:33:47 2021 +0000) on "linux_simarm"
pid=3754136, thread=3754136, isolate_group=main(0x1bd2c00), isolate=main(0x1bf8480)
isolate_instructions=f4f11000, vm_instructions=f4f07000
Stack dump aborted because GetAndValidateThreadStackBounds failed.
fish: Job 1, 'out/ReleaseSIMARM/dart_precompi…' terminated by signal SIGABRT (Abort)

@TahaTesser TahaTesser added passed first triage and removed waiting for customer response The Flutter team cannot make further progress on this issue until the original reporter responds labels Apr 16, 2021
@mkustermann
Copy link
Member

I've uploaded a fix cl/195682 for this.

dart-bot pushed a commit to dart-lang/sdk that referenced this issue Apr 19, 2021
… acount

There was a missing check for whether a backwards call is out-of-range.

The CL also removes leftover code related to [max_offset_into_target_] -
which wasn't used. The intended use is already over approximated via
[max_instructions_size_].

The CL makes the pc-relative call/tail-call distances pluggable so a
newly added test can modify them for testing in-range/out-of-range
forward/backwards calls.

TEST=vm/cc/CodeRelocator_*

Fixes flutter/flutter#80043

Change-Id: Id4bdb7176108b61235dafb7ffc125da4a2bf07fa
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/195682
Commit-Queue: Martin Kustermann <kustermann@google.com>
Reviewed-by: Vyacheslav Egorov <vegorov@google.com>
@chinmaygarde
Copy link
Member

The patch has landed in the Dart VM. Per Ray, this will fixed in the next release.

@mkustermann
Copy link
Member

Merge-to-stable request is here dart-lang/sdk#45755

dart-bot pushed a commit to dart-lang/sdk that referenced this issue Apr 26, 2021
… acount

There was a missing check for whether a backwards call is out-of-range.

The CL also removes leftover code related to [max_offset_into_target_] -
which wasn't used. The intended use is already over approximated via
[max_instructions_size_].

The CL makes the pc-relative call/tail-call distances pluggable so a
newly added test can modify them for testing in-range/out-of-range
forward/backwards calls.

TEST=vm/cc/CodeRelocator_*

Fixes flutter/flutter#80043

Change-Id: Id4bdb7176108b61235dafb7ffc125da4a2bf07fa
Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/195682
Commit-Queue: Martin Kustermann <kustermann@google.com>
Reviewed-by: Vyacheslav Egorov <vegorov@google.com>
@github-actions
Copy link

github-actions bot commented Aug 2, 2021

This thread has been automatically locked since there has not been any recent activity after it was closed. If you are still experiencing a similar issue, please open a new bug, including the output of flutter doctor -v and a minimal reproduction of the issue.

@github-actions github-actions bot locked as resolved and limited conversation to collaborators Aug 2, 2021
@flutter-triage-bot flutter-triage-bot bot added P0 Critical issues such as a build break or regression and removed P2 labels Jun 28, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
a: production Issues experienced in live production apps a: release Challenges faced when attempting to productionize an app c: fatal crash Crashes that terminate the process customer: vroom engine flutter/engine repository. See also e: labels. P0 Critical issues such as a build break or regression platform-android Android applications specifically
Projects
None yet
Development

No branches or pull requests

6 participants