[sentry] Missing guide on configuration/use for production builds (--obfuscate --split-debug-info)

[binaris-no]

The first time I set it up and sometime after it was reporting errors just fine. All I get now when errors get captured is:

NoSuchMethodError: The method '' was called on null.
Receiver: null
Tried calling: ()
...
unparsed in Warning: This VM has been configured to produce stack traces that violate the Dart standard.
...

This was a sort of a surprise since I haven't changed anything directly related to Sentry / its config within or outside of the app.

Update (3 Jul 2020):
This seems to be the outcome of moving to builds produced with --obfuscate --split-debug-info=somefolder

So this issue is likely a question to which I haven't found a good and complete answer yet. How does one correctly configure/use Sentry for iOS and Android apps that use the flags above? Those are or should be used for production releases.

(At least a few of the events on top came from a Google Pixel device with Android 10 on it.)

Environment:

sentry 3.0.1

Flutter 1.19.0-4.1.pre • channel beta • https://github.com/flutter/flutter.git
Framework • revision f994b76 (3 days ago) • 2020-06-09 15:53:13 -0700
Engine • revision 9a28c3bcf4
Tools • Dart 2.9.0 (build 2.9.0-14.1.beta)

[✓] Flutter (Channel beta, 1.19.0-4.1.pre, on Mac OS X 10.15.5 19F101, locale en-NO)
[✓] Android toolchain - develop for Android devices (Android SDK version 29.0.3)
[✓] Xcode - develop for iOS and macOS (Xcode 11.5)
[✓] Chrome - develop for the web
[!] Android Studio (version 4.0)
✗ Flutter plugin not installed; this adds Flutter specific functionality.
✗ Dart plugin not installed; this adds Dart specific functionality.
[✓] VS Code (version 1.46.0)
[✓] Connected device (2 available)

[binaris-no]

Seems like the same problem with iOS...

[binaris-no]

Does this have to do with obfuscation maybe?
We use --obfuscate --split-debug-info=somefolder
when building for both Android and iOS.

I couldn't find any specific info on using Sentry and Flutter re. obfuscation, also nothing about uploading the .symbols files (app.android-arm64.symbols, app.ios-arm64.symbols, app.ios-armv7.symbols).

[diegopq]

I have a similar error

I have not found the error, I do not know why it occurs. also use the --obfuscate --split-debug-info = flag

[njovy]

I am having this problem as well on iOS with enabling DART_OBFUSCATION. I am not using Sentry though.

This VM has been configured to produce stack traces that violate the Dart standard is printed out.

[matpag]

Same for me, when building iOS and Android release apps with --obfuscate --split-debug-info I get this error when I open the app with Flutter 1.18.0-11.1.pre

E/flutter: [ERROR:flutter/lib/ui/ui_dart_state.cc(166)] Unhandled Exception: NoSuchMethodError: The method '<optimized out>' was called on null.
Receiver: null
Tried calling: <optimized out>()
Warning: This VM has been configured to produce stack traces that violate the Dart standard.

I'm not using Sentry, I'm just building and running the app on a physical device

[SeanZom]

I have the same issue here when building a release with the --obfuscate flag, the caught errors are no longer readable in Sentry dashboard

[bruno-garcia]

The Sentry dart/flutter SDK uses the stacktrace provided by the app, at runtime.
When obfuscation is selected, Flutter will not make the original function names available in the app anymore. That is the goal of obfuscation to make it harder for someone to reverse engineer your app.

It looks like we'll need to add support for this. To allow uploading the generated symbols and once a crash comes in, symbolicate it to show you the original source code. Sentry already does similar for for JS/Dart sourcemaps, proguard, PDBs, etc.

Quick looked up and it seems it uses DWARF symbols which Sentry already has support. So that's good news.

[ganeshrvel]

any updates on this?

[avvari-da]

Any update on this? Possible to upload symbols files?

[zoechi]

@bruno-garcia uploading the symbols does not seem to do anything yet

flutter build apk --obfuscate --split-debug-info=sym
sentry-cli --auth-token <token> upload-dif --org my_org --project sentry_debug sym

The Sentry event shows

RangeError: RangeError (index): Invalid value: Only valid value is 0: 1
  File "unparsed", in Warning: This VM has been configured to produce stack traces that violate the Dart standard.
  File "unparsed", in *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
  File "unparsed", in pid: 15986, tid: 16159, name 2.ui
  File "unparsed", in isolate_dso_base: 7d0040b000, vm_dso_base: 7d0040b000
  File "unparsed", in isolate_instructions: 7d0041b000, vm_instructions: 7d0040d000
  File "unparsed", in     #00 abs 0000007d0066e893 virt 0000000000263893 _kDartIsolateSnapshotInstructions+0x253893
  File "unparsed", in     #01 abs 0000007d0069201f virt 000000000028701f _kDartIsolateSnapshotInstructions+0x27701f
  File "unparsed", in     #02 abs 0000007d005bcb37 virt 00000000001b1b37 _kDartIsolateSnapshotInstructions+0x1a1b37
  File "unparsed", in     #03 abs 0000007d0069229b virt 000000000028729b _kDartIsolateSnapshotInstructions+0x27729b
  File "<asynchronous suspension>"
  File "unparsed", in #04 abs 0000007d00693f17 virt 0000000000288f17 _kDartIsolateSnapshotInstructions+0x278f17
  File "<asynchronous suspension>"
  File "unparsed"

When I copy that into a local file strace.txt and run

flutter symbolize -i strace.txt -d sym/app.android-arm64.symbols

results in

RangeError: RangeError (index): Invalid value: Only valid value is 0: 1
  File "unparsed", in Warning: This VM has been configured to produce stack traces that violate the Dart standard.
  File "unparsed", in *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
  File "unparsed", in pid: 15986, tid: 16159, name 2.ui
  File "unparsed", in isolate_dso_base: 7d0040b000, vm_dso_base: 7d0040b000
  File "unparsed", in isolate_instructions: 7d0041b000, vm_instructions: 7d0040d000
  File "unparsed", in #0      _GrowableList.[] (dart:core-patch/growable_array.dart:177)
  File "unparsed", in #1      FlutterWebviewPlugin.getCookies.<anonymous closure> (package:flutter_webview_plugin/src/base.dart:305)
  File "unparsed", in #2      _GrowableList.forEach (dart:core-patch/growable_array.dart:313)
  File "unparsed", in #3      FlutterWebviewPlugin.getCookies (package:flutter_webview_plugin/src/base.dart:303)
  File "<asynchronous suspension>"
  File "unparsed", in #04 abs 0000007d00693f17 virt 0000000000288f17 _kDartIsolateSnapshotInstructions+0x278f17
  File "<asynchronous suspension>"
  File "unparsed"

What do you think is missing that Sentry uses the uploaded symbols to deobfuscate the stack trace?

[bruno-garcia]

@zoechi
The uploaded DWARF file should show up here: https://sentry.io/settings/<org-slug>/projects/<project-slug>/debug-symbols/ like:

Can you confirm it's there?

To get it to work we will also need to set the platform field of the frame to native to trigger the symbolication.

Also add the debug_meta in the payload, including the code_id of the DWARF file involved.
The payload looks like this:

The docs for it are here: https://develop.sentry.dev/sdk/event-payloads/debugmeta

It's similar to what a iOS crash works.

It will show up like this in the event's view:

[zoechi]

@bruno-garcia

Can you confirm it's there?

Yup, they are there

[zoechi]

@jonahwilliams I have seen creating a few PRs related to build-id (NT_GNU_BUILD_ID, .note.gnu.build-id, LC_UUID).

What I think is missing is a way to get the build-id at runtime to send it with an Sentry event to the Sentry service, so that debug symbols and stack traces can be associated.

Do you know if there is a way to get this build-id at runtime in Flutter?

If not, I guess this would need to be a build step to add an asset file containing the info
or modifying the APK after the build step. I haven't looked into iOS yet.

See also code_id and debug_id in https://develop.sentry.dev/sdk/event-payloads/debugmeta/ in ELF Images and MachO Images section.

[jonahwilliams]

I would be surprised if that info was available at runtime, but if there is something the tool could do to make it available during the build perhaps that is a better route?

Also FYI @sstrickl

[sstrickl]

Unfortunately it's not currently available at runtime, though I actually would like it to be! (Right now, we'd either have to extend the embedder interface or figure out a similar workaround as we did for BSS sections, but we can't take the same approach as we've already used up all the space in the Image header on 64-bit platforms.)

There's a known symbol for the build ID in the dynamic symbol table of ELF snapshots (_kDartSnapshotBuildId), but for assembly snapshots, we don't currently create one ourselves, but instead the assembler does and so I don't know what symbol name the assemblers we use create for it by default. I also haven't checked yet if the assembler will omit its own if we create one manually in the output that's assembled, or if there'd need to be specific flags to keep it from doing so.

[jonahwilliams]

@sstrickl are there bugs on the dart-sdk that could be followed for these features?

[fzyzcjy]

Hi how is everything going? Will we have the feature in the near future?

[bruno-garcia]

Currently we're blocked on being able to get the debug_id to report to the server: dart-lang/sdk#43274

[fzyzcjy]

@bruno-garcia Ah... So do I understand correctly: We should not use obfuscate currently, otherwise we will not get stacktraces and not work with Sentry. Thanks very much!

[marandaneto]

--split-debug-info should work on Android already if using https://pub.dev/packages/sentry_flutter you'd need to upload debug symbols though
iOS depends on dart-lang/sdk#43612

[naim5am]

--split-debug-info should work on Android already if using https://pub.dev/packages/sentry_flutter you'd need to upload debug symbols though
iOS depends on dart-lang/sdk#43612

Where? Upload where? Googling android upload debug symbols sentry returns nothing relevant. There's a document for iOS though.

Umm, on further digging, it seems I have to use sentry-cli. docs >> https://docs.sentry.io/product/cli/

it would help to have a link somewhere or even a document google can find

[marandaneto]

@naim5am

that's our official docs https://docs.sentry.io/platforms/flutter/ and here is the section for uploading debug symbols https://docs.sentry.io/platforms/flutter/usage/advanced-usage/#uploading-debug-symbols-android-and-ios

[fzyzcjy]

Hi, from dart-lang/sdk#43612, it is said that

You can still manually symbolicate and deobfuscate your crashes, we emit enough information for you to do it.

Therefore, maybe this is possible on iOS? (For example, after we upload "enough information" to Sentry and sentry should be able to do it?) I am not very familiar with iOS...

[marandaneto]

@fzyzcjy afaik, Flutter generates a Linux executable (ELF file) when splitting debug info instead of an iOS one. It should create the right file.
That's why flutter symbolize works but no other tolling does.
Once Flutter generates the right file format, Sentry will be able to symbolize those stack traces ootb.

[fzyzcjy]

@marandaneto Hmm so you mean flutter symbolize can work? then maybe sentry can use it?

[marandaneto]

@fzyzcjy yes and no, we think that the right approach is Flutter emitting the right file as it does for Android for example.
We already symbolicate iOS events when captured thru the iOS code, the problem is only when its captured thru the Dart layer.

[bruno-garcia]

As part of triaging general Sentry issues in the Flutter issue tracker we decided to close this issue in favor of an issue tracking on Sentry's repo. Dependencies are linked in there.

[fzyzcjy]

@marandaneto Hi, I have made an attempt on manually symbolize the stack, but it does not work :(

I manually trigger an error, and see:

https://user-images.githubusercontent.com/5236035/126902011-a7557f18-a0a8-48e7-93cc-206eff8c0632.png

I try to manually cook up some pseudo-traces that is acceptable by flutter symbolize:

    #00 abs c21f1bdb virt 005dfbdb _kDartIsolateSnapshotInstructions+0x5dfbdb
    #01 abs c1f4d4af virt 0033b4af _kDartIsolateSnapshotInstructions+0x33b4af
    #02 abs c1ce894f virt 000d694f _kDartIsolateSnapshotInstructions+0x0d694f

and even

    #00 abs 123 virt 123 _kDartIsolateSnapshotInstructions+0x5dfbdb

unfortunately, they output something that is completely unrelated!

#0      ExtEduClassVO|get#grade (package:yplusplus/misc/model_extensions_manual.dart:191:3)
#1      XYT.explain (package:si_core/src/phd/phd_format_misc.dart:28:41)
#2      RenderPadding.computeMinIntrinsicHeight (package:flutter/src/rendering/shifted_box.dart:188:21)

where the real thing lies in debug_page.dart.

I use dwarfdump and see:

0x001b3d78:   DW_TAG_subprogram
                DW_AT_abstract_origin	(0x000bf66f "__BodyState.build.<anonymous closure>")
                DW_AT_low_pc	(0x00000000005dfb18)
                DW_AT_high_pc	(0x00000000005dfbe0)

0x001b3d85:     DW_TAG_inlined_subroutine
                  DW_AT_abstract_origin	(0x00006727 "new DateTime.now")
                  DW_AT_low_pc	(0x00000000005dfb54)
                  DW_AT_high_pc	(0x00000000005dfb64)
                  DW_AT_call_file	("package:yplusplus/pages/debug_page.dart")
                  DW_AT_call_line	(119)
                  DW_AT_call_column	(102)

0x001b3d96:       DW_TAG_inlined_subroutine
                    DW_AT_abstract_origin	(0x0000673c "new DateTime._now")
                    DW_AT_low_pc	(0x00000000005dfb54)
                    DW_AT_high_pc	(0x00000000005dfb64)
                    DW_AT_call_file	("dart:core/date_time.dart")
                    DW_AT_call_line	(209)
                    DW_AT_call_column	(25)

0x001b3da7:         NULL

0x001b3da8:       NULL

0x001b3da9:     DW_TAG_inlined_subroutine
                  DW_AT_abstract_origin	(0x00005e4e "new Exception")
                  DW_AT_low_pc	(0x00000000005dfbbc)
                  DW_AT_high_pc	(0x00000000005dfbc8)
                  DW_AT_call_file	("package:yplusplus/pages/debug_page.dart")
                  DW_AT_call_line	(119)
                  DW_AT_call_column	(40)

0x001b3dba:       NULL

0x001b3dbb:     NULL

so the .symbols file should be correct - since the relative address does lie in debug_page.dart

but flutter symbolize just does not work :( what should I do? thank you very much!

p.s. that is android, but since ios does not have automatic symbolize in sentry, i am trying to solve it in the manual way as you suggest.

[t-kozak]

For me, I ended up printing the stack trace, as given by Flutter to the breadcrumbs just before reporting the crash with sentry. For me the whole report like this:

Got an not caught exception --  (null): Null check operator used on a null value 
Warning: This VM has been configured to produce stack traces that violate the Dart standard.
*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
pid: 34543, tid: 6136770560, name io.flutter.1.ui
isolate_dso_base: 109000000, vm_dso_base: 109000000
isolate_instructions: 109009000, vm_instructions: 109004000
    #00 abs 00000001090b6763 _kDartIsolateSnapshotInstructions+0xad763
    #01 abs 00000001090b6abb _kDartIsolateSnapshotInstructions+0xadabb
    #02 abs 00000001092e6ba7 _kDartIsolateSnapshotInstructions+0x2ddba7
    #03 abs 00000001092e6fab _kDartIsolateSnapshotInstructions+0x2ddfab
    #04 abs 00000001092e6f53 _kDartIsolateSnapshotInstructions+0x2ddf53
    #05 abs 00000001090127cf _kDartIsolateSnapshotInstructions+0x97cf
    #06 abs 0000000109012e87 _kDartIsolateSnapshotInstructions+0x9e87
    #07 abs 0000000109012c3f _kDartIsolateSnapshotInstructions+0x9c3f
    #08 abs 000000010900cbbb _kDartIsolateSnapshotInstructions+0x3bbb
    #09 abs 00000001090116a7 _kDartIsolateSnapshotInstructions+0x86a7
    #10 abs 00000001092212af _kDartIsolateSnapshotInstructions+0x2182af
    #11 abs 00000001090133df _kDartIsolateSnapshotInstructions+0xa3df
    #12 abs 00000001093cb303 _kDartIsolateSnapshotInstructions+0x3c2303
    #13 abs 00000001093cbca7 _kDartIsolateSnapshotInstructions+0x3c2ca7
    #14 abs 00000001093cb9db _kDartIsolateSnapshotInstructions+0x3c29db
    #15 abs 0000000109013443 _kDartIsolateSnapshotInstructions+0xa443
    #16 abs 00000001093cb303 _kDartIsolateSnapshotInstructions+0x3c2303
    #17 abs 00000001093ca627 _kDartIsolateSnapshotInstructions+0x3c1627
    #18 abs 00000001090138ef _kDartIsolateSnapshotInstructions+0xa8ef
    #19 abs 000000010900f4bf _kDartIsolateSnapshotInstructions+0x64bf
    #20 abs 000000010900f253 _kDartIsolateSnapshotInstructions+0x6253
    #21 abs 000000010900e5bf _kDartIsolateSnapshotInstructions+0x55bf

does work with flutter symbolize. I get that from:

@override
  void e(
    final String msg, {
    final Object? error,
    final bool report = false,
    final String? culprit,
    final StackTrace? stackTrace,
  }) {
    final errMsg = error?.toString() ?? '';

    var stackString;
    if (stackTrace != null) {
      stackString = '\n$stackTrace';
    } else if (error is Error) {
      stackString = '\n${error.stackTrace}';
    } else {
      stackString = '';
    }
    output(LogEntry(category, FoodieLogLevel.error, DateTime.now(),
        '$msg ($culprit): $errMsg $stackString'));
  }

You use the Log.e in the following way:

runZonedGuarded(() => runApp(FoodieApp()), errorHandler);
  FlutterError.onError = (details, {bool forceReport = false}) =>
      errorHandler(details.exception, details.stack);
// ...

void errorHandler(err, stack) =>
    createLogFactory().getLogger().e('Got an not caught exception -- ',
        error: err, stackTrace: stack, report: true);

[fzyzcjy]

@t-kozak Thanks for the suggestion! However, I am curious: why my manually cooked up pseudo-stacktrace does not work, given that the address seems to be correct...

[fzyzcjy]

oh i see...

what sentry captures:

raw stacktrace：

FlutterError.onError raw stackTrace=Warning: This VM has been configured to produce stack traces that violate the Dart standard.
*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
pid: 28642, tid: 28673, name 1.ui
build_id: '4ff9bfa06d864cee918d3ff750b69494'
isolate_dso_base: c1a55000, vm_dso_base: c1a55000
isolate_instructions: c1a61000, vm_instructions: c1a57000
    #00 abs c205ee87 virt 00609e87 _kDartIsolateSnapshotInstructions+0x5fde87
    #01 abs c1d9db43 virt 00348b43 _kDartIsolateSnapshotInstructions+0x33cb43
    #02 abs c1b2b453 virt 000d6453 _kDartIsolateSnapshotInstructions+0xca453

[fzyzcjy]

Seems that the real useful address is the 0x5fde87 in _kDartIsolateSnapshotInstructions+0x5fde87, but Sentry only provides the address in the virt part (or abs part).

@marandaneto Hi, is this a bug of Sentry? could this be fixed? Thanks!

[github-actions]

This thread has been automatically locked since there has not been any recent activity after it was closed. If you are still experiencing a similar issue, please open a new bug, including the output of flutter doctor -v and a minimal reproduction of the issue.