Support symbolicating native call stacks with --split-debug-info

[xster]

Downstream feature request: flutter/flutter#60189

#35851 now supports translating obfuscated Dart stacks into the original stack, but as far as I understand, after applying --split-debug-info, we no longer have enough symbols left in the binary to then generate Xcode Debugging Symbols (dSym) from the binary/framework. This means users can no longer symbolize native call stacks into Dart call stacks.

Would it be possible for --split-debug-info to output both a obfuscated Dart -> Dart map as well as a native -> Dart map?

cc @sstrickl and @mraleph

[mkustermann]

/cc @sstrickl

[sstrickl]

So the separate debugging information (which is in ELF format) contains both the symbols and the DWARF information to symbolicate non-symbolic Dart stack traces. Currently, one way to do so is to use pkg/native_stack_traces.

However, I'm not sure if an easy way to convert ELF files containing static symbol tables and DWARF information to a dSym file (which I'm guessing is the Mach-O equivalent of having a separate file with just debugging information similar to our ELF one) already exists. I'll look into this.

(I'd be happy to just make a Mach-O output instead, but it seems like documentation of the Mach-O format was pulled at some point, so I'm not sure how stable it is outside of "what XCode produces" nowadays.)

[jan-auer]

@sstrickl are you saying that you're generating ELF containers even for iOS / macOS builds?

File formats like MachO and ELF are usually meant to be platform-dependent, so I'm not sure if it makes sense to even convert an ELF file into a MachO file. Many debuggers and symbolizers, including llvm-symbolizer and addr2line are able to read both file formats, but for their respective platform (ELF for Linux and MachO for Apple). The DWARF information contained in the debug sections is equal in its format.

Usually, on iOS builds you would output the entire debugging information into the __debug_* sections of the binary and then have the build toolchain run dsymutil to split it out into the debug companion. This takes care of creating a reference in the binary and moves all relevant sections and load commands over. Note that the convention for section naming on macOS is with two leading underscores instead of a dot.

Another important point is to create a LC_UUID load command before invoking dsymutil. It is essentially the same as the .gnu.build-id section in ELF and allows debuggers to identify a debug companion.

Edit: I just had a look at an iOS build and it is indeed emitting ELF for iOS builds. It would definitely be better to follow the above approach, since that allows tools like XCode to use the debug information. The standard command line utilities for Darwin development by Apple are not able to process ELF.

[fzyzcjy]

Hi is there any updates? I want to obfuscate my flutter code in release build, and I definitely want to see error stack trace

[mraleph]

We are not actively working on this at the moment, because high priority uses cases are all covered to the best of our knowledge.

I want to obfuscate my flutter code in release build, and I definitely want to see error stack trace

You can still manually symbolicate and deobfuscate your crashes, we emit enough information for you to do it.

[mraleph]

I just want to follow up on this given that @bruno-garcia indicated on getsentry/sentry-dart#444 that this is an important issue for Sentry team to resolve.

To the best of my knowledge you have all necessary pieces to make this work, though it would require some changes to the Flutter CLI. We generate Mach-O through XCode toolchain, so I suggest we go the same route for dSYMs. Here is what I suggest we do: instead of using gen_snapshot's --save-debugging-info we instead shell out to dsymtool to do the job. The code that needs to be altered resides around these lines.

Here is quick and dirty prototype, which would produce necessary dSYM files:

diff --git a/packages/flutter_tools/lib/src/base/build.dart b/packages/flutter_tools/lib/src/base/build.dart
index 74e40a37a9..7c9c5285ce 100644
--- a/packages/flutter_tools/lib/src/base/build.dart
+++ b/packages/flutter_tools/lib/src/base/build.dart
@@ -138,9 +138,15 @@ class AOTSnapshotter {
       '--deterministic',
     ];
 
-    // We strip snapshot by default, but allow to suppress this behavior
-    // by supplying --no-strip in extraGenSnapshotOptions.
-    bool shouldStrip = true;
+    final bool targetingApplePlatform =
+        platform == TargetPlatform.ios || platform == TargetPlatform.darwin_x64;
+    _logger.printTrace("targetingApplePlatform = ${targetingApplePlatform}");
+    final bool shouldSplitDebugInfo = splitDebugInfo?.isNotEmpty ?? false;
+
+    // We strip snapshot by default unless we need to split a dSYM out, but
+    // we allow to suppress stripping by supplying --no-strip in
+    // extraGenSnapshotOptions.
+    bool shouldStrip = !(targetingApplePlatform && shouldSplitDebugInfo);
 
     if (extraGenSnapshotOptions != null && extraGenSnapshotOptions.isNotEmpty) {
       _logger.printTrace('Extra gen_snapshot options: $extraGenSnapshotOptions');
@@ -186,7 +192,6 @@ class AOTSnapshotter {
     // multiple debug files.
     final String archName = getNameForTargetPlatform(platform, darwinArch: darwinArch);
     final String debugFilename = 'app.$archName.symbols';
-    final bool shouldSplitDebugInfo = splitDebugInfo?.isNotEmpty ?? false;
     if (shouldSplitDebugInfo) {
       _fileSystem.directory(splitDebugInfo)
         .createSync(recursive: true);
@@ -197,7 +202,11 @@ class AOTSnapshotter {
       // Faster async/await
       if (shouldSplitDebugInfo) ...<String>[
         '--dwarf-stack-traces',
-        '--save-debugging-info=${_fileSystem.path.join(splitDebugInfo, debugFilename)}'
+        // --save-debugging-info produces non-native symbol format.
+        // NOTE: this will  break `flutter symbolize` so it needs to be changed
+        // to use `atos`.
+        if (!targetingApplePlatform)
+          '--save-debugging-info=${_fileSystem.path.join(splitDebugInfo, debugFilename)}'
       ],
       if (dartObfuscation)
         '--obfuscate',
@@ -218,7 +227,7 @@ class AOTSnapshotter {
 
     // On iOS and macOS, we use Xcode to compile the snapshot into a dynamic library that the
     // end-developer can link into their app.
-    if (platform == TargetPlatform.ios || platform == TargetPlatform.darwin_x64) {
+    if (targetingApplePlatform) {
       final RunResult result = await _buildFramework(
         appleArch: darwinArch,
         isIOS: platform == TargetPlatform.ios,
@@ -227,6 +236,7 @@ class AOTSnapshotter {
         outputPath: outputDir.path,
         bitcode: bitcode,
         quiet: quiet,
+        splitDebugInfo: shouldSplitDebugInfo ? splitDebugInfo : null,
       );
       if (result.exitCode != 0) {
         return result.exitCode;
@@ -244,7 +254,8 @@ class AOTSnapshotter {
     @required String assemblyPath,
     @required String outputPath,
     @required bool bitcode,
-    @required bool quiet
+    @required bool quiet,
+    @required String splitDebugInfo,
   }) async {
     final String targetArch = getNameForDarwinArch(appleArch);
     if (!quiet) {
@@ -298,8 +309,17 @@ class AOTSnapshotter {
     final RunResult linkResult = await _xcode.clang(linkArgs);
     if (linkResult.exitCode != 0) {
       _logger.printError('Failed to link AOT snapshot. Linker terminated with exit code ${compileResult.exitCode}');
+      return linkResult;
+    }
+    if (splitDebugInfo == null) {
+      return linkResult;
+    }
+    final String dSYMPath = _fileSystem.path.join(splitDebugInfo, targetArch, 'App.dSYM');
+    final RunResult dsymutilResult = await _xcode.dsymutil(['-o', dSYMPath, appLib]);
+    if (dsymutilResult.exitCode != 0) {
+      _logger.printError('Failed to generate dSYM. dsymutil terminated with exit code ${compileResult.exitCode}');
     }
-    return linkResult;
+    return dsymutilResult;
   }
 
   bool _isValidAotPlatform(TargetPlatform platform, BuildMode buildMode) {
diff --git a/packages/flutter_tools/lib/src/macos/xcode.dart b/packages/flutter_tools/lib/src/macos/xcode.dart
index a181927b74..be869d61d5 100644
--- a/packages/flutter_tools/lib/src/macos/xcode.dart
+++ b/packages/flutter_tools/lib/src/macos/xcode.dart
@@ -192,6 +192,15 @@ class Xcode {
     );
   }
 
+
+  Future<RunResult> dsymutil(List<String> args) {
+    return _processUtils.run(
+      <String>[...xcrunCommand(), 'dsymutil', ...args],
+      throwOnError: true,
+    );
+  }
+
+
   Future<String> sdkLocation(EnvironmentType environmentType) async {
     assert(environmentType != null);
     final RunResult runResult = await _processUtils.run(

This breaks flutter symbolize though which only works with ELF inputs, so that probably needs to be updated to shell out to atos (which probably requires some minor refactoring work in native_stack_traces package to make DwarfStackTraceDecoder into something which takes a debug-info format agnostic symboliser).

@bruno-garcia do you think you (or somebody else from Sentry) will be able to take over this from here and land necessary changes in Flutter CLI and Dart's native_stack_traces package?

[TeeMoYan]

I just want to follow up on this given that @bruno-garcia indicated on getsentry/sentry-dart#444 that this is an important issue for Sentry team to resolve.

To the best of my knowledge you have all necessary pieces to make this work, though it would require some changes to the Flutter CLI. We generate Mach-O through XCode toolchain, so I suggest we go the same route for dSYMs. Here is what I suggest we do: instead of using gen_snapshot's --save-debugging-info we instead shell out to dsymtool to do the job. The code that needs to be altered resides around these lines.

Here is quick and dirty prototype, which would produce necessary dSYM files:

diff --git a/packages/flutter_tools/lib/src/base/build.dart b/packages/flutter_tools/lib/src/base/build.dart
index 74e40a37a9..7c9c5285ce 100644
--- a/packages/flutter_tools/lib/src/base/build.dart
+++ b/packages/flutter_tools/lib/src/base/build.dart
@@ -138,9 +138,15 @@ class AOTSnapshotter {
       '--deterministic',
     ];
 
-    // We strip snapshot by default, but allow to suppress this behavior
-    // by supplying --no-strip in extraGenSnapshotOptions.
-    bool shouldStrip = true;
+    final bool targetingApplePlatform =
+        platform == TargetPlatform.ios || platform == TargetPlatform.darwin_x64;
+    _logger.printTrace("targetingApplePlatform = ${targetingApplePlatform}");
+    final bool shouldSplitDebugInfo = splitDebugInfo?.isNotEmpty ?? false;
+
+    // We strip snapshot by default unless we need to split a dSYM out, but
+    // we allow to suppress stripping by supplying --no-strip in
+    // extraGenSnapshotOptions.
+    bool shouldStrip = !(targetingApplePlatform && shouldSplitDebugInfo);
 
     if (extraGenSnapshotOptions != null && extraGenSnapshotOptions.isNotEmpty) {
       _logger.printTrace('Extra gen_snapshot options: $extraGenSnapshotOptions');
@@ -186,7 +192,6 @@ class AOTSnapshotter {
     // multiple debug files.
     final String archName = getNameForTargetPlatform(platform, darwinArch: darwinArch);
     final String debugFilename = 'app.$archName.symbols';
-    final bool shouldSplitDebugInfo = splitDebugInfo?.isNotEmpty ?? false;
     if (shouldSplitDebugInfo) {
       _fileSystem.directory(splitDebugInfo)
         .createSync(recursive: true);
@@ -197,7 +202,11 @@ class AOTSnapshotter {
       // Faster async/await
       if (shouldSplitDebugInfo) ...<String>[
         '--dwarf-stack-traces',
-        '--save-debugging-info=${_fileSystem.path.join(splitDebugInfo, debugFilename)}'
+        // --save-debugging-info produces non-native symbol format.
+        // NOTE: this will  break `flutter symbolize` so it needs to be changed
+        // to use `atos`.
+        if (!targetingApplePlatform)
+          '--save-debugging-info=${_fileSystem.path.join(splitDebugInfo, debugFilename)}'
       ],
       if (dartObfuscation)
         '--obfuscate',
@@ -218,7 +227,7 @@ class AOTSnapshotter {
 
     // On iOS and macOS, we use Xcode to compile the snapshot into a dynamic library that the
     // end-developer can link into their app.
-    if (platform == TargetPlatform.ios || platform == TargetPlatform.darwin_x64) {
+    if (targetingApplePlatform) {
       final RunResult result = await _buildFramework(
         appleArch: darwinArch,
         isIOS: platform == TargetPlatform.ios,
@@ -227,6 +236,7 @@ class AOTSnapshotter {
         outputPath: outputDir.path,
         bitcode: bitcode,
         quiet: quiet,
+        splitDebugInfo: shouldSplitDebugInfo ? splitDebugInfo : null,
       );
       if (result.exitCode != 0) {
         return result.exitCode;
@@ -244,7 +254,8 @@ class AOTSnapshotter {
     @required String assemblyPath,
     @required String outputPath,
     @required bool bitcode,
-    @required bool quiet
+    @required bool quiet,
+    @required String splitDebugInfo,
   }) async {
     final String targetArch = getNameForDarwinArch(appleArch);
     if (!quiet) {
@@ -298,8 +309,17 @@ class AOTSnapshotter {
     final RunResult linkResult = await _xcode.clang(linkArgs);
     if (linkResult.exitCode != 0) {
       _logger.printError('Failed to link AOT snapshot. Linker terminated with exit code ${compileResult.exitCode}');
+      return linkResult;
+    }
+    if (splitDebugInfo == null) {
+      return linkResult;
+    }
+    final String dSYMPath = _fileSystem.path.join(splitDebugInfo, targetArch, 'App.dSYM');
+    final RunResult dsymutilResult = await _xcode.dsymutil(['-o', dSYMPath, appLib]);
+    if (dsymutilResult.exitCode != 0) {
+      _logger.printError('Failed to generate dSYM. dsymutil terminated with exit code ${compileResult.exitCode}');
     }
-    return linkResult;
+    return dsymutilResult;
   }
 
   bool _isValidAotPlatform(TargetPlatform platform, BuildMode buildMode) {
diff --git a/packages/flutter_tools/lib/src/macos/xcode.dart b/packages/flutter_tools/lib/src/macos/xcode.dart
index a181927b74..be869d61d5 100644
--- a/packages/flutter_tools/lib/src/macos/xcode.dart
+++ b/packages/flutter_tools/lib/src/macos/xcode.dart
@@ -192,6 +192,15 @@ class Xcode {
     );
   }
 
+
+  Future<RunResult> dsymutil(List<String> args) {
+    return _processUtils.run(
+      <String>[...xcrunCommand(), 'dsymutil', ...args],
+      throwOnError: true,
+    );
+  }
+
+
   Future<String> sdkLocation(EnvironmentType environmentType) async {
     assert(environmentType != null);
     final RunResult runResult = await _processUtils.run(

This breaks flutter symbolize though which only works with ELF inputs, so that probably needs to be updated to shell out to atos (which probably requires some minor refactoring work in native_stack_traces package to make DwarfStackTraceDecoder into something which takes a debug-info format agnostic symboliser).

@bruno-garcia do you think you (or somebody else from Sentry) will be able to take over this from here and land necessary changes in Flutter CLI and Dart's native_stack_traces package?

Modified Flutter CLI, build does not generate App.dSYM file?

[scott-the-brewer]

From the dart side, this is being worked on at https://dart-review.googlesource.com/c/sdk/+/242108

[mraleph]

Modified Flutter CLI, build does not generate App.dSYM file?

You mean you patched your Flutter with similar patch, run it and did not get dSYM? Did you make sure that Flutter CLI is recompiled by deleting bin/cache/flutter_tools.snapshot? If you did - could you provide verbose build output (with -v).

[TeeMoYan]

Modified Flutter CLI, build does not generate App.dSYM file?

You mean you patched your Flutter with similar patch, run it and did not get dSYM? Did you make sure that Flutter CLI is recompiled by deleting bin/cache/flutter_tools.snapshot? If you did - could you provide verbose build output (with -v).

Cool, clearing the cache this worked. already have a dSYM. thanks

[vaind]

A flutter tool PR draft that uses the native_stack_traces support for dSYM is at flutter/flutter#101586

[sstrickl]

Version 0.5.0 of pkg/native_stack_traces, which both handles dSYM and has better support for DWARF5 (which has some non-backwards compatible changes from earlier DWARF versions) has relanded in the Dart tree and been pushed to pub.

[mraleph]

I think @sstrickl has pushed it over the finish line and everything is working now as it should.