[LLDB][Minidump] Backtrace broken on minidumps when binary also loaded

[strangewiz]

After e9c8f75 [LLDB][Minidump] Have Minidumps save off and properly read TLS data (#109477), minidumps loaded with the binary adjacent cannot show a correct backtrace beyond the first frame.

e.g. we see:

(lldb) bt
* thread #1
  * frame #0: 0x000000018d8813c8
    frame #1: 0x000000018d760714

instead of:

(lldb) bt
* thread #1
  * frame #0: 0x000000018d8813c8 libsystem_kernel.dylib`__semwait_signal + 8
    frame #1: 0x000000018d760714 libsystem_c.dylib`nanosleep + 220
    frame #2: 0x000000018d7696f8 libsystem_c.dylib`sleep + 52
    frame #3: 0x0000000100373f40 simple`func4 + 16
    frame #4: 0x0000000100373f54 simple`func3 + 12
    frame #5: 0x0000000100373f68 simple`func2 + 12
    frame #6: 0x0000000100373f7c simple`func1 + 12
    frame #7: 0x0000000100373f90 simple`main + 12
    frame #8: 0x000000018d53c274 dyld`start + 2840

Steps to reproduce:

lldb built with:

 cmake -GNinja -DCMAKE_BUILD_TYPE=Debug -DLLVM_ENABLE_ASSERTIONS=ON       -DLLVM_ENABLE_PROJECTS='clang;lld;compiler-rt;lldb'       -DCMAKE_C_COMPILER=$CLANG_BASE_PATH/bin/clang       -DCMAKE_CXX_COMPILER=$CLANG_BASE_PATH/bin/clang++       -DCMAKE_ASM_COMPILER=$CLANG_BASE_PATH/bin/clang       -DCMAKE_ASM_COMPILER_ID="Clang"   -DLLVM_ENABLE_RUNTIMES="libcxx;libcxxabi"    -DCMAKE_OSX_SYSROOT=$PWD/../sysroot/MacOSX.sdk       -DDARWIN_macosx_CACHED_SYSROOT=$PWD/../sysroot/MacOSX.sdk       -DDARWIN_iphoneos_CACHED_SYSROOT=$PWD/../sysroot/iPhoneOS.sdk       -DDARWIN_iphonesimulator_CACHED_SYSROOT=$PWD/../sysroot/iPhoneSimulator.sdk       ../llvm

and:

 $ ninja lldb

at revision: e9c8f75

$ cd reproduce_folder
$ cat simple.c
#include <unistd.h>

[[clang::optnone]]
void func4() {
    sleep(1000);
}

[[clang::optnone]]
void func3() {
    func4();
}

[[clang::optnone]]
void func2() {
    func3();
}

[[clang::optnone]]
void func1() {
    func2();
}

int main() {
    func1();
    return 0;
}

$ clang -Os simple.c -o simple

$ ./simple &
[1] 88598

$ sudo generate_dump 88598
Password:

$ ~/llvm-project/build/bin/lldb
error: invalid value path 'symbols.swift-module-loading-mode'
(lldb) target create -c minidump.88598
warning: (arm64e) /usr/lib/system/libsystem_kernel.dylib address 0x000000018d8c5000 maps to more than one section: libsystem_kernel.dylib.__DATA_DIRTY and libdyld.dylib.__TEXT
warning: (arm64e) /usr/lib/system/libsystem_platform.dylib address 0x000000018d8f8000 maps to more than one section: libsystem_platform.dylib.__DATA_CONST and libsystem_info.dylib.__TEXT
Core file 'reproduce_folder/minidump.88598' (aarch64) was loaded.
(lldb) bt
* thread #1
  * frame #0: 0x000000018d8813c8
    frame #1: 0x000000018d760714
(lldb) ^D

// rebuild with DynamicLoader *GetDynamicLoader() override { return nullptr; }, or just reset to previous revision, 2ff4c25

diff --git a/lldb/source/Plugins/Process/minidump/ProcessMinidump.h b/lldb/source/Plugins/Process/minidump/ProcessMinidump.h
index 3d235670a3..83ff1512cc 100644
--- a/lldb/source/Plugins/Process/minidump/ProcessMinidump.h
+++ b/lldb/source/Plugins/Process/minidump/ProcessMinidump.h
@@ -53,6 +53,7 @@ public:

   Status DoLoadCore() override;

+  DynamicLoader *GetDynamicLoader() override { return nullptr; }
   // Returns AUXV structure found in the core file
   lldb_private::DataExtractor GetAuxvData() override;

// and recompile lldb

$ ~/work/llvm-project/build/bin/lldb
error: invalid value path 'symbols.swift-module-loading-mode'
(lldb) target create -c minidump.88598
warning: (arm64e) /usr/lib/system/libsystem_kernel.dylib address 0x000000018d8c5000 maps to more than one section: libsystem_kernel.dylib.__DATA_DIRTY and libdyld.dylib.__TEXT
warning: (arm64e) /usr/lib/system/libsystem_platform.dylib address 0x000000018d8f8000 maps to more than one section: libsystem_platform.dylib.__DATA_CONST and libsystem_info.dylib.__TEXT
Core file 'reproduce_folder/minidump.88598' (aarch64) was loaded.
(lldb) bt
* thread #1
  * frame #0: 0x000000018d8813c8 libsystem_kernel.dylib`__semwait_signal + 8
    frame #1: 0x000000018d760714 libsystem_c.dylib`nanosleep + 220
    frame #2: 0x000000018d7696f8 libsystem_c.dylib`sleep + 52
    frame #3: 0x0000000100373f40 simple`func4 + 16
    frame #4: 0x0000000100373f54 simple`func3 + 12
    frame #5: 0x0000000100373f68 simple`func2 + 12
    frame #6: 0x0000000100373f7c simple`func1 + 12
    frame #7: 0x0000000100373f90 simple`main + 12
    frame #8: 0x000000018d53c274 dyld`start + 2840
(lldb) ^D

// rebuild again with change commented out or go back to revision e9c8f75

$ ~/work/llvm-project/build/bin/lldb
error: invalid value path 'symbols.swift-module-loading-mode'
(lldb) target create -c minidump.88598
warning: (arm64e) /usr/lib/system/libsystem_kernel.dylib address 0x000000018d8c5000 maps to more than one section: libsystem_kernel.dylib.__DATA_DIRTY and libdyld.dylib.__TEXT
warning: (arm64e) /usr/lib/system/libsystem_platform.dylib address 0x000000018d8f8000 maps to more than one section: libsystem_platform.dylib.__DATA_CONST and libsystem_info.dylib.__TEXT
Core file 'reproduce_folder/minidump.88598' (aarch64) was loaded.
(lldb) bt
* thread #1
  * frame #0: 0x000000018d8813c8
    frame #1: 0x000000018d760714
(lldb)

[llvmbot]

@llvm/issue-subscribers-lldb

Author: Justin (strangewiz)

After e9c8f75 `[LLDB][Minidump] Have Minidumps save off and properly read TLS data (#109477)`, minidumps loaded with the binary adjacent cannot show a correct backtrace beyond the first frame.

e.g. we see:

(lldb) bt
* thread #<!-- -->1
  * frame #<!-- -->0: 0x000000018d8813c8
    frame #<!-- -->1: 0x000000018d760714

instead of:

(lldb) bt
* thread #<!-- -->1
  * frame #<!-- -->0: 0x000000018d8813c8 libsystem_kernel.dylib`__semwait_signal + 8
    frame #<!-- -->1: 0x000000018d760714 libsystem_c.dylib`nanosleep + 220
    frame #<!-- -->2: 0x000000018d7696f8 libsystem_c.dylib`sleep + 52
    frame #<!-- -->3: 0x0000000100373f40 simple`func4 + 16
    frame #<!-- -->4: 0x0000000100373f54 simple`func3 + 12
    frame #<!-- -->5: 0x0000000100373f68 simple`func2 + 12
    frame #<!-- -->6: 0x0000000100373f7c simple`func1 + 12
    frame #<!-- -->7: 0x0000000100373f90 simple`main + 12
    frame #<!-- -->8: 0x000000018d53c274 dyld`start + 2840

Steps to reproduce:

lldb built with:

 cmake -GNinja -DCMAKE_BUILD_TYPE=Debug -DLLVM_ENABLE_ASSERTIONS=ON       -DLLVM_ENABLE_PROJECTS='clang;lld;compiler-rt;lldb'       -DCMAKE_C_COMPILER=$CLANG_BASE_PATH/bin/clang       -DCMAKE_CXX_COMPILER=$CLANG_BASE_PATH/bin/clang++       -DCMAKE_ASM_COMPILER=$CLANG_BASE_PATH/bin/clang       -DCMAKE_ASM_COMPILER_ID="Clang"   -DLLVM_ENABLE_RUNTIMES="libcxx;libcxxabi"    -DCMAKE_OSX_SYSROOT=$PWD/../sysroot/MacOSX.sdk       -DDARWIN_macosx_CACHED_SYSROOT=$PWD/../sysroot/MacOSX.sdk       -DDARWIN_iphoneos_CACHED_SYSROOT=$PWD/../sysroot/iPhoneOS.sdk       -DDARWIN_iphonesimulator_CACHED_SYSROOT=$PWD/../sysroot/iPhoneSimulator.sdk       ../llvm

and:

 $ ninja lldb

at revision: e9c8f75

$ cd reproduce_folder
$ cat simple.c
#include <unistd.h>

[[clang::optnone]]
void func4() {
    sleep(1000);
}

[[clang::optnone]]
void func3() {
    func4();
}

[[clang::optnone]]
void func2() {
    func3();
}

[[clang::optnone]]
void func1() {
    func2();
}

int main() {
    func1();
    return 0;
}

$ clang -Os simple.c -o simple

$ ./simple &
[1] 88598

$ sudo crashpad_generate_dump 88598
Password:

$ ~/llvm-project/build/bin/lldb
error: invalid value path 'symbols.swift-module-loading-mode'
(lldb) target create -c minidump.88598
warning: (arm64e) /usr/lib/system/libsystem_kernel.dylib address 0x000000018d8c5000 maps to more than one section: libsystem_kernel.dylib.__DATA_DIRTY and libdyld.dylib.__TEXT
warning: (arm64e) /usr/lib/system/libsystem_platform.dylib address 0x000000018d8f8000 maps to more than one section: libsystem_platform.dylib.__DATA_CONST and libsystem_info.dylib.__TEXT
Core file 'reproduce_folder/minidump.88598' (aarch64) was loaded.
(lldb) bt
* thread #<!-- -->1
  * frame #<!-- -->0: 0x000000018d8813c8
    frame #<!-- -->1: 0x000000018d760714
(lldb) ^D

// rebuild with DynamicLoader *GetDynamicLoader() override { return nullptr; }, or just reset to previous revision, 2ff4c25

diff --git a/lldb/source/Plugins/Process/minidump/ProcessMinidump.h b/lldb/source/Plugins/Process/minidump/ProcessMinidump.h
index 3d235670a3..83ff1512cc 100644
--- a/lldb/source/Plugins/Process/minidump/ProcessMinidump.h
+++ b/lldb/source/Plugins/Process/minidump/ProcessMinidump.h
@@ -53,6 +53,7 @@ public:

   Status DoLoadCore() override;

+  DynamicLoader *GetDynamicLoader() override { return nullptr; }
   // Returns AUXV structure found in the core file
   lldb_private::DataExtractor GetAuxvData() override;

// and recompile lldb

$ ~/work/llvm-project/build/bin/lldb
error: invalid value path 'symbols.swift-module-loading-mode'
(lldb) target create -c minidump.88598
warning: (arm64e) /usr/lib/system/libsystem_kernel.dylib address 0x000000018d8c5000 maps to more than one section: libsystem_kernel.dylib.__DATA_DIRTY and libdyld.dylib.__TEXT
warning: (arm64e) /usr/lib/system/libsystem_platform.dylib address 0x000000018d8f8000 maps to more than one section: libsystem_platform.dylib.__DATA_CONST and libsystem_info.dylib.__TEXT
Core file 'reproduce_folder/minidump.88598' (aarch64) was loaded.
(lldb) bt
* thread #<!-- -->1
  * frame #<!-- -->0: 0x000000018d8813c8 libsystem_kernel.dylib`__semwait_signal + 8
    frame #<!-- -->1: 0x000000018d760714 libsystem_c.dylib`nanosleep + 220
    frame #<!-- -->2: 0x000000018d7696f8 libsystem_c.dylib`sleep + 52
    frame #<!-- -->3: 0x0000000100373f40 simple`func4 + 16
    frame #<!-- -->4: 0x0000000100373f54 simple`func3 + 12
    frame #<!-- -->5: 0x0000000100373f68 simple`func2 + 12
    frame #<!-- -->6: 0x0000000100373f7c simple`func1 + 12
    frame #<!-- -->7: 0x0000000100373f90 simple`main + 12
    frame #<!-- -->8: 0x000000018d53c274 dyld`start + 2840
(lldb) ^D

// rebuild again with change commented out or go back to revision e9c8f75

$ ~/work/llvm-project/build/bin/lldb
error: invalid value path 'symbols.swift-module-loading-mode'
(lldb) target create -c minidump.88598
warning: (arm64e) /usr/lib/system/libsystem_kernel.dylib address 0x000000018d8c5000 maps to more than one section: libsystem_kernel.dylib.__DATA_DIRTY and libdyld.dylib.__TEXT
warning: (arm64e) /usr/lib/system/libsystem_platform.dylib address 0x000000018d8f8000 maps to more than one section: libsystem_platform.dylib.__DATA_CONST and libsystem_info.dylib.__TEXT
Core file 'reproduce_folder/minidump.88598' (aarch64) was loaded.
(lldb) bt
* thread #<!-- -->1
  * frame #<!-- -->0: 0x000000018d8813c8
    frame #<!-- -->1: 0x000000018d760714
(lldb)

[strangewiz]

Steps for making Crashpad's generate_dump (from https://chromium.googlesource.com/crashpad/crashpad/+/HEAD/doc/developing.md)

$ mkdir crashpad
$ cd crashpad
$ fetch crashpad
$ cd crashpad
$ gclient sync
$ gn gen out/Default
$ ninja -C out/Default/ generate_dump

[strangewiz]

@Jlalond Sorry that it took a while to find this. This breaks an important use case for us. Do you have an idea when you'll be able to take a look? If it's not until next year, do you think it makes sense to revert this upstream? Or should we revert locally on a branch? Thanks!

[Jlalond]

@Jlalond Sorry that it took a while to find this. This breaks an important use case for us. Do you have an idea when you'll be able to take a look? If it's not until next year, do you think it makes sense to revert this upstream? Or should we revert locally on a branch? Thanks!

Hey @strangewiz, I haven't had a chance to look deeply into why the DYLD suddenly doesn't play nice with crashpad. My plan was to add a flag that LLDB generated the Minidump and when that flag is set to use the DYLD as a work around.

Specifically when we emit it, we can set the flags in the header to some LLDB specific flag for now.

[Jlalond]

@strangewiz Hi, could you see if the recent changes fixes this for you?

[strangewiz]

@Jlalond Yep, looks like things are working again. Thanks!