Symbolication misses entries in Application Specific Backtrace

[gpdawson]

I am running MacSymbolicator 2.3 (5)

When analyzing a crash report it symbolicates items from a thread, but misses items in main backtrace (items "Astro Gold"). My best guess as to why this happens is some problem with regex matching causing the entries in the main backtrace to be missed.

This is the log generated by the app:

Searching Spotlight for UUIDs: ["F6CBD55D-BEF2-3C06-9950-9B36D48695C4"]
Found F6CBD55D-BEF2-3C06-9950-9B36D48695C4: /Users/gpdawson/Library/Developer/Xcode/Archives/2021-10-06/Astro Gold 6-10-21, 8.43 am.xcarchive/dSYMs/Astro Gold.app.dSYM
Running command: xcrun atos -o "/Users/gpdawson/Library/Developer/Xcode/Archives/2021-10-06/Astro Gold 6-10-21, 8.43 am.xcarchive/dSYMs/Astro Gold.app.dSYM/Contents/Resources/DWARF/Astro Gold" -arch x86_64 -l 0x100a49000 0x0000000100c74d2a 0x0000000100c76b13
STDOUT: FIRCLSSDKFileLog (in Astro Gold) (FIRCLSInternalLogging.c:0)
FIRCLSMachExceptionServer (in Astro Gold) (FIRCLSMachException.c:170)
STDERR:
Replacing matches in sample report: []
Replacing matches in crash report: ["0x0000000100c74d2a 0x100a49000 + 2276650"]
Replacing matches in sample report: []
Replacing matches in crash report: ["0x0000000100c76b13 0x100a49000 + 2284307"]

Two relevant extracts from crash file are as follow.

Application Specific Information:
Crashing on exception: *** -[__NSArrayM objectAtIndex:]: index 14 beyond bounds [0 .. 12]

Application Specific Backtrace 1:
0 CoreFoundation 0x00007fff420bff3b __exceptionPreprocess + 171
1 libobjc.A.dylib 0x00007fff69418942 objc_exception_throw + 48
2 CoreFoundation 0x00007fff420fffcc _CFThrowFormattedException + 194
3 CoreFoundation 0x00007fff41fe1faf -[__NSArrayM objectAtIndex:] + 175
4 Astro Gold 0x0000000100b74ad0 Astro Gold + 1227472
5 Astro Gold 0x0000000100a942cb Astro Gold + 307915
6 AppKit 0x00007fff3fe75f29 _NSViewDrawRect + 83
7 AppKit 0x00007fff3fe831c4 -[NSView(NSInternal)

Thread 2:: com.google.firebase.crashlytics.MachExceptionServer
0 libsystem_kernel.dylib 0x00007fff6a1856fa write + 10
1 com.ajnaware.Astro-Gold 0x0000000100c74d2a 0x100a49000 + 2276650
2 com.ajnaware.Astro-Gold 0x0000000100c76b13 0x100a49000 + 2284307
3 libsystem_pthread.dylib 0x00007fff6a34b661 _pthread_body + 340
4 libsystem_pthread.dylib 0x00007fff6a34b50d _pthread_start + 377
5 libsystem_pthread.dylib 0x00007fff6a34abf9 thread_start + 13

[inket]

Thanks for the detailed report!

It seems when scanning the backtrace, taking the frames that have an address is not enough. We need to also match names from the Binary Images section.

I'm very curious about why the crash log looks like that though. Would you mind sharing the Binary Images section?

This part specifically:
https://github.com/inket/MacSymbolicator/blob/master/MacSymbolicatorTests/Resources/Crashes/single-target-crash.crash#L54-L56

I'm trying to see if there are two entries (Astro Gold and com.ajnaware.Astro-Gold) or if it's just one.

(Sidenote: If it's just one, you can replace Astro Gold + in the crash report with 0x100a49000 + to get that specific crash report file symbolicated with MacSymbolicator.)

This would be really helpful for the fix. Thanks! 🙏

[gpdawson]

Below is the beginning of the crash report up until the first half dozen binary images. So the only relevant binary image I see listed is "com.ajnaware.Astro-Gold", which would explain why the symbolication doesn't work on the "Astro Gold" entries. But I am certainly puzzled as to why there are two separate items that need to be somehow recognised as pointing to the same binary. Makes me wonder if there is something strange about my project configuration.

===============================

Process: Astro Gold [709]
Path: /Applications/Astro Gold.app/Contents/MacOS/Astro Gold
Identifier: com.ajnaware.Astro-Gold
Version: 5.4.1 (241)
Code Type: X86-64 (Native)
Parent Process: ??? [1]
Responsible: Astro Gold [709]
User ID: 501

Date/Time: 2021-10-07 13:49:40.414 -0300
OS Version: Mac OS X 10.13.6 (17G14042)
Report Version: 12
Anonymous UUID: 9616F083-F981-19C7-4A8C-7F64558B3A78

Time Awake Since Boot: 630 seconds

System Integrity Protection: enabled

Crashed Thread: 0 Dispatch queue: com.apple.main-thread

Exception Type: EXC_BAD_INSTRUCTION (SIGILL)
Exception Codes: 0x0000000000000001, 0x0000000000000000
Exception Note: EXC_CORPSE_NOTIFY

Termination Signal: Illegal instruction: 4
Termination Reason: Namespace SIGNAL, Code 0x4
Terminating Process: exc handler [0]

Application Specific Information:
Crashing on exception: *** -[__NSArrayM objectAtIndex:]: index 14 beyond bounds [0 .. 12]

Application Specific Backtrace 1:
0 CoreFoundation 0x00007fff420bff3b __exceptionPreprocess + 171
1 libobjc.A.dylib 0x00007fff69418942 objc_exception_throw + 48
2 CoreFoundation 0x00007fff420fffcc _CFThrowFormattedException + 194
3 CoreFoundation 0x00007fff41fe1faf -[__NSArrayM objectAtIndex:] + 175
4 Astro Gold 0x0000000100b74ad0 Astro Gold + 1227472
5 Astro Gold 0x0000000100a942cb Astro Gold + 307915
6 AppKit 0x00007fff3fe75f29 _NSViewDrawRect + 83
7 AppKit 0x00007fff3fe831c4 -[NSView(NSInternal) _recursive:displayRectIgnoringOpacity:inGraphicsContext:shouldChangeFontReferenceColor:] + 1578
8 AppKit 0x00007fff3fe82b94 -[NSView(NSInternal) _recursive:displayRectIgnoringOpacity:inContext:shouldChangeFontReferenceColor:] + 56
9 AppKit 0x00007fff3f714074 __46-[NSView(NSLayerKitGlue) drawLayer:inContext:]_block_invoke + 289
10 AppKit 0x00007fff3f713c08 -[NSView(NSLayerKitGlue) drawViewBackingLayer:inContext:drawingHandler:] + 1577
11 AppKit 0x00007fff3f7135d9 -[NSView(NSLayerKitGlue) drawLayer:inContext:] + 77
12 QuartzCore 0x00007fff4d3f8556 CABackingStoreUpdate + 300
13 QuartzCore 0x00007fff4d3f841b ___ZN2CA5Layer8display_Ev_block_invoke + 53
14 QuartzCore 0x00007fff4d3f7cb2 -[CALayer _display] + 2238
15 AppKit 0x00007fff3f71259e _NSBackingLayerDisplay + 556
16 AppKit 0x00007fff3f707409 -[_NSViewBackingLayer display] + 242
17 QuartzCore 0x00007fff4d3e8e29 _ZN2CA5Layer17display_if_neededEPNS_11TransactionE + 633
18 QuartzCore 0x00007fff4d3e88e7 _ZN2CA5Layer28layout_and_display_if_neededEPNS_11TransactionE + 35
19 QuartzCore 0x00007fff4d3e78d8 _ZN2CA7Context18commit_transactionEPNS_11TransactionE + 326
20 QuartzCore 0x00007fff4d3e7475 _ZN2CA11Transaction6commitEv + 487
21 AppKit 0x00007fff3febea72 __65+[CATransaction(NSCATransaction) NS_setFlushesWithDisplayRefresh]_block_invoke + 283
22 CoreFoundation 0x00007fff42058507 CFRUNLOOP_IS_CALLING_OUT_TO_AN_OBSERVER_CALLBACK_FUNCTION + 23
23 CoreFoundation 0x00007fff4205842f __CFRunLoopDoObservers + 511
24 CoreFoundation 0x00007fff4203aef8 __CFRunLoopRun + 1240
25 CoreFoundation 0x00007fff4203a787 CFRunLoopRunSpecific + 487
26 HIToolbox 0x00007fff4131ad96 RunCurrentEventLoopInMode + 286
27 HIToolbox 0x00007fff4131aa0f ReceiveNextEventCommon + 366
28 HIToolbox 0x00007fff4131a884 _BlockUntilNextEventMatchingListInModeWithFilter + 64
29 AppKit 0x00007fff3f5c7a3b _DPSNextEvent + 2085
30 AppKit 0x00007fff3fd5de34 -[NSApplication(NSEvent) _nextEventMatchingEventMask:untilDate:inMode:dequeue:] + 3044
31 AppKit 0x00007fff3f5bc84d -[NSApplication run] + 764
32 AppKit 0x00007fff3f58ba3a NSApplicationMain + 804
33 libdyld.dylib 0x00007fff6a033015 start + 1
34 ??? 0x0000000000000001 0x0 + 1

Thread 0 Crashed:: Dispatch queue: com.apple.main-thread
0 com.apple.AppKit 0x00007fff3f8f2454 -[NSApplication _crashOnException:] + 109
1 com.apple.AppKit 0x00007fff3febecb1 __65+[CATransaction(NSCATransaction) NS_setFlushesWithDisplayRefresh]_block_invoke + 858
2 com.apple.CoreFoundation 0x00007fff42058507 CFRUNLOOP_IS_CALLING_OUT_TO_AN_OBSERVER_CALLBACK_FUNCTION + 23
3 com.apple.CoreFoundation 0x00007fff4205842f __CFRunLoopDoObservers + 511
4 com.apple.CoreFoundation 0x00007fff4203aef8 __CFRunLoopRun + 1240
5 com.apple.CoreFoundation 0x00007fff4203a787 CFRunLoopRunSpecific + 487
6 com.apple.HIToolbox 0x00007fff4131ad96 RunCurrentEventLoopInMode + 286
7 com.apple.HIToolbox 0x00007fff4131aa0f ReceiveNextEventCommon + 366
8 com.apple.HIToolbox 0x00007fff4131a884 _BlockUntilNextEventMatchingListInModeWithFilter + 64
9 com.apple.AppKit 0x00007fff3f5c7a3b _DPSNextEvent + 2085
10 com.apple.AppKit 0x00007fff3fd5de34 -[NSApplication(NSEvent) _nextEventMatchingEventMask:untilDate:inMode:dequeue:] + 3044
11 com.apple.AppKit 0x00007fff3f5bc84d -[NSApplication run] + 764
12 com.apple.AppKit 0x00007fff3f58ba3a NSApplicationMain + 804
13 libdyld.dylib 0x00007fff6a033015 start + 1

Thread 1:: com.apple.NSURLConnectionLoader
0 libsystem_kernel.dylib 0x00007fff6a17a1fa mach_msg_trap + 10
1 libsystem_kernel.dylib 0x00007fff6a179714 mach_msg + 60
2 com.apple.CoreFoundation 0x00007fff4203bdc5 __CFRunLoopServiceMachPort + 341
3 com.apple.CoreFoundation 0x00007fff4203b117 __CFRunLoopRun + 1783
4 com.apple.CoreFoundation 0x00007fff4203a787 CFRunLoopRunSpecific + 487
5 com.apple.CFNetwork 0x00007fff4117a158 -[__CoreSchedulingSetRunnable runForever] + 722
6 com.apple.Foundation 0x00007fff441649d8 NSThread__start + 1197
7 libsystem_pthread.dylib 0x00007fff6a34b661 _pthread_body + 340
8 libsystem_pthread.dylib 0x00007fff6a34b50d _pthread_start + 377
9 libsystem_pthread.dylib 0x00007fff6a34abf9 thread_start + 13

Thread 2:: com.google.firebase.crashlytics.MachExceptionServer
0 libsystem_kernel.dylib 0x00007fff6a1856fa write + 10
1 com.ajnaware.Astro-Gold 0x0000000100c74d2a 0x100a49000 + 2276650
2 com.ajnaware.Astro-Gold 0x0000000100c76b13 0x100a49000 + 2284307
3 libsystem_pthread.dylib 0x00007fff6a34b661 _pthread_body + 340
4 libsystem_pthread.dylib 0x00007fff6a34b50d _pthread_start + 377
5 libsystem_pthread.dylib 0x00007fff6a34abf9 thread_start + 13

Thread 3:: com.apple.NSEventThread
0 libsystem_kernel.dylib 0x00007fff6a17a1fa mach_msg_trap + 10
1 libsystem_kernel.dylib 0x00007fff6a179714 mach_msg + 60
2 com.apple.CoreFoundation 0x00007fff4203bdc5 __CFRunLoopServiceMachPort + 341
3 com.apple.CoreFoundation 0x00007fff4203b117 __CFRunLoopRun + 1783
4 com.apple.CoreFoundation 0x00007fff4203a787 CFRunLoopRunSpecific + 487
5 com.apple.AppKit 0x00007fff3f704fc4 _NSEventThread + 184
6 libsystem_pthread.dylib 0x00007fff6a34b661 _pthread_body + 340
7 libsystem_pthread.dylib 0x00007fff6a34b50d _pthread_start + 377
8 libsystem_pthread.dylib 0x00007fff6a34abf9 thread_start + 13

Thread 4:
0 libsystem_kernel.dylib 0x00007fff6a18428a __workq_kernreturn + 10
1 libsystem_pthread.dylib 0x00007fff6a34b20e _pthread_wqthread + 1552
2 libsystem_pthread.dylib 0x00007fff6a34abe9 start_wqthread + 13

Thread 5:
0 libsystem_pthread.dylib 0x00007fff6a34abdc start_wqthread + 0

Thread 6:
0 libsystem_kernel.dylib 0x00007fff6a18428a __workq_kernreturn + 10
1 libsystem_pthread.dylib 0x00007fff6a34b20e _pthread_wqthread + 1552
2 libsystem_pthread.dylib 0x00007fff6a34abe9 start_wqthread + 13

Thread 0 crashed with X86 Thread State (64-bit):
rax: 0x00007fb887808200 rbx: 0x00006080000520c0 rcx: 0x0000000000000000 rdx: 0x0000000000000e01
rdi: 0x0000000008000101 rsi: 0x00007fb887921dd1 rbp: 0x00007ffeef1b5370 rsp: 0x00007ffeef1b5360
r8: 0x0000000000000000 r9: 0x0000000000000000 r10: 0x00007ffeef1b4e68 r11: 0xffffffffffee71ff
r12: 0x00006040001285c0 r13: 0x00007fb888805048 r14: 0x00007fff6940ae80 r15: 0x00007fff420587b0
rip: 0x00007fff3f8f2454 rfl: 0x0000000000010206 cr2: 0x0000000100d38790

Logical CPU: 2
Error Code: 0x00000000
Trap Number: 6

Binary Images:
0x100a49000 - 0x100d38ff7 +com.ajnaware.Astro-Gold (5.4.1 - 241) /Applications/Astro Gold.app/Contents/MacOS/Astro Gold
0x100e5f000 - 0x100eaafff +org.sparkle-project.Sparkle (1.26.0 - 1.26.0) <7EA2139C-DE4A-35CE-8DDB-C37FBDB6BF52> /Applications/Astro Gold.app/Contents/Frameworks/Sparkle.framework/Versions/A/Sparkle
0x104900000 - 0x1049b8ffb com.apple.AMDRadeonX3000GLDriver (1.68.25 - 1.6.8) <459B13E8-61E3-3B24-8D1F-E1E08D1F020D> /System/Library/Extensions/AMDRadeonX3000GLDriver.bundle/Contents/MacOS/AMDRadeonX3000GLDriver
0x108d49000 - 0x108d93adf dyld (551.5) /usr/lib/dyld
0x7fff3a643000 - 0x7fff3a86bff3 ATIRadeonX3000SCLib.dylib (1.68.25) <497DC8AE-0279-37F9-A190-F63DEBCD369D> /System/Library/Extensions/AMDRadeonX3000GLDriver.bundle/Contents/MacOS/ATIRadeonX3000SCLib.dylib
0x7fff3def8000 - 0x7fff3e0d8ff3 com.apple.avfoundation (2.0 - 1536.36) /System/Library/Frameworks/AVFoundation.framework/Versions/A/AVFoundation

[inket]

Thanks for sharing!

So it's not multiple binary images like I initially guessed. The complete crash report made me notice that it's in the "Application Specific Backtrace" section, and not "Thread N".

I think it just displays differently when it's under "Application Specific Backtrace".

The fix will have to match frames using the process name. I will close this ticket once the update is released 👍

[inket]

Thanks for reporting this and providing all the helpful information, and sorry it took so long to get back to it.

This should be fixed with MacSymbolicator 2.6. Please give it a try and create a new issue if you see any problems.