x/mobile: stack traces are wrong on iOS and Android

[steeve]

What version of Go are you using (go version)?

go version devel +4f178d157d Fri Oct 20 13:59:37 2017 +0000 darwin/amd64

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

GOARCH="amd64"
GOBIN=""
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOOS="darwin"
GOPATH="/Users/steeve/go"
GORACE=""
GOROOT="/usr/local/Cellar/go/1.9.1/libexec"
GOTOOLDIR="/usr/local/Cellar/go/1.9.1/libexec/pkg/tool/darwin_amd64"
GCCGO="gccgo"
CC="clang"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/bs/51dlb_nn5k35xq9qfsxv9wc00000gr/T/go-build785508759=/tmp/go-build -gno-record-gcc-switches -fno-common"
CXX="clang++"
CGO_ENABLED="1"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"

What did you do?

Bind an iOS framework using gomobile bind and create a panic.

What did you expect to see?

The real stack trace.

What did you see instead?

As commented in #20392 (comment):

# Crashlytics - plaintext stacktrace downloaded by Steeve Morin at Tue, 14 Nov 2017 00:11:12 GMT
# Platform: ios
# Application: testapp-swift
# Version: 1.0 (1)
# Issue #: 2
# Issue ID: 591c99d0be077a4dccdaf367
# Session ID: e591a6fc5b8641d8b4571ce7df6c11f4_4c0f40a1c8d011e7a1bd56847afe9799_0_v2
# Date: 2017-11-14T00:10:00Z
# OS Version: 11.1.1 (15B150)
# Device: iPhone X
# RAM Free: 7.2%
# Disk Free: 69.1%

#0. Crashed: com.apple.main-thread
0  testapp-swift                  0x1048d5798 runtime.raiseproc + 24
1  testapp-swift                  0x1048c1fdc runtime.dieFromSignal + 60
2  testapp-swift                  0x1048c1fdc runtime.dieFromSignal + 60
3  testapp-swift                  0x1048c1fdc runtime.dieFromSignal + 60
4  testapp-swift                  0x1048c1fdc runtime.dieFromSignal + 60
5  testapp-swift                  0x1048c1fdc runtime.dieFromSignal + 60
6  testapp-swift                  0x1048c1fdc runtime.dieFromSignal + 60
7  testapp-swift                  0x1048c1fdc runtime.dieFromSignal + 60
8  testapp-swift                  0x1048c1fdc runtime.dieFromSignal + 60
9  testapp-swift                  0x1048c1fdc runtime.dieFromSignal + 60
10 testapp-swift                  0x1048c1fdc runtime.dieFromSignal + 60

--

#0. Crashed: com.apple.main-thread
0  testapp-swift                  0x1048d5798 runtime.raiseproc + 24
1  testapp-swift                  0x1048c1fdc runtime.dieFromSignal + 60
2  testapp-swift                  0x1048c1fdc runtime.dieFromSignal + 60
3  testapp-swift                  0x1048c1fdc runtime.dieFromSignal + 60
4  testapp-swift                  0x1048c1fdc runtime.dieFromSignal + 60
5  testapp-swift                  0x1048c1fdc runtime.dieFromSignal + 60
6  testapp-swift                  0x1048c1fdc runtime.dieFromSignal + 60
7  testapp-swift                  0x1048c1fdc runtime.dieFromSignal + 60
8  testapp-swift                  0x1048c1fdc runtime.dieFromSignal + 60
9  testapp-swift                  0x1048c1fdc runtime.dieFromSignal + 60
10 testapp-swift                  0x1048c1fdc runtime.dieFromSignal + 60

#1. Thread
0  testapp-swift                  0x1048d5b1c runtime.mach_semaphore_wait + 12
1  testapp-swift                  0x1048ac55c runtime.semasleep1 + 60
2  testapp-swift                  0x1048ac55c runtime.semasleep1 + 60
3  testapp-swift                  0x1048ac55c runtime.semasleep1 + 60
4  testapp-swift                  0x1048ac55c runtime.semasleep1 + 60
5  testapp-swift                  0x1048ac55c runtime.semasleep1 + 60
6  testapp-swift                  0x1048ac55c runtime.semasleep1 + 60
7  testapp-swift                  0x1048ac55c runtime.semasleep1 + 60
8  testapp-swift                  0x1048ac55c runtime.semasleep1 + 60
9  testapp-swift                  0x1048ac55c runtime.semasleep1 + 60
10 testapp-swift                  0x1048ac55c runtime.semasleep1 + 60

#2. Thread
0  libsystem_kernel.dylib         0x183c65dbc __workq_kernreturn + 8
1  libsystem_pthread.dylib        0x183d76fa0 _pthread_wqthread + 884
2  libsystem_pthread.dylib        0x183d76c20 start_wqthread + 4

#3. Thread
0  libsystem_pthread.dylib        0x183d76c1c start_wqthread + 122

#4. Thread
0  libsystem_kernel.dylib         0x183c65dbc __workq_kernreturn + 8
1  libsystem_pthread.dylib        0x183d77134 _pthread_wqthread + 1288
2  libsystem_pthread.dylib        0x183d76c20 start_wqthread + 4

#5. Thread
0  libsystem_kernel.dylib         0x183c65dbc __workq_kernreturn + 8
1  libsystem_pthread.dylib        0x183d77134 _pthread_wqthread + 1288
2  libsystem_pthread.dylib        0x183d76c20 start_wqthread + 4

#6. com.apple.uikit.eventfetch-thread
0  libsystem_kernel.dylib         0x183c44bc4 mach_msg_trap + 8
1  libsystem_kernel.dylib         0x183c44a3c mach_msg + 72
2  CoreFoundation                 0x1840f5c74 __CFRunLoopServiceMachPort + 196
3  CoreFoundation                 0x1840f3840 __CFRunLoopRun + 1424
4  CoreFoundation                 0x184013fb8 CFRunLoopRunSpecific + 436
5  Foundation                     0x184a3d6e4 -[NSRunLoop(NSRunLoop) runMode:beforeDate:] + 304
6  Foundation                     0x184a5cafc -[NSRunLoop(NSRunLoop) runUntilDate:] + 96
7  UIKit                          0x18e1472f4 -[UIEventFetcher threadMain] + 136
8  Foundation                     0x184b3e860 __NSThread__start__ + 996
9  libsystem_pthread.dylib        0x183d7831c _pthread_body + 308
10 libsystem_pthread.dylib        0x183d781e8 _pthread_body + 310
11 libsystem_pthread.dylib        0x183d76c28 thread_start + 4

#7. Thread
0  libsystem_kernel.dylib         0x183c65dbc __workq_kernreturn + 8
1  libsystem_pthread.dylib        0x183d76fa0 _pthread_wqthread + 884
2  libsystem_pthread.dylib        0x183d76c20 start_wqthread + 4

#8. Thread
0  testapp-swift                  0x1048d5b44 runtime.mach_semaphore_timedwait + 20
1  testapp-swift                  0x1048ac5e0 runtime.semasleep1 + 192
2  testapp-swift                  0x1048ac5e0 runtime.semasleep1 + 192
3  testapp-swift                  0x1048ac5e0 runtime.semasleep1 + 192
4  testapp-swift                  0x1048ac5e0 runtime.semasleep1 + 192
5  testapp-swift                  0x1048ac5e0 runtime.semasleep1 + 192
6  testapp-swift                  0x1048ac5e0 runtime.semasleep1 + 192
7  testapp-swift                  0x1048ac5e0 runtime.semasleep1 + 192
8  testapp-swift                  0x1048ac5e0 runtime.semasleep1 + 192
9  testapp-swift                  0x1048ac5e0 runtime.semasleep1 + 192
10 testapp-swift                  0x1048ac5e0 runtime.semasleep1 + 192

#9. Thread
0  testapp-swift                  0x1048d5b1c runtime.mach_semaphore_wait + 12
1  testapp-swift                  0x1048ac55c runtime.semasleep1 + 60
2  testapp-swift                  0x1048ac55c runtime.semasleep1 + 60
3  testapp-swift                  0x1048ac55c runtime.semasleep1 + 60
4  testapp-swift                  0x1048ac55c runtime.semasleep1 + 60
5  testapp-swift                  0x1048ac55c runtime.semasleep1 + 60
6  testapp-swift                  0x1048ac55c runtime.semasleep1 + 60
7  testapp-swift                  0x1048ac55c runtime.semasleep1 + 60
8  testapp-swift                  0x1048ac55c runtime.semasleep1 + 60
9  testapp-swift                  0x1048ac55c runtime.semasleep1 + 60
10 testapp-swift                  0x1048ac55c runtime.semasleep1 + 60

#10. Thread
0  testapp-swift                  0x1048d5b1c runtime.mach_semaphore_wait + 12
1  testapp-swift                  0x1048ac55c runtime.semasleep1 + 60
2  testapp-swift                  0x1048ac55c runtime.semasleep1 + 60
3  testapp-swift                  0x1048ac55c runtime.semasleep1 + 60
4  testapp-swift                  0x1048ac55c runtime.semasleep1 + 60
5  testapp-swift                  0x1048ac55c runtime.semasleep1 + 60
6  testapp-swift                  0x1048ac55c runtime.semasleep1 + 60
7  testapp-swift                  0x1048ac55c runtime.semasleep1 + 60
8  testapp-swift                  0x1048ac55c runtime.semasleep1 + 60
9  testapp-swift                  0x1048ac55c runtime.semasleep1 + 60
10 testapp-swift                  0x1048ac55c runtime.semasleep1 + 60

#11. Thread
0  testapp-swift                  0x1048d5b1c runtime.mach_semaphore_wait + 12
1  testapp-swift                  0x1048ac55c runtime.semasleep1 + 60
2  testapp-swift                  0x1048ac55c runtime.semasleep1 + 60
3  testapp-swift                  0x1048ac55c runtime.semasleep1 + 60
4  testapp-swift                  0x1048ac55c runtime.semasleep1 + 60
5  testapp-swift                  0x1048ac55c runtime.semasleep1 + 60
6  testapp-swift                  0x1048ac55c runtime.semasleep1 + 60
7  testapp-swift                  0x1048ac55c runtime.semasleep1 + 60
8  testapp-swift                  0x1048ac55c runtime.semasleep1 + 60
9  testapp-swift                  0x1048ac55c runtime.semasleep1 + 60
10 testapp-swift                  0x1048ac55c runtime.semasleep1 + 60

#12. Thread
0  testapp-swift                  0x1048d5b1c runtime.mach_semaphore_wait + 12
1  testapp-swift                  0x1048ac55c runtime.semasleep1 + 60
2  testapp-swift                  0x1048ac55c runtime.semasleep1 + 60
3  testapp-swift                  0x1048ac55c runtime.semasleep1 + 60
4  testapp-swift                  0x1048ac55c runtime.semasleep1 + 60
5  testapp-swift                  0x1048ac55c runtime.semasleep1 + 60
6  testapp-swift                  0x1048ac55c runtime.semasleep1 + 60
7  testapp-swift                  0x1048ac55c runtime.semasleep1 + 60
8  testapp-swift                  0x1048ac55c runtime.semasleep1 + 60
9  testapp-swift                  0x1048ac55c runtime.semasleep1 + 60
10 testapp-swift                  0x1048ac55c runtime.semasleep1 + 60

#13. Thread
0  testapp-swift                  0x1048d5b1c runtime.mach_semaphore_wait + 12
1  testapp-swift                  0x1048ac55c runtime.semasleep1 + 60
2  testapp-swift                  0x1048ac55c runtime.semasleep1 + 60
3  testapp-swift                  0x1048ac55c runtime.semasleep1 + 60
4  testapp-swift                  0x1048ac55c runtime.semasleep1 + 60
5  testapp-swift                  0x1048ac55c runtime.semasleep1 + 60
6  testapp-swift                  0x1048ac55c runtime.semasleep1 + 60
7  testapp-swift                  0x1048ac55c runtime.semasleep1 + 60
8  testapp-swift                  0x1048ac55c runtime.semasleep1 + 60
9  testapp-swift                  0x1048ac55c runtime.semasleep1 + 60
10 testapp-swift                  0x1048ac55c runtime.semasleep1 + 60

#14. Thread
0  testapp-swift                  0x1048d5b1c runtime.mach_semaphore_wait + 12
1  testapp-swift                  0x1048ac55c runtime.semasleep1 + 60
2  testapp-swift                  0x1048ac55c runtime.semasleep1 + 60
3  testapp-swift                  0x1048ac55c runtime.semasleep1 + 60
4  testapp-swift                  0x1048ac55c runtime.semasleep1 + 60
5  testapp-swift                  0x1048ac55c runtime.semasleep1 + 60
6  testapp-swift                  0x1048ac55c runtime.semasleep1 + 60
7  testapp-swift                  0x1048ac55c runtime.semasleep1 + 60
8  testapp-swift                  0x1048ac55c runtime.semasleep1 + 60
9  testapp-swift                  0x1048ac55c runtime.semasleep1 + 60
10 testapp-swift                  0x1048ac55c runtime.semasleep1 + 60

#15. com.twitter.crashlytics.ios.MachExceptionServer
0  libsystem_kernel.dylib         0x183c44bc4 mach_msg_trap + 8
1  libsystem_kernel.dylib         0x183c44a3c mach_msg + 72
2  testapp-swift                  0x104918aa4 CLSMachExceptionServer + 100
3  libsystem_pthread.dylib        0x183d7831c _pthread_body + 308
4  libsystem_pthread.dylib        0x183d781e8 _pthread_body + 310
5  libsystem_pthread.dylib        0x183d76c28 thread_start + 4

#16. Thread
0  libsystem_kernel.dylib         0x183c65dbc __workq_kernreturn + 8
1  libsystem_pthread.dylib        0x183d77134 _pthread_wqthread + 1288
2  libsystem_pthread.dylib        0x183d76c20 start_wqthread + 4

#17. com.apple.NSURLConnectionLoader
0  libsystem_kernel.dylib         0x183c44bc4 mach_msg_trap + 8
1  libsystem_kernel.dylib         0x183c44a3c mach_msg + 72
2  CoreFoundation                 0x1840f5c74 __CFRunLoopServiceMachPort + 196
3  CoreFoundation                 0x1840f3840 __CFRunLoopRun + 1424
4  CoreFoundation                 0x184013fb8 CFRunLoopRunSpecific + 436
5  CFNetwork                      0x18477e264 +[NSURLConnection(Loader) _resourceLoadLoop:] + 404
6  Foundation                     0x184b3e860 __NSThread__start__ + 996
7  libsystem_pthread.dylib        0x183d7831c _pthread_body + 308
8  libsystem_pthread.dylib        0x183d781e8 _pthread_body + 310
9  libsystem_pthread.dylib        0x183d76c28 thread_start + 4

[steeve]

According to @eliasnaur at #20392 (comment) it could be due to missing frame pointers.

There is a CL to add them to arm64 at https://go-review.googlesource.com/c/go/+/61511, however, it doesn't seem to solve the issue. See https://groups.google.com/d/topic/golang-dev/v1TxCJNemPY/discussion

cc @williamweixiao

[steeve]

I spent some time to take a look at this, on Android this time.
FWIW it looks like systemstack is the culprit, as the stack trace starts from it in case of SIGABRT.

As expected however, frame pointers actually make the local variables visible by the debugger.

[steeve]

In the case of a CGo crash, however, the stack trace seems ok, although truncated to runtime.asmcgocall.

[steeve]

Reading systemstack, I guess it makes sense:

// systemstack runs fn on a system stack.
// If systemstack is called from the per-OS-thread (g0) stack, or
// if systemstack is called from the signal handling (gsignal) stack,
// systemstack calls fn directly and returns.
// Otherwise, systemstack is being called from the limited stack <----------
// of an ordinary goroutine. In this case, systemstack switches
// to the per-OS-thread stack, calls fn, and switches back.
// It is common to use a func literal as the argument, in order
// to share inputs and outputs with the code around the call
// to system stack:
//
//	... set up y ...
//	systemstack(func() {
//		x = bigcall(y)
//	})
//	... use x ...
//

[eliasnaur]

I'm not a Go runtime developer, but one way to move this issue forward is to make it more concrete:

1. What is confusing Crashlytics and similar Go-oblivious crash handlers and debuggers that are not confusing gdb (and lldb and delve)? I see a lot of work going into debug information on the Go side, so I assume users of gdb, lldb, delve are happy with their crash traces.
2. In light of 1, can a self-contained case be constructed that demonstrates the problem without having to use Xcode, Android studio or similar large and opaque dependencies? If possible, a test case directly runnable from the standard library misc/ios/go_darwin_arm_exec.go or misc/android/go_android_exec.go exec wrappers.
3. If possible, demonstrate the failure on a android/amd64 or darwin/amd64 iOS emulator, to rule out problem with missing framepointers. Or, to demonstrate that missing framepointers is the problem.
4. If possible, demonstrate the failure on a desktop platform (e.g. linux/amd64 or regular darwin/amd64). This will rule out any mobile specific issues. Or, demonstrate that the issue is mobile specific.

[williamweixiao]

Yes, a self-contained case is important for post mortem of the wrong stack traces.
As I know, frame-pointer is enabled on amd64 desktop platform (e.g. linux/amd64 and darwin/amd64) by default. If we can reproduce the failure on the platform we can rule out the problem with missing framepointers.

[eliasnaur]

It occurred to me that there is a small chance that the crash logger is somehow confused by the DWARF information the Go compiler generates. Have you tried to crash log a binary compiled with -ldflags="-w" on a GOARCH with framepointers (amd64)?

[williamweixiao]

hi, can you try the latest version of my patch (https://go-review.googlesource.com/c/go/+/61511) on darwin/arm64?

[steeve]

So I've spent more time and indeed, the DWARF on darwin/arm64, while there, is empty:

$ gomobile bind -v -target=ios -prefix "" github.com/steeve/adder
$ dwarfdump -a --arch=arm64 Adder.framework/Adder
----------------------------------------------------------------------
 File: Adder.framework/Adder(go.o) (arm64)
----------------------------------------------------------------------
.debug_abbrev contents:
< EMPTY >

.debug_aranges contents:
< EMPTY >

.debug_info contents:
< EMPTY >
.debug_inlined contents:
< EMPTY >
.debug_frame contents:
< EMPTY >

.debug_line contents:
< EMPTY >

.debug_macinfo contents:
< EMPTY >

.debug_pubnames contents:
< EMPTY >

.debug_pubtypes contents:
< EMPTY >

.apple_names contents:

.apple_types contents:

.apple_exttypes contents:

.apple_namespaces contents:

.apple_objc contents:


.debug_str contents:
< EMPTY >

Indeed this may be confusing the unwinder, however, as tested by @eliasnaur, removing it doesn't seem to make the situation better either with, or without frame pointers, at least on Darwin.