x/mobile: bind with go1.22.5,1.22.6 leads to: fatal error: runtime: stack split at bad time #68760

xcolwell · 2024-08-07T10:41:56Z

Go version

go version go1.22.5 darwin/arm64

Output of `go env` in your module/workspace:

GO111MODULE=''
GOARCH='arm64'
GOBIN=''
GOCACHE='/Users/brien/Library/Caches/go-build'
GOENV='/Users/brien/Library/Application Support/go/env'
GOEXE=''
GOEXPERIMENT=''
GOFLAGS=''
GOHOSTARCH='arm64'
GOHOSTOS='darwin'
GOINSECURE=''
GOMODCACHE='/Users/brien/go/pkg/mod'
GONOPROXY=''
GONOSUMDB=''
GOOS='darwin'
GOPATH='/Users/brien/go'
GOPRIVATE=''
GOPROXY='https://proxy.golang.org,direct'
GOROOT='/usr/local/go'
GOSUMDB='sum.golang.org'
GOTMPDIR=''
GOTOOLCHAIN='auto'
GOTOOLDIR='/usr/local/go/pkg/tool/darwin_arm64'
GOVCS=''
GOVERSION='go1.22.5'
GCCGO='gccgo'
AR='ar'
CC='clang'
CXX='clang++'
CGO_ENABLED='1'
GOMOD='/dev/null'
GOWORK=''
CGO_CFLAGS='-O2 -g'
CGO_CPPFLAGS=''
CGO_CXXFLAGS='-O2 -g'
CGO_FFLAGS='-O2 -g'
CGO_LDFLAGS='-O2 -g'
PKG_CONFIG='pkg-config'
GOGCCFLAGS='-fPIC -arch arm64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -ffile-prefix-map=/var/folders/r3/v_3z60rx2cxg0s1r9tl9fbmw0000gn/T/go-build1492278578=/tmp/go-build -gno-record-gcc-switches -fno-common'

What did you do?

Build a native aar for Android as:

	gomobile bind \
		-ldflags "-X client.Version=$$WARP_VERSION" \
		-target=android -androidapi 19 \
		-javapkg com.bringyour \
		-trimpath \
		-o build/android/BringYourClient.aar \

The code has interface callbacks that call from Go -> Kotlin -> Go.

What did you see happen?

Works up to go 1.22.4. On 1.22.5, the interface callback raises the error below. This happens in a Kotlin interface implementation called from Go, that is making another call to Go.

2024-08-07 02:52:16.207 26398-0     Go                      com.bringyour.network                E  runtime: newstack at runtime.printlock+0x78 sp=0x400005eb00 stack=[0x400005e000, 0x400005f000]
2024-08-07 02:52:16.207 26398-0     Go                      com.bringyour.network                E  	morebuf={pc:0x6f97d94980 sp:0x400005eb00 lr:0x0}
2024-08-07 02:52:16.207 26398-0     Go                      com.bringyour.network                E  	sched={pc:0x6f97dca218 sp:0x400005eb00 lr:0x6f97d94980 ctxt:0x0}
2024-08-07 02:52:16.207 26398-0     Go                      com.bringyour.network                E  runtime.callbackUpdateSystemStack(0x4000052008, 0x7fd7e4b9d0, 0x0)
2024-08-07 02:52:16.207 26398-0     Go                      com.bringyour.network                E  	/usr/local/go/src/runtime/cgocall.go:241 +0x90 fp=0x400005eb60 sp=0x400005eb00 pc=0x6f97d94980
2024-08-07 02:52:16.207 26398-0     Go                      com.bringyour.network                E  runtime.cgocallbackg(0x6f9815db90, 0x7fd7e4bac8, 0x0)
2024-08-07 02:52:16.207 26398-0     Go                      com.bringyour.network                E  	/usr/local/go/src/runtime/cgocall.go:306 +0x68 fp=0x400005ebe0 sp=0x400005eb60 pc=0x6f97d94b48
2024-08-07 02:52:16.207 26398-0     Go                      com.bringyour.network                E  runtime.cgocallbackg(0x6f9815db90, 0x7fd7e4bac8, 0x0)
2024-08-07 02:52:16.207 26398-0     Go                      com.bringyour.network                E  	<autogenerated>:1 +0x1c fp=0x400005ec10 sp=0x400005ebe0 pc=0x6f97e0163c
2024-08-07 02:52:16.207 26398-0     Go                      com.bringyour.network                E  runtime.cgocallback(0x400005ec88, 0x6f98136f90, 0x6f9817bd10)
2024-08-07 02:52:16.207 26398-0     Go                      com.bringyour.network                E  	/usr/local/go/src/runtime/asm_arm64.s:1130 +0xb0 fp=0x400005ec40 sp=0x400005ec10 pc=0x6f97dff980
2024-08-07 02:52:16.207 26398-0     Go                      com.bringyour.network                E  runtime.systemstack_switch()
2024-08-07 02:52:16.207 26398-0     Go                      com.bringyour.network                E  	/usr/local/go/src/runtime/asm_arm64.s:200 +0x8 fp=0x400005ec50 sp=0x400005ec40 pc=0x6f97dfd548
2024-08-07 02:52:16.207 26398-0     Go                      com.bringyour.network                E  runtime.cgocall(0x6f9817bd10, 0x400005ecc8)
2024-08-07 02:52:16.207 26398-0     Go                      com.bringyour.network                E  	/usr/local/go/src/runtime/cgocall.go:175 +0x70 fp=0x400005ec90 sp=0x400005ec50 pc=0x6f97d94890
2024-08-07 02:52:16.207 26398-0     Go                      com.bringyour.network                E  main._Cfunc_cproxyclient_ProvideChangeListener_ProvideChanged(0x2a, 0x1)
2024-08-07 02:52:16.207 26398-0     Go                      com.bringyour.network                E  	_cgo_gotypes.go:550 +0x30 fp=0x400005ecc0 sp=0x400005ec90 pc=0x6f98136f90
2024-08-07 02:52:16.207 26398-0     Go                      com.bringyour.network                E  main.(*proxyclient_ProvideChangeListener).ProvideChanged(0x40002fc204, 0x1)
2024-08-07 02:52:16.207 26398-0     Go                      com.bringyour.network                E  	/private/var/folders/r3/v_3z60rx2cxg0s1r9tl9fbmw0000gn/T/gomobile-work-233819857/src-android-arm64/gobind/go_clientmain.go:7243 +0x58 fp=0x400005ece0 sp=0x400005ecc0 pc=0x6f981560a8
2024-08-07 02:52:16.207 26398-0     Go                      com.bringyour.network                E  bringyour.com/client.(*BringYourDevice).provideChanged.func1()
2024-08-07 02:52:16.207 26398-0     Go                      com.bringyour.network                E  	/Users/brien/bringyour/bringyour/client/device.go:241 +0x2c fp=0x400005ed00 sp=0x400005ece0 pc=0x6f9812423c
2024-08-07 02:52:16.207 26398-0     Go                      com.bringyour.network                E  bringyour.com/connect.HandleError(0x40001280c0?, {0x0?, 0x400005edb8?, 0x6f981247a8?})
2024-08-07 02:52:16.207 26398-0     Go                      com.bringyour.network                E  	/Users/brien/bringyour/connect/connect/trace.go:65 +0x70 fp=0x400005ed60 sp=0x400005ed00 pc=0x6f980f0e70
2024-08-07 02:52:16.207 26398-0     Go                      com.bringyour.network                E  bringyour.com/client.(*BringYourDevice).provideChanged(0x4000314240?, 0x1)
2024-08-07 02:52:16.207 26398-0     Go                      com.bringyour.network                E  	/Users/brien/bringyour/bringyour/client/device.go:240 +0x70 fp=0x400005edc0 sp=0x400005ed60 pc=0x6f981241e0
2024-08-07 02:52:16.207 26398-0     Go                      com.bringyour.network                E  bringyour.com/client.(*BringYourDevice).SetProvideMode(0x4000314240, 0x6f97d94f88?)
2024-08-07 02:52:16.207 26398-0     Go                      com.bringyour.network                E  	/Users/brien/bringyour/bringyour/client/device.go:311 +0x34 fp=0x400005ede0 sp=0x400005edc0 pc=0x6f981247b4
2024-08-07 02:52:16.207 26398-0     Go                      com.bringyour.network                E  main.proxyclient_BringYourDevice_SetProvideMode(0x5ee01?, 0x3)
2024-08-07 02:52:16.207 26398-0     Go                      com.bringyour.network                E  	/private/var/folders/r3/v_3z60rx2cxg0s1r9tl9fbmw0000gn/T/gomobile-work-233819857/src-android-arm64/gobind/go_clientmain.go:1610 +0x40 fp=0x400005ee10 sp=0x400005ede0 pc=0x6f9813e7b0
2024-08-07 02:52:16.207 26398-0     Go                      com.bringyour.network                E  _cgoexp_5858fad8bb02_proxyclient_BringYourDevice_SetProvideMode(0x400005ee88?)
2024-08-07 02:52:16.207 26398-0     Go                      com.bringyour.network                E  	_cgo_gotypes.go:2126 +0x28 fp=0x400005ee30 sp=0x400005ee10 pc=0x6f9815df18
2024-08-07 02:52:16.207 26398-0     Go                      com.bringyour.network                E  runtime.cgocallbackg1(0x6f9815def0, 0x7fd7e50960, 0x0)
2024-08-07 02:52:16.207 26398-0     Go                      com.bringyour.network                E  	/usr/local/go/src/runtime/cgocall.go:420 +0x228 fp=0x400005ef00 sp=0x400005ee30 pc=0x6f97d94ec8
2024-08-07 02:52:16.207 26398-0     Go                      com.bringyour.network                E  runtime.cgocallbackg(0x6f9815def0, 0x7fd7e50960, 0x0)
2024-08-07 02:52:16.207 26398-0     Go                      com.bringyour.network                E  	/usr/local/go/src/runtime/cgocall.go:339 +0x10c fp=0x400005ef80 sp=0x400005ef00 pc=0x6f97d94bec
2024-08-07 02:52:16.207 26398-0     Go                      com.bringyour.network                E  runtime.cgocallbackg(0x6f9815def0, 0x7fd7e50960, 0x0)
2024-08-07 02:52:16.207 26398-0     Go                      com.bringyour.network                E  	<autogenerated>:1 +0x1c fp=0x400005efb0 sp=0x400005ef80 pc=0x6f97e0163c
2024-08-07 02:52:16.207 26398-0     Go                      com.bringyour.network                E  runtime.cgocallback(0x0, 0x0, 0x0)
2024-08-07 02:52:16.207 26398-0     Go                      com.bringyour.network                E  	/usr/local/go/src/runtime/asm_arm64.s:1130 +0xb0 fp=0x400005efe0 sp=0x400005efb0 pc=0x6f97dff980
2024-08-07 02:52:16.207 26398-0     Go                      com.bringyour.network                E  runtime.goexit({})
2024-08-07 02:52:16.207 26398-0     Go                      com.bringyour.network                E  	/usr/local/go/src/runtime/asm_arm64.s:1222 +0x4 fp=0x400005efe0 sp=0x400005efe0 pc=0x6f97dffa74
2024-08-07 02:52:16.207 26398-0     Go                      com.bringyour.network                E  fatal error: runtime: stack split at bad time

What did you expect to see?

No error.

The text was updated successfully, but these errors were encountered:

gabyhelp · 2024-08-07T10:42:51Z

Related Issues and Documentation

_{(Emoji vote if this was helpful or unhelpful; more detailed feedback welcome in this discussion.)}

prattmic · 2024-08-07T12:48:50Z

cc @cherrymui @golang/runtime

prattmic · 2024-08-07T14:44:05Z

This is presumably a bug in https://go.googlesource.com/go/+/3560cf0afb3c29300a6c88ccd98256949ca7a6f6, which is in go1.22.5.

@xcolwell could you test with this commit reverted? Something like this should do the trick:

$ git clone https://go.googlesource.com/go
$ cd go
$ git checkout release-branch.go1.22
$ git revert 3560cf0afb3c29300a6c88ccd98256949ca7a6f6
$ cd src
$ ./make.bash

Then build with the toolchain in $CHECKOUT/bin. (I guess this means have this in PATH when running gomobile? I'm not familiar with gomobile)

xcolwell · 2024-08-07T18:59:17Z

@prattmic that worked. I changed the installed go to the patched on with the below, and verified that the gomobile build worked when using the patched go, and failed when using 1.22.5.

cd /Users/brien/bringyour/temp/go-issue-68760
# your steps
sudo cp -r /Users/brien/bringyour/temp/go-issue-68760/go /usr/local/go.patched
sudo mv /usr/local/go /usr/local/go.1.22.5
sudo mv /usr/local/go.patched /usr/local/go
go version
> go version go1.22.6 darwin/arm64

As a summary of what the mobile code looks like, we have a gomobile lib that lets us register callbacks from Kotlin. There is a sequence like below. When the Kotlin callback is called from within the gomobile lib, the first function call back into the gomobile lib crashes with the error.

// byDevice is a Go object in the gomobile lib
deviceProvideSub = byDevice?.addProvideChangeListener { provideEnabled ->
    runBlocking(Dispatchers.Main.immediate) {
        updateVpnService()
    }
}

private fun updateVpnService() {
  val byDevice = byDevice ?: return
  
  // the first call back into the gomobile lib crashes with the error
  val provideEnabled = byDevice.isProvideEnabled
  // ...
}

ignoramous · 2024-08-19T21:31:56Z

As a summary of what the mobile code looks like, we have a gomobile lib that lets us register callbacks from Kotlin. There is a sequence like below. When the Kotlin callback is called from within the gomobile lib, the first function call back into the gomobile lib crashes with the error.

Can confirm. We see the same error but in go1.23: celzero/firestack#84

gopherbot added the mobile Android, iOS, and x/mobile label Aug 7, 2024

gopherbot added this to the Unreleased milestone Aug 7, 2024

dr2chase added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Aug 7, 2024

This was referenced Aug 28, 2024

x/mobile: gomobile bind: Crash on android: dlopen failed: TLS symbol "(null)" in dlopened x86_64/libgojni.so using IE access model #69109

Open

x/mobile,runtime: missing stack trace on Android crash #69101

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

x/mobile: bind with go1.22.5,1.22.6 leads to: fatal error: runtime: stack split at bad time #68760

x/mobile: bind with go1.22.5,1.22.6 leads to: fatal error: runtime: stack split at bad time #68760

xcolwell commented Aug 7, 2024

gabyhelp commented Aug 7, 2024

prattmic commented Aug 7, 2024

prattmic commented Aug 7, 2024 •

edited

Loading

xcolwell commented Aug 7, 2024 •

edited

Loading

ignoramous commented Aug 19, 2024

x/mobile: bind with go1.22.5,1.22.6 leads to: fatal error: runtime: stack split at bad time #68760

x/mobile: bind with go1.22.5,1.22.6 leads to: fatal error: runtime: stack split at bad time #68760

Comments

xcolwell commented Aug 7, 2024

Go version

Output of go env in your module/workspace:

What did you do?

What did you see happen?

What did you expect to see?

gabyhelp commented Aug 7, 2024

prattmic commented Aug 7, 2024

prattmic commented Aug 7, 2024 • edited Loading

xcolwell commented Aug 7, 2024 • edited Loading

ignoramous commented Aug 19, 2024

Output of `go env` in your module/workspace:

prattmic commented Aug 7, 2024 •

edited

Loading

xcolwell commented Aug 7, 2024 •

edited

Loading