Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

runtime: "fatal: systemstack called from unexpected goroutine" on armv6 #38539

Open
diamondo25 opened this issue Apr 20, 2020 · 6 comments
Open

runtime: "fatal: systemstack called from unexpected goroutine" on armv6 #38539

diamondo25 opened this issue Apr 20, 2020 · 6 comments

Comments

@diamondo25
Copy link
Contributor

@diamondo25 diamondo25 commented Apr 20, 2020

Note: I am using karalabe's xgo for compiling to ARM6

What version of Go are you using (go version)?

$ go version
go version go1.12.17 linux/amd64

Does this issue reproduce with the latest release?

Yes, 1.12.17 and 1.14.0 both seem affected.

What did you do?

Cross-compile application with github.com/mattn/go-sqlite3 as only import to armv6, with CGO_ENABLED=1

Code:

package main

import _ "github.com/mattn/go-sqlite3"

func main() {
	println("Hello")
}

Longer story is that my host OS is macOS, so I am using karalabe/go-1.12.12 by default, as target. However, I have also tested this on go-1.12.17 (cd docker/go-1.12.17 && docker build -t karalabe/xgo-1.12.17) and the latest, 1.14.0 (cd docker/go-1.14.0 && docker build -t karalabe/xgo-1.14.0), but no difference.

I am able to get Segfault on three occasions:

  1. use code without gomod, output of built executable will be Segmentation fault
  2. use code with gomod (only go get github.com/mattn/go-sqlite3 as package), output of executable will be Segmentation fault
  3. use code with gomod (only go get github.com/mattn/go-sqlite3@v1.11.0 as package), output of executable will be fatal: systemstack called from unexpected goroutineSegmentation fault

Building

Test environment

Raspberry Pi Zero (not W). Building through cross-compilation because of no internet.

pi@raspberrypi:~ $ cat /proc/cpuinfo
processor	: 0
model name	: ARMv6-compatible processor rev 7 (v6l)
BogoMIPS	: 997.08
Features	: half thumb fastmult vfp edsp java tls
CPU implementer	: 0x41
CPU architecture: 7
CPU variant	: 0x0
CPU part	: 0xb76
CPU revision	: 7

Hardware	: BCM2835
Revision	: 900093
Serial		: 0000000088a33a5e

pi@raspberrypi:~ $ uname -a
Linux raspberrypi 4.14.98+ #1200 Tue Feb 12 20:11:02 GMT 2019 armv6l GNU/Linux

Not using gomod

Build code:

$ mkdir diamondo25 && cd diamondo25
# write the main.go file from the code above
$ nano main.go
$ docker run -it -v $PWD:/code/ -w /code/ --entrypoint bash karalabe/xgo-1.12.17  
root@91db62d6dde7:/code# export CC=arm-linux-gnueabihf-gcc-5 GOOS=linux GOARCH=arm GOARM=6 CGO_ENABLED=1

root@91db62d6dde7:/code# CGO_CFLAGS='-march=armv6' CGO_CXXFLAGS='-march=armv6' go install std

root@91db62d6dde7:/code# go get github.com/mattn/go-sqlite3

root@91db62d6dde7:/code# go build -o hello .

root@91db62d6dde7:/code# exit

There's now a hello executable in the 'diamondo25' folder.

Copy the binary to a armv6l box. Running the executable SEGFAULTs right away:

pi@raspberrypi:~ $ ./hello
Segmentation fault

pi@raspberrypi:~ $ file hello
hello: ELF 32-bit LSB executable, ARM, EABI5 version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux-armhf.so.3, for GNU/Linux 3.2.0, BuildID[sha1]=c60aae380bfbd3f2292bfa0a5718ecaece5cb328, not stripped

GDB doesn't help much:

pi@raspberrypi:~ $ gdb hello
GNU gdb (Raspbian 7.12-6) 7.12.0.20161007-git
Copyright (C) 2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "arm-linux-gnueabihf".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from hello...done.
warning: Missing auto-load script at offset 0 in section .debug_gdb_scripts
of file /home/pi/hello.
Use `info auto-load python-scripts [REGEXP]' to list them.
(gdb) r
Starting program: /home/pi/hello
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/arm-linux-gnueabihf/libthread_db.so.1".

Program received signal SIGSEGV, Segmentation fault.
runtime.tracebackHexdump (stk=..., frame=0xbefffeb8, bad=3204447949) at /usr/local/go/src/runtime/traceback.go:992
992	/usr/local/go/src/runtime/traceback.go: No such file or directory.
(gdb) bt
#0  runtime.tracebackHexdump (stk=..., frame=0xbefffeb8, bad=3204447949) at /usr/local/go/src/runtime/traceback.go:992
#1  0x00000000 in ?? ()
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

Using gomod

$ mkdir diamondo25_gomod && cd diamondo25_gomod
# write the main.go file from the code above
$ nano main.go
$ docker run -it -v $PWD:/code/ -w /code/ --entrypoint bash karalabe/xgo-1.12.17
root@e35c28fde210:/code# go mod init hello
go: creating new go.mod: module hello

root@e35c28fde210:/code# go get github.com/mattn/go-sqlite3 
go: finding github.com/mattn/go-sqlite3 v2.0.3+incompatible
go: downloading github.com/mattn/go-sqlite3 v2.0.3+incompatible
go: extracting github.com/mattn/go-sqlite3 v2.0.3+incompatible

root@e35c28fde210:/code# export CC=arm-linux-gnueabihf-gcc-5 GOOS=linux GOARCH=arm GOARM=6 CGO_ENABLED=1

root@e35c28fde210:/code# CGO_CFLAGS='-march=armv6' CGO_CXXFLAGS='-march=armv6' go install std

root@e35c28fde210:/code# go build -o hello .
# Keep open for next variant

There's now a hello executable in the 'diamondo25_gomod' folder. The building of this executable took longer than the previous, no-gomod variant.

Running this also reports Segmentation fault

Using gomod with go-sqlite3@v1.11.0

Using the docker container shell, change the go-sqlite3 module to v1.11.0

root@e35c28fde210:/code# go get github.com/mattn/go-sqlite3@v1.11.0
go: finding github.com/mattn/go-sqlite3 v1.11.0
go: downloading github.com/mattn/go-sqlite3 v1.11.0
go: extracting github.com/mattn/go-sqlite3 v1.11.0

root@c1619b650cc7:/code# go build -o hello .

Now, copying and running this executable will throw the fatal: systemstack called from unexpected goroutineSegmentation fault error:

pi@raspberrypi:~ $ ./hello
fatal: systemstack called from unexpected goroutineSegmentation fault

GDB reports different info this time:

pi@raspberrypi:~ $ gdb hello
GNU gdb (Raspbian 7.12-6) 7.12.0.20161007-git
Copyright (C) 2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "arm-linux-gnueabihf".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from hello...
done.
warning: Missing auto-load script at offset 0 in section .debug_gdb_scripts
of file /home/pi/hello.
Use `info auto-load python-scripts [REGEXP]' to list them.
(gdb) r
Starting program: /home/pi/hello
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/arm-linux-gnueabihf/libthread_db.so.1".
fatal: systemstack called from unexpected goroutine
Program received signal SIGSEGV, Segmentation fault.
runtime.abort () at /usr/local/go/src/runtime/asm_arm.s:801
801	/usr/local/go/src/runtime/asm_arm.s: No such file or directory.
(gdb) bt
#0  runtime.abort () at /usr/local/go/src/runtime/asm_arm.s:801
#1  0x00063a48 in runtime.badsystemstack () at /usr/local/go/src/runtime/stubs.go:62
#2  0x00049748 in runtime.throw (s=...) at /usr/local/go/src/runtime/panic.go:610
#3  0x00000012 in ?? ()
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

It is unclear where it really crashes, as it never reports the stack. I think this is causing the fatal exception.

@diamondo25
Copy link
Contributor Author

@diamondo25 diamondo25 commented Apr 20, 2020

Buildlog for v1.11 library

root@c1619b650cc7:/code# go clean -cache -modcache  -r
root@c1619b650cc7:/code# go get github.com/mattn/go-sqlite3@v1.11.0
go: finding github.com/mattn/go-sqlite3 v2.0.3+incompatible
go: finding github.com/mattn/go-sqlite3 v1.11.0
go: downloading github.com/mattn/go-sqlite3 v1.11.0
go: extracting github.com/mattn/go-sqlite3 v1.11.0
root@c1619b650cc7:/code# go clean -cache -modcache  -r
root@c1619b650cc7:/code# go build -v -x -o hello . &> buildlog_v1.11.txt

buildlog_v1.11.txt

@diamondo25
Copy link
Contributor Author

@diamondo25 diamondo25 commented Apr 20, 2020

Buildlog with latest (v2.0.3+incompatible) library

root@c1619b650cc7:/code# go clean -cache -modcache  -r
root@c1619b650cc7:/code# go get github.com/mattn/go-sqlite3        
go: finding github.com/mattn/go-sqlite3 v1.11.0
go: finding github.com/mattn/go-sqlite3 v2.0.3+incompatible
go: downloading github.com/mattn/go-sqlite3 v2.0.3+incompatible
go: extracting github.com/mattn/go-sqlite3 v2.0.3+incompatible
root@c1619b650cc7:/code# go clean -cache -modcache  -r
root@c1619b650cc7:/code# go build -v -x -o hello . &> buildlog_v2.0.3.txt

buildlog_v2.0.3.txt

@andybons andybons added this to the Unplanned milestone Apr 20, 2020
@andybons
Copy link
Member

@andybons andybons commented Apr 20, 2020

@diamondo25
Copy link
Contributor Author

@diamondo25 diamondo25 commented Apr 21, 2020

I've got myself the cables to download the packages etc. Now installing the sqlite package, so see you next year lol.

(this really takes time, as its compiling on a single core)

@diamondo25
Copy link
Contributor Author

@diamondo25 diamondo25 commented Apr 21, 2020

When I built directly on the pi, it works.
buildlog_v2.0.3.txt

total 9400
   4 drwxr-xr-x 2 pi pi    4096 Apr 21 13:33 .
   4 drwxr-xr-x 8 pi pi    4096 Apr 21 12:56 ..
   4 -rw-r--r-- 1 pi pi      28 Apr 21 13:34 buildlog_v2.0.3.txt
4692 -rwxr-xr-x 1 pi pi 4800652 Apr 21 13:34 hello
   4 -rw-r--r-- 1 pi pi      89 Apr 21 12:45 main.go
pi@raspberrypi:~/diamondo25 $ go env
GO111MODULE=""
GOARCH="arm"
GOBIN=""
GOCACHE="/home/pi/.cache/go-build"
GOENV="/home/pi/.config/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="arm"
GOHOSTOS="linux"
GOINSECURE=""
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/home/pi/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/local/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/linux_arm"
GCCGO="gccgo"
GOARM="6"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -marm -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build150874878=/tmp/go-build -gno-record-gcc-switches"


pi@raspberrypi:~/diamondo25 $ gcc -v
Using built-in specs.
COLLECT_GCC=gcc
COLLECT_LTO_WRAPPER=/usr/lib/gcc/arm-linux-gnueabihf/6/lto-wrapper
Target: arm-linux-gnueabihf
Configured with: ../src/configure -v --with-pkgversion='Raspbian 6.3.0-18+rpi1+deb9u1' --with-bugurl=file:///usr/share/doc/gcc-6/README.Bugs --enable-languages=c,ada,c++,java,go,d,fortran,objc,obj-c++ --prefix=/usr --program-suffix=-6 --program-prefix=arm-linux-gnueabihf- --enable-shared --enable-linker-build-id --libexecdir=/usr/lib --without-included-gettext --enable-threads=posix --libdir=/usr/lib --enable-nls --with-sysroot=/ --enable-clocale=gnu --enable-libstdcxx-debug --enable-libstdcxx-time=yes --with-default-libstdcxx-abi=new --enable-gnu-unique-object --disable-libitm --disable-libquadmath --enable-plugin --with-system-zlib --disable-browser-plugin --enable-java-awt=gtk --enable-gtk-cairo --with-java-home=/usr/lib/jvm/java-1.5.0-gcj-6-armhf/jre --enable-java-home --with-jvm-root-dir=/usr/lib/jvm/java-1.5.0-gcj-6-armhf --with-jvm-jar-dir=/usr/lib/jvm-exports/java-1.5.0-gcj-6-armhf --with-arch-directory=arm --with-ecj-jar=/usr/share/java/eclipse-ecj.jar --with-target-system-zlib --enable-objc-gc=auto --enable-multiarch --disable-sjlj-exceptions --with-arch=armv6 --with-fpu=vfp --with-float=hard --enable-checking=release --build=arm-linux-gnueabihf --host=arm-linux-gnueabihf --target=arm-linux-gnueabihf
Thread model: posix
gcc version 6.3.0 20170516 (Raspbian 6.3.0-18+rpi1+deb9u1)
@diamondo25
Copy link
Contributor Author

@diamondo25 diamondo25 commented Apr 21, 2020

The xgo docker image runs the following version of gcc:

root@4b1de367ed35:/code# arm-linux-gnueabihf-gcc-5 -v
Using built-in specs.
COLLECT_GCC=arm-linux-gnueabihf-gcc-5
COLLECT_LTO_WRAPPER=/usr/lib/gcc-cross/arm-linux-gnueabihf/5/lto-wrapper
Target: arm-linux-gnueabihf
Configured with: ../src/configure -v --with-pkgversion='Ubuntu/Linaro 5.4.0-6ubuntu1~16.04.4' --with-bugurl=file:///usr/share/doc/gcc-5/README.Bugs --enable-languages=c,ada,c++,java,go,d,fortran,objc,obj-c++ --prefix=/usr --program-suffix=-5 --enable-shared --enable-linker-build-id --libexecdir=/usr/lib --without-included-gettext --enable-threads=posix --libdir=/usr/lib --enable-nls --with-sysroot=/ --enable-clocale=gnu --enable-libstdcxx-debug --enable-libstdcxx-time=yes --with-default-libstdcxx-abi=new --enable-gnu-unique-object --disable-libitm --disable-libquadmath --enable-plugin --with-system-zlib --disable-browser-plugin --enable-java-awt=gtk --enable-gtk-cairo --with-java-home=/usr/lib/jvm/java-1.5.0-gcj-5-armhf-cross/jre --enable-java-home --with-jvm-root-dir=/usr/lib/jvm/java-1.5.0-gcj-5-armhf-cross --with-jvm-jar-dir=/usr/lib/jvm-exports/java-1.5.0-gcj-5-armhf-cross --with-arch-directory=arm --with-ecj-jar=/usr/share/java/eclipse-ecj.jar --disable-libgcj --enable-objc-gc --enable-multiarch --enable-multilib --disable-sjlj-exceptions --with-arch=armv7-a --with-fpu=vfpv3-d16 --with-float=hard --with-mode=thumb --disable-werror --enable-multilib --enable-checking=release --build=x86_64-linux-gnu --host=x86_64-linux-gnu --target=arm-linux-gnueabihf --program-prefix=arm-linux-gnueabihf- --includedir=/usr/arm-linux-gnueabihf/include
Thread model: posix
gcc version 5.4.0 20160609 (Ubuntu/Linaro 5.4.0-6ubuntu1~16.04.4) 

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
2 participants
You can’t perform that action at this time.