Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

runtime: panic 'exitsyscall: syscall frame is no longer valid' in go tool #24656

Closed
mappu opened this issue Apr 3, 2018 · 16 comments
Closed
Labels
FrozenDueToAge NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Milestone

Comments

@mappu
Copy link

mappu commented Apr 3, 2018

What version of Go are you using (go version)?

go version go1.9.2 linux/arm

Does this issue reproduce with the latest release?

Unsure.
It barely reproduces under the in-use version, either. This is mostly an FYI.

What operating system and processor architecture are you using (go env)?

A little bit strange one: Docker container 'arm32v7/golang:1.9.2-stretch on x86_64 host. That's an armhf rootfs where every binary executes under qemu-user.

GOARCH="arm"
GOBIN=""
GOEXE=""
GOHOSTARCH="arm"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/go"
GORACE=""
GOROOT="/usr/local/go"
GOTOOLDIR="/usr/local/go/pkg/tool/linux_arm"
GCCGO="gccgo"
GOARM="6"
CC="gcc"
GOGCCFLAGS="-fPIC -marm -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build911040395=/tmp/go-build -gno-record-gcc-switches"
CXX="g++"
CGO_ENABLED="1"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"

What did you do?

Compile a Go program.
It may be notable that the target source code was hosted on a Docker mapped volume,

What did you expect to see?

No error

What did you see instead?

The Go compiler crashed with the following message:

fatal error: exitsyscall: syscall frame is no longer valid

runtime stack:
runtime.throw(0x3d4579, 0x2d)
        /usr/local/go/src/runtime/panic.go:605 +0x70
runtime.exitsyscall.func1()
        /usr/local/go/src/runtime/proc.go:2634 +0x24
runtime.systemstack(0x1091ea80)
        /usr/local/go/src/runtime/asm_arm.s:264 +0x80
runtime.mstart()
        /usr/local/go/src/runtime/proc.go:1135

goroutine 5 [syscall]:
syscall.Syscall(0xc3, 0x10c0cb40, 0x10d26268, 0x0, 0x0, 0x3, 0x57ca9fb0)
        /usr/local/go/src/syscall/asm_linux_arm.s:17 +0x8 fp=0x10e052f8 sp=0x10e052f4 pc=0x85d14
syscall.Stat(0x3ecc94, 0x36, 0x10d26268, 0x10972c40, 0x0)
        /usr/local/go/src/syscall/zsyscall_linux_arm.go:1576 +0x70 fp=0x10e0532c sp=0x10e052f8 pc=0x85664
os.Stat(0x10b38230, 0x36, 0xf3cf8, 0x3720b8, 0x0, 0x623)
        /usr/local/go/src/os/stat_unix.go:32 +0x40 fp=0x10e05354 sp=0x10e0532c pc=0xa4ff0
syscall.Syscall6(0x36, 0xf3cf8, 0x3720b8, 0x0, 0x623, 0x0, 0x10d20760, 0x10b85900, 0x0, 0x1091e000)
        /usr/local/go/src/syscall/asm_linux_arm.s:73 +0x68 fp=0x10e05358 sp=0x10e05354 pc=0x85de8
os.StartProcess(0x10, 0x20, 0x21, 0x7, 0x3, 0xf0a74, 0x10b92fc0, 0xf667d000, 0x0)
        /usr/local/go/src/os/exec.go:94 +0x44 fp=0x10e05380 sp=0x10e05358 pc=0x9f1e8
created by cmd/go/internal/work.(*Builder).Do
        /usr/local/go/src/cmd/go/internal/work/build.go:1192 +0x44c

goroutine 1 [semacquire, 1 minutes]:
sync.runtime_Semacquire(0x10d670bc)
        /usr/local/go/src/runtime/sema.go:56 +0x2c
sync.(*WaitGroup).Wait(0x10d670b0)
        /usr/local/go/src/sync/waitgroup.go:131 +0x88
cmd/go/internal/work.(*Builder).Do(0x10954240, 0x10a76310)
        /usr/local/go/src/cmd/go/internal/work/build.go:1214 +0x46c
cmd/go/internal/work.runBuild(0x5b2da0, 0x10974068, 0x0, 0x1)
        /usr/local/go/src/cmd/go/internal/work/build.go:455 +0x4ec
main.main()
        /usr/local/go/src/cmd/go/main.go:133 +0x670

goroutine 19 [syscall, 1 minutes]:
os/signal.signal_recv(0x0)
        /usr/local/go/src/runtime/sigqueue.go:131 +0x134
os/signal.loop()
        /usr/local/go/src/os/signal/signal_unix.go:22 +0x14
created by os/signal.init.0
        /usr/local/go/src/os/signal/signal_unix.go:28 +0x30

goroutine 6 [syscall]:
syscall.Syscall6(0x118, 0x1, 0x629, 0x10e07674, 0x1000004, 0x0, 0x0, 0x379230, 0x99d04, 0x0)
        /usr/local/go/src/syscall/asm_linux_arm.s:48 +0x8
os.(*Process).blockUntilWaitable(0x10c83920, 0x0, 0x17, 0x18)
        /usr/local/go/src/os/wait_waitid.go:31 +0x64
os.(*Process).wait(0x10c83920, 0x10bd8edc, 0x1cab84, 0x10bd8edc)
        /usr/local/go/src/os/exec_unix.go:22 +0x2c
os.(*Process).Wait(0x10c83920, 0x0, 0x0, 0x3ecbb0)
        /usr/local/go/src/os/exec.go:115 +0x1c
os/exec.(*Cmd).Wait(0x10bd8e70, 0x0, 0x0)
        /usr/local/go/src/os/exec/exec.go:446 +0x40
os/exec.(*Cmd).Run(0x10bd8e70, 0x0, 0x0)
        /usr/local/go/src/os/exec/exec.go:289 +0x44
cmd/go/internal/work.(*Builder).runOut(0x10954240, 0x10bbef90, 0x2f, 0x10b5b8c0, 0x1d, 0x0, 0x0, 0x0, 0x10bd8dc0, 0x10, ...)
        /usr/local/go/src/cmd/go/internal/work/build.go:1983 +0x378
cmd/go/internal/work.gcToolchain.gc(0x10954240, 0x10c466c0, 0x10b90100, 0x36, 0x10b900c0, 0x3a, 0x10914b00, 0x10c55340, 0x2, 0x2, ...)
        /usr/local/go/src/cmd/go/internal/work/build.go:2272 +0xc14
cmd/go/internal/work.(*gcToolchain).gc(0x5c7fa8, 0x10954240, 0x10c466c0, 0x10b90100, 0x36, 0x10b900c0, 0x3a, 0x0, 0x10c55340, 0x2, ...)
        <autogenerated>:1 +0x94
cmd/go/internal/work.(*Builder).build(0x10954240, 0x10910850, 0x0, 0x0)
        /usr/local/go/src/cmd/go/internal/work/build.go:1393 +0x1048
cmd/go/internal/work.(*Builder).Do.func1(0x10910850)
        /usr/local/go/src/cmd/go/internal/work/build.go:1148 +0x58
cmd/go/internal/work.(*Builder).Do.func2(0x10d670b0, 0x10954240, 0x10ce2270)
        /usr/local/go/src/cmd/go/internal/work/build.go:1205 +0x84
created by cmd/go/internal/work.(*Builder).Do
        /usr/local/go/src/cmd/go/internal/work/build.go:1192 +0x44c
@mappu
Copy link
Author

mappu commented Apr 3, 2018

Another crash in the same environment:

fatal error: exitsyscall: syscall frame is no longer valid
fatal error: stopm holding locks
panic during panic
stack trace unavailable

runtime stack:

@odeke-em
Copy link
Member

odeke-em commented Apr 3, 2018

Hello there @mappu, thank you for filing this issue!

Could you please provide a minimal reproduction that's runnable for anyone? That helps a lot in prognosis/diagnosis of the issue affecting you, and makes bug fixing faster. Thank you!

@odeke-em odeke-em added the WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided. label Apr 3, 2018
@ianlancetaylor
Copy link
Contributor

Is the stopm holding locks case also from running the go tool?

@ianlancetaylor ianlancetaylor added NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. and removed WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided. labels Apr 18, 2018
@ianlancetaylor ianlancetaylor added this to the Go1.11 milestone Apr 18, 2018
@ianlancetaylor ianlancetaylor changed the title runtime: panic 'exitsyscall: syscall frame is no longer valid' in go compiler runtime: panic 'exitsyscall: syscall frame is no longer valid' in go tool Apr 18, 2018
@mappu
Copy link
Author

mappu commented Apr 18, 2018

@ianlancetaylor yes, it was.

I tried to reproduce this today on a different machine. I got some different failures instead.

The basic steps to reproduce:

  1. Set up qemu-user-static and binfmt-support to allow running armhf binaries from x86_64 hardware (my qemu is 2.8+dfsg-6+deb9u3)
  2. docker run --rm -it arm32v7/golang:1.9.2-stretch /bin/bash
  3. (inside container) Clone go.git, then run make.bash

First attempt:

root@container:~/go/src# ./make.bash
Building Go cmd/dist using /usr/local/go.
Building Go toolchain1 using /usr/local/go.
Building Go bootstrap cmd/go (go_bootstrap) using Go toolchain1.
qemu-arm-static: /build/qemu-DqynNa/qemu-2.8+dfsg/translate-all.c:175: tb_lock: Assertion `!have_tb_lock' failed.
qemu-arm-static: /build/qemu-DqynNa/qemu-2.8+dfsg/translate-all.c:175: tb_lock: Assertion `!have_tb_lock' failed.
./make.bash: line 196:   237 Segmentation fault      (core dumped) ./cmd/dist/dist bootstrap $buildall $vflag $GO_DISTFLAGS "$@"

Go binary segfaulted, but with qemu displaying some warnings.

Second attempt:

root@container:~/go/src# ./make.bash
Building Go cmd/dist using /usr/local/go.
Building Go toolchain1 using /usr/local/go.

^C

the process hung and must be stopped.

Third attempt:

root@container:~/go/src# ./make.bash
Building Go cmd/dist using /usr/local/go.
Building Go toolchain1 using /usr/local/go.
Building Go bootstrap cmd/go (go_bootstrap) using Go toolchain1.
Building Go toolchain2 using go_bootstrap and Go toolchain1.
fatal error: sigaction failed

runtime stack:
runtime.throw(0x295c16, 0x10)
        /root/go/src/runtime/panic.go:589 +0x54
runtime.sysSigaction.func1()
        /root/go/src/runtime/os_linux.go:419 +0x24
runtime.sysSigaction(0x40, 0xf6fffba4, 0x0)
        /root/go/src/runtime/os_linux.go:418 +0x44
runtime.sigaction(0x40, 0xf6fffba4, 0x0)
        /root/go/src/runtime/sigaction.go:15 +0x20
runtime.setsig(0x40, 0x4dbac)
        /root/go/src/runtime/os_linux.go:381 +0xbc
runtime.initsig(0x1200)
        /root/go/src/runtime/signal_unix.go:111 +0x10c
runtime.mstartm0()
        /root/go/src/runtime/proc.go:1254 +0x3c
runtime.mstart1(0x0)
        /root/go/src/runtime/proc.go:1225 +0xc8
runtime.mstart()
        /root/go/src/runtime/proc.go:1195 +0x60

goroutine 1 [runnable]:
runtime.main()
        /root/go/src/runtime/proc.go:109
runtime.goexit()
        /root/go/src/runtime/asm_arm.s:840 +0x4
go tool dist: FAILED: /root/go/pkg/tool/linux_arm/go_bootstrap install -gcflags=all= -ldflags=all= -i cmd/asm cmd/cgo cmd/compile cmd/link: exit status 2
root@34061c589a23:~/go/src# 

This time a crash in Go runtime package. But it's a different panic compared to the earlier ones.

Fourth attempt:

root@container:~/go/src# ./make.bash
Building Go cmd/dist using /usr/local/go.
Building Go toolchain1 using /usr/local/go.
Building Go bootstrap cmd/go (go_bootstrap) using Go toolchain1.
qemu-arm-static: /build/qemu-DqynNa/qemu-2.8+dfsg/translate-all.c:175: tb_lock: Assertion `!have_tb_lock' failed.
qemu-arm-static: /build/qemu-DqynNa/qemu-2.8+dfsg/translate-all.c:175: tb_lock: Assertion `!have_tb_lock' failed.
./make.bash: line 196:  5352 Segmentation fault      (core dumped) ./cmd/dist/dist bootstrap $buildall $vflag $GO_DISTFLAGS "$@"

Go binary segfaulted, again with the qemu failures.

Fives attempt:

root@container:~/go/src# ./make.bash
Building Go cmd/dist using /usr/local/go.
Building Go toolchain1 using /usr/local/go.
Building Go bootstrap cmd/go (go_bootstrap) using Go toolchain1.
./make.bash: line 196:  6624 Segmentation fault      (core dumped) ./cmd/dist/dist bootstrap $buildall $vflag $GO_DISTFLAGS "$@"

Go binary segfaulted, but no qemu failure messages.

Go works reliably when running on a real machine. I think these issues in runtime are probably something about this weird qemu&docker environment, not really Go's fault. So if that's out-of-scope, then please close this ticket.

@ianlancetaylor
Copy link
Contributor

Thanks for the info. It's not out of scope but I don't know who is going to be able to fix it.

@ALTree
Copy link
Member

ALTree commented May 2, 2018

FWIW I can reproduce this without docker. It appears that go1.11 arm and arm64 binaries cannot be run under qemu user emulation (this is a "regression", since 1.10 binaries work).

On a linux/amd64 system with qemu-user installed:

$ go version
go version go1.10 linux/amd64
$ GOARCH=arm64 go build test.go; ./test
Hello, world!

$ gotip version
go version devel +500d79c410 Wed May 2 04:20:25 2018 +0000 linux/amd64
$ GOARCH=arm64 gotip build test.go; ./test
fatal error: sigaction failed

runtime stack:
runtime.throw(0xc0e2f, 0x10)
	/home/adonizetti/go/src/runtime/panic.go:589 +0x48
runtime.sysSigaction.func1()
	/home/adonizetti/go/src/runtime/os_linux.go:420 +0x2c
runtime.sysSigaction(0x40, 0x4000800250, 0x0)
	/home/adonizetti/go/src/runtime/os_linux.go:419 +0x48
runtime.sigaction(0x40, 0x4000800250, 0x0)
	/home/adonizetti/go/src/runtime/sigaction.go:15 +0x20
runtime.setsig(0x40, 0x446b0)
	/home/adonizetti/go/src/runtime/os_linux.go:382 +0xb8
runtime.initsig(0x4000003800)
	/home/adonizetti/go/src/runtime/signal_unix.go:113 +0x178
runtime.mstartm0()
	/home/adonizetti/go/src/runtime/proc.go:1254 +0x3c
runtime.mstart1()
	/home/adonizetti/go/src/runtime/proc.go:1225 +0xcc
runtime.mstart()
	/home/adonizetti/go/src/runtime/proc.go:1195 +0x50

goroutine 1 [runnable]:
runtime.main()
	/home/adonizetti/go/src/runtime/proc.go:109
runtime.goexit()
	/home/adonizetti/go/src/runtime/asm_arm64.s:1084 +0x4

I know we don't guarantee that Go binaries work under qemu user emulation mode, but it was nice to be able to use it...

@ianlancetaylor
Copy link
Contributor

Can you find out why it is failing? The stack trace indicates that the sigaction system call failed, but why?

@ALTree
Copy link
Member

ALTree commented May 2, 2018

Tracing says 29923 rt_sigaction(64,0xfffef304,NULL) = -1 errno=22 (Invalid argument).

Command and full trace:

$ GOARCH=arm gotip build test.go; qemu-arm -strace ./test
29923 getpid() = 29923
29923 sched_getaffinity(0,8192,-76952,8964,0,31) = 64
29923 mmap2(NULL,262144,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANONYMOUS,-1,0) = 0xff7af000
29923 mmap2(NULL,270532608,PROT_NONE,MAP_PRIVATE|MAP_ANONYMOUS,-1,0) = 0xef5af000
29923 brk(NULL) = 0x00132000
29923 mmap2(0x00400000,541065216,PROT_NONE,MAP_PRIVATE|MAP_ANONYMOUS,-1,0) = 0x00400000
29923 clock_gettime(1,-68856,1597334677,0,-68840,1177936) = 0
29923 mmap2(0x00400000,4194304,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANONYMOUS|MAP_FIXED,-1,0) = 0x00400000
29923 mmap2(0xef5af000,266240,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANONYMOUS|MAP_FIXED,-1,0) = 0xef5af000
29923 mmap2(NULL,65536,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANONYMOUS,-1,0) = 0xef59f000
29923 clock_gettime(1,-69384,1,1194560,-69364,1177936) = 0
29923 mmap2(NULL,65536,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANONYMOUS,-1,0) = 0xef58f000
29923 rt_sigprocmask(SIG_SETMASK,NULL,0x0011f994) = 0
29923 clock_gettime(1,-68824,1194560,0,-68804,1177936) = 0
29923 clock_gettime(1,-68920,0,44,-68900,1177936) = 0
29923 sigaltstack((nil),0xfffef338) = 0
29923 sigaltstack(0xfffef31c,(nil)) = 0
29923 rt_sigprocmask(SIG_SETMASK,0xfffef33c,NULL) = 0
29923 gettid() = 29923
29923 rt_sigaction(SIGHUP,NULL,0xfffef31c) = 0
29923 rt_sigaction(SIGHUP,0xfffef304,NULL) = 0
29923 rt_sigaction(SIGINT,NULL,0xfffef31c) = 0
29923 rt_sigaction(SIGINT,0xfffef304,NULL) = 0
29923 rt_sigaction(SIGQUIT,NULL,0xfffef31c) = 0
29923 rt_sigaction(SIGQUIT,0xfffef304,NULL) = 0
29923 rt_sigaction(SIGILL,NULL,0xfffef31c) = 0
29923 rt_sigaction(SIGILL,0xfffef304,NULL) = 0
29923 rt_sigaction(5,NULL,0xfffef31c) = 0
29923 rt_sigaction(5,0xfffef304,NULL) = 0
29923 rt_sigaction(SIGABRT,NULL,0xfffef31c) = 0
29923 rt_sigaction(SIGABRT,0xfffef304,NULL) = 0
29923 rt_sigaction(7,NULL,0xfffef31c) = 0
29923 rt_sigaction(7,0xfffef304,NULL) = 0
29923 rt_sigaction(SIGFPE,NULL,0xfffef31c) = 0
29923 rt_sigaction(SIGFPE,0xfffef304,NULL) = 0
29923 rt_sigaction(SIGUSR1,NULL,0xfffef31c) = 0
29923 rt_sigaction(SIGUSR1,0xfffef304,NULL) = 0
29923 rt_sigaction(SIGSEGV,NULL,0xfffef31c) = 0
29923 rt_sigaction(SIGSEGV,0xfffef304,NULL) = 0
29923 rt_sigaction(SIGUSR2,NULL,0xfffef31c) = 0
29923 rt_sigaction(SIGUSR2,0xfffef304,NULL) = 0
29923 rt_sigaction(SIGPIPE,NULL,0xfffef31c) = 0
29923 rt_sigaction(SIGPIPE,0xfffef304,NULL) = 0
29923 rt_sigaction(SIGALRM,NULL,0xfffef31c) = 0
29923 rt_sigaction(SIGALRM,0xfffef304,NULL) = 0
29923 rt_sigaction(SIGTERM,NULL,0xfffef31c) = 0
29923 rt_sigaction(SIGTERM,0xfffef304,NULL) = 0
29923 rt_sigaction(16,NULL,0xfffef31c) = 0
29923 rt_sigaction(16,0xfffef304,NULL) = 0
29923 rt_sigaction(SIGCHLD,NULL,0xfffef31c) = 0
29923 rt_sigaction(SIGCHLD,0xfffef304,NULL) = 0
29923 rt_sigaction(23,NULL,0xfffef31c) = 0
29923 rt_sigaction(23,0xfffef304,NULL) = 0
29923 rt_sigaction(24,NULL,0xfffef31c) = 0
29923 rt_sigaction(24,0xfffef304,NULL) = 0
29923 rt_sigaction(25,NULL,0xfffef31c) = 0
29923 rt_sigaction(25,0xfffef304,NULL) = 0
29923 rt_sigaction(26,NULL,0xfffef31c) = 0
29923 rt_sigaction(26,0xfffef304,NULL) = 0
29923 rt_sigaction(27,NULL,0xfffef31c) = 0
29923 rt_sigaction(27,0xfffef304,NULL) = 0
29923 rt_sigaction(28,NULL,0xfffef31c) = 0
29923 rt_sigaction(28,0xfffef304,NULL) = 0
29923 rt_sigaction(29,NULL,0xfffef31c) = 0
29923 rt_sigaction(29,0xfffef304,NULL) = 0
29923 rt_sigaction(30,NULL,0xfffef31c) = 0
29923 rt_sigaction(30,0xfffef304,NULL) = 0
29923 rt_sigaction(31,NULL,0xfffef31c) = 0
29923 rt_sigaction(31,0xfffef304,NULL) = 0
29923 rt_sigaction(32,NULL,0xfffef31c) = 0
29923 rt_sigaction(33,NULL,0xfffef31c) = 0
29923 rt_sigaction(34,NULL,0xfffef31c) = 0
29923 rt_sigaction(34,0xfffef304,NULL) = 0
29923 rt_sigaction(35,NULL,0xfffef31c) = 0
29923 rt_sigaction(35,0xfffef304,NULL) = 0
29923 rt_sigaction(36,NULL,0xfffef31c) = 0
29923 rt_sigaction(36,0xfffef304,NULL) = 0
29923 rt_sigaction(37,NULL,0xfffef31c) = 0
29923 rt_sigaction(37,0xfffef304,NULL) = 0
29923 rt_sigaction(38,NULL,0xfffef31c) = 0
29923 rt_sigaction(38,0xfffef304,NULL) = 0
29923 rt_sigaction(39,NULL,0xfffef31c) = 0
29923 rt_sigaction(39,0xfffef304,NULL) = 0
29923 rt_sigaction(40,NULL,0xfffef31c) = 0
29923 rt_sigaction(40,0xfffef304,NULL) = 0
29923 rt_sigaction(41,NULL,0xfffef31c) = 0
29923 rt_sigaction(41,0xfffef304,NULL) = 0
29923 rt_sigaction(42,NULL,0xfffef31c) = 0
29923 rt_sigaction(42,0xfffef304,NULL) = 0
29923 rt_sigaction(43,NULL,0xfffef31c) = 0
29923 rt_sigaction(43,0xfffef304,NULL) = 0
29923 rt_sigaction(44,NULL,0xfffef31c) = 0
29923 rt_sigaction(44,0xfffef304,NULL) = 0
29923 rt_sigaction(45,NULL,0xfffef31c) = 0
29923 rt_sigaction(45,0xfffef304,NULL) = 0
29923 rt_sigaction(46,NULL,0xfffef31c) = 0
29923 rt_sigaction(46,0xfffef304,NULL) = 0
29923 rt_sigaction(47,NULL,0xfffef31c) = 0
29923 rt_sigaction(47,0xfffef304,NULL) = 0
29923 rt_sigaction(48,NULL,0xfffef31c) = 0
29923 rt_sigaction(48,0xfffef304,NULL) = 0
29923 rt_sigaction(49,NULL,0xfffef31c) = 0
29923 rt_sigaction(49,0xfffef304,NULL) = 0
29923 rt_sigaction(50,NULL,0xfffef31c) = 0
29923 rt_sigaction(50,0xfffef304,NULL) = 0
29923 rt_sigaction(51,NULL,0xfffef31c) = 0
29923 rt_sigaction(51,0xfffef304,NULL) = 0
29923 rt_sigaction(52,NULL,0xfffef31c) = 0
29923 rt_sigaction(52,0xfffef304,NULL) = 0
29923 rt_sigaction(53,NULL,0xfffef31c) = 0
29923 rt_sigaction(53,0xfffef304,NULL) = 0
29923 rt_sigaction(54,NULL,0xfffef31c) = 0
29923 rt_sigaction(54,0xfffef304,NULL) = 0
29923 rt_sigaction(55,NULL,0xfffef31c) = 0
29923 rt_sigaction(55,0xfffef304,NULL) = 0
29923 rt_sigaction(56,NULL,0xfffef31c) = 0
29923 rt_sigaction(56,0xfffef304,NULL) = 0
29923 rt_sigaction(57,NULL,0xfffef31c) = 0
29923 rt_sigaction(57,0xfffef304,NULL) = 0
29923 rt_sigaction(58,NULL,0xfffef31c) = 0
29923 rt_sigaction(58,0xfffef304,NULL) = 0
29923 rt_sigaction(59,NULL,0xfffef31c) = 0
29923 rt_sigaction(59,0xfffef304,NULL) = 0
29923 rt_sigaction(60,NULL,0xfffef31c) = 0
29923 rt_sigaction(60,0xfffef304,NULL) = 0
29923 rt_sigaction(61,NULL,0xfffef31c) = 0
29923 rt_sigaction(61,0xfffef304,NULL) = 0
29923 rt_sigaction(62,NULL,0xfffef31c) = 0
29923 rt_sigaction(62,0xfffef304,NULL) = 0
29923 rt_sigaction(63,NULL,0xfffef31c) = 0
29923 rt_sigaction(63,0xfffef304,NULL) = 0
29923 rt_sigaction(64,NULL,0xfffef31c) = 0
29923 rt_sigaction(64,0xfffef304,NULL) = -1 errno=22 (Invalid argument)
29923 write(2,0xb134a,13)fatal error:  = 13
29923 write(2,0xb1a98,16)sigaction failed = 16
29923 write(2,0xb017b,1)
 = 1

[other runtime/panic calls elided]

@ianlancetaylor
Copy link
Contributor

Thanks. So the problem seems to be that QEMU does not permit us to set the signal handler for signal 64. That seems like a bug in QEMU. And what has changed is probably that we used to simply ignore the result of sigaction, but now, as of https://golang.org/cl/99077, we check it.

@ALTree
Copy link
Member

ALTree commented May 2, 2018

I think we re-discovered this issue from 2013 (or it went away and now it's back):

https://groups.google.com/d/topic/golang-nuts/MqKTX_XIOKE/discussion

@ianlancetaylor
Copy link
Contributor

I think you're right. Perhaps someone could actually report it to QEMU.

@ALTree
Copy link
Member

ALTree commented May 3, 2018

I think they're aware. From here: https://git.qemu.org/?p=qemu.git;a=blob;f=linux-user/signal.c;h=a3022c2f04e83ddf370d30905e51b6fa62c1f23d;hb=HEAD#l77

/* Nasty hack: Reverse SIGRTMIN and SIGRTMAX to avoid overlap with
   host libpthread signals.  This assumes no one actually uses SIGRTMAX :-/
   To fix this properly we need to do manual signal delivery multiplexed
   over a single host signal.  */
[__SIGRTMIN] = __SIGRTMAX,
[__SIGRTMAX] = __SIGRTMIN,

I couldn't find a bug in their tracker, but I found a thread where someone was complaining about not being able to run Go binaries: https://lists.gnu.org/archive/html/qemu-discuss/2016-03/msg00002.html

Also it appears that we had workarounds in place for this issue in the past:

https://codereview.appspot.com/124900043/
https://go-review.googlesource.com/c/go/+/16853

Either we add the signal 64 workaround back or we declare once and for all that running Go binaries under qemu-user is not supported and it's not supposed to work (until they change the way they handle signals).

@gopherbot
Copy link

Change https://golang.org/cl/111176 mentions this issue: runtime: ignore sigaction error if it is for SIGRTMAX

@ALTree
Copy link
Member

ALTree commented May 3, 2018

I've verified that, with the change linked above applied on tip, Go binaries no longer crash at start-up under qemu-user.

@ianlancetaylor
Copy link
Contributor

Thanks, let's do that, I guess.

gopherbot pushed a commit that referenced this issue May 4, 2018
The Go runtime registers a handler for every signal. This prevents Go
binaries from working on QEMU in user-emulation mode, since the hacky
way QEMU implements signals on Linux assumes that no-one uses signal
64 (SIGRTMAX).

In the past, we had a workaround in the runtime to prevent crashes on
start-up when running on QEMU:

  golang.org/cl/124900043
  golang.org/cl/16853

but it went lost during the 1.11 dev cycle. More precisely, the test
for SIGRTMAX was dropped in CL 18150 when we stopped testing the
result of sigaction in the Linux implementation of setsig. That change
was made to avoid a stack split overflow because code started calling
setsig from nosplit functions. Then in CL 99077 we started testing the
result of sigaction again, this time using systemstack to avoid to
stack split overflow. When this test was added back, we did not bring
back the test of SIGRTMAX.

As a result, Go1.10 binaries work on QEMU, while 1.11 binaries
immediately crash on startup.

This change restores the QEMU workaround.

Updates #24656

Change-Id: I46380b1e1b4bf47db7bc7b3d313f00c4e4c11ea3
Reviewed-on: https://go-review.googlesource.com/111176
Reviewed-by: Ian Lance Taylor <iant@golang.org>
@ianlancetaylor
Copy link
Contributor

As far as I can tell, this is fixed, so closing.

fstab added a commit to fstab/docker-grok_exporter-compiler that referenced this issue Oct 7, 2018
@golang golang locked and limited conversation to collaborators Jul 10, 2019
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
FrozenDueToAge NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Projects
None yet
Development

No branches or pull requests

5 participants