runtime: cannot run cross compiled ARM binary on QEMU

[DeadNumbers]

What version of Go are you using (go version)?

go version go1.8.3 linux/amd64

What operating system and processor architecture are you using (go env)?

GOARCH="amd64"
GOBIN=""
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/home/user/go"
GORACE=""
GOROOT="/usr/lib/go"
GOTOOLDIR="/usr/lib/go/pkg/tool/linux_amd64"
GCCGO="gccgo"
CC="gcc"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build057525304=/tmp/go-build -gno-record-gcc-switches"
CXX="g++"
CGO_ENABLED="1"
PKG_CONFIG="pkg-config"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"

What did you do?

I'm try compile for ARMv* and test it on QEMU (qemu version 2.9.0)

Source

$ cat hello.go
package main

import "fmt"

func main() {
	fmt.Printf("Hello ARM world!\n")
}

Build GOARCH=arm64 go build -o hello_go hello.go and run

$ qemu-aarch64 hello_go 
runtime: failed to create new OS thread (have 2 already; errno=22)
fatal error: newosproc

runtime stack:
runtime.throw(0xb8109, 0x9)
	/usr/lib/go/src/runtime/panic.go:596 +0x70
runtime.newosproc(0x1442002a000, 0x1442003a000)
	/usr/lib/go/src/runtime/os_linux.go:163 +0x168
runtime.newm(0xbe310, 0x0)
	/usr/lib/go/src/runtime/proc.go:1628 +0x110
runtime.main.func1()
	/usr/lib/go/src/runtime/proc.go:126 +0x28
runtime.systemstack(0x114a00)
	/usr/lib/go/src/runtime/asm_arm64.s:241 +0x90
runtime.mstart()
	/usr/lib/go/src/runtime/proc.go:1132

goroutine 1 [running]:
runtime.systemstack_switch()
	/usr/lib/go/src/runtime/asm_arm64.s:190 +0x8 fp=0x14420026780 sp=0x14420026770
runtime.main()
	/usr/lib/go/src/runtime/proc.go:127 +0x58 fp=0x144200267d0 sp=0x14420026780
runtime.goexit()
	/usr/lib/go/src/runtime/asm_arm64.s:981 +0x4 fp=0x144200267d0 sp=0x144200267d0

For ARMv5 GOARCH=arm go build -o hello_go hello.go

$ qemu-arm hello_go                  
runtime: failed to create new OS thread (have 2 already; errno=22)
fatal error: newosproc

runtime stack:
runtime.throw(0xaba2e, 0x9)
	/usr/lib/go/src/runtime/panic.go:596 +0x70
runtime.newosproc(0x10426000, 0x10436000)
	/usr/lib/go/src/runtime/os_linux.go:163 +0x158
runtime.newm(0xb1c98, 0x0)
	/usr/lib/go/src/runtime/proc.go:1628 +0x10c
runtime.main.func1()
	/usr/lib/go/src/runtime/proc.go:126 +0x24
runtime.systemstack(0x103000)
	/usr/lib/go/src/runtime/asm_arm.s:264 +0x80
runtime.mstart()
	/usr/lib/go/src/runtime/proc.go:1132

goroutine 1 [running]:
runtime.systemstack_switch()
	/usr/lib/go/src/runtime/asm_arm.s:209 +0x4 fp=0x104227bc sp=0x104227b8
runtime.main()
	/usr/lib/go/src/runtime/proc.go:127 +0x58 fp=0x104227e4 sp=0x104227bc
runtime.goexit()
	/usr/lib/go/src/runtime/asm_arm.s:1017 +0x4 fp=0x104227e4 sp=0x104227e4

I'm not sure that the error in the Go compiler, but

#include <stdio.h>

int main(void) {
    printf("Hello ARM world!\n");
}

$ aarch64-linux-gnu-gcc hello.c -static -o hello_c && qemu-aarch64 hello_c
Hello ARM world!

[odeke-em]

/cc @aclements @ianlancetaylor @rsc

[ALTree]

qemu in user-mode emulation is not supported, and I don't think it ever worked well (at least not in recent times). See this comment #13024 (comment)

[bradfitz]

Working around QEMU bugs is not a priority for the Go team so I won't target this for any release, but we can keep it open for tracking. We test on real hardware. And QEMU in full system emulation mode works as well. But if somebody else wants to investigate this, it's all yours.

[ncw]

Here is a patch for go which fixes the issue.

It might even be correct but I don't really know!

Maybe the issue should be re-opened?

See also ubuntu bug tracker.

diff --git a/src/runtime/os_linux.go b/src/runtime/os_linux.go
index a6efc0e3d1..64218e3f7e 100644
--- a/src/runtime/os_linux.go
+++ b/src/runtime/os_linux.go
@@ -132,7 +132,8 @@ const (
 		_CLONE_FS | /* share cwd, etc */
 		_CLONE_FILES | /* share fd table */
 		_CLONE_SIGHAND | /* share sig handler table */
-		_CLONE_THREAD /* revisit - okay for now */
+		_CLONE_THREAD | /* revisit - okay for now */
+		_CLONE_SYSVSEM
 )
 
 //go:noescape

[bradfitz]

Sure, I'll reopen given the renewed discussion and investigation in that Ubuntu bug.

@aclements, @ianlancetaylor?

[aclements]

Yikes. Clone and SysV semaphores. Two of my favorite things.

@ncw's patch is clearly safe and arguably even the right thing to do (though since Go never exits Go-created threads I don't think the difference is actually observable even if you tried, but I've got patches lined up that will change that). glibc passes CLONE_SYSVSEM when creating threads and it has the right semantics for threads.

I'll send this as a CL in a bit (unless you want to send it @ncw).