cmd/compile: illegal instruction in cgo call mygetgrgid_r on arm (cross)

[paralin]

Background info

• Cross compiling from amd64 -> armv7
• This is a bug I'm fixing on behalf of the Buildroot project
• The error appears to be a cgo illegal instruction
• Device is a Bananapi M1 Plus
• A20 ARM Cortex-A7 Dual-core

What version of Go are you using (go version)?

$ go version
go version go1.11.2 linux/amd64

Does this issue reproduce with the latest release?

Yes. Reproduce by compiling dockerd with:

  GOARCH=arm
  GOARM=7
  GOROOT="/workspaces/m1_plus/host/lib/go"
  CC="/workspaces/m1_plus/host/bin/arm-linux-gnueabihf-gcc"
  CXX="/workspaces/m1_plus/host/bin/arm-linux-gnueabihf-g++"
  GOTOOLDIR="/workspaces/m1_plus/host/lib/go/pkg/tool/linux_arm"
  PATH="/workspaces/m1_plus/host/bin:/workspaces/m1_plus/host/sbin:/usr/x86_64-pc-linux-gnu/gcc-bin/6.4.0:/usr/lib/llvm/5/bin:/usr/local/bin:/usr/bin:/bin:/opt/bin:/opt/golang/bin"
  GOBIN=
  CGO_ENABLED=1
  GOPATH="/workspaces/m1_plus/build/docker-engine-v18.06.1-ce/_gopath"
  /workspaces/m1_plus/host/bin/go build -v -asmflags "-trimpath=/workspaces/m1_plus/build/" -gcflags \
  "-trimpath=/workspaces/m1_plus/build/" \
  -ldflags "-X main.GitCommit=v18.06.1-ce -X main.Version=v18.06.1-ce" \
  -tags "cgo exclude_graphdriver_zfs autogen systemd journald exclude_graphdriver_btrfs exclude_graphdriver_devicemapper exclude_graphdriver _vfs" \
  -o /workspaces/m1_plus/build/docker-engine-v18.06.1-ce/bin/dockerd
  ./cmd/dockerd

The Docker daemon initially works correctly on the target system, but if you run "docker build ." the cgo call mygetgrgid_r causes the SIGILL.

What operating system and processor architecture are you using (go env)?

go env Output

$ go env
GOARCH="arm"
GOBIN=""
GOCACHE="/home/paralin/.cache/go-build"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/home/paralin/Documents/synrobo/skiff/workspaces/m1_plus/build/docker-engine-v18.06.1-ce/_gopath"
GOPROXY=""
GORACE=""
GOROOT="/home/paralin/Documents/synrobo/skiff/workspaces/m1_plus/host/lib/go"
GOTMPDIR=""
GOTOOLDIR="/home/paralin/Documents/synrobo/skiff/workspaces/m1_plus/host/lib/go/pkg/tool/linux_amd64"
GCCGO="gccgo"
GOARM="7"
CC="/home/paralin/Documents/synrobo/skiff/workspaces/m1_plus/host/bin/arm-linux-gnueabihf-gcc"
CXX="/home/paralin/Documents/synrobo/skiff/workspaces/m1_plus/host/bin/arm-linux-gnueabihf-g++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -marm -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build687750654=/tmp/go-build -gno-record-gcc-switches"                                                  

What did you do?

• Compile dockerd using the above commands
• Execute dockerd on the target system
• Use the Docker CLI to try to "docker build ."

Note: this bug only happens if the FROM declaration in the Dockerfile specifies a docker image that does not currently exist on the system (the cgo call must happen during the image pull phase).

What did you expect to see?

Program runs correctly

What did you see instead?

SIGILL illegal instruction

# docker build .
SIGILL: illegal instruction
PC=0x4f64e m=7 sigcode=1

goroutine 0 [idle]:
runtime: unexpected return pc for runtime.dumpregs called from 0xdabba0
stack: frame={sp:0xa3ffeccc, fp:0xa3ffed2c} stack=[0xa37ff0d4,0xa3ffecd4)
a3ffec4c:  00000000  00000002  01eeb980  00000000 
a3ffec5c:  a439fad5  00000077  0000007c  00000001 
a3ffec6c:  a2e00010  00000400  020589c8  02059000 
a3ffec7c:  00000002  00ebd330  0204e780  00000008 
a3ffec8c:  02000d20  0000001c  00dabc4c  02145340 
a3ffec9c:  00dabbec  00000670  020589c8  00000002 
a3ffecac:  00066ebc <runtime.asmcgocall+108>  01ed9604  00000001  a4ccb060 
a3ffecbc:  a4ccb060  00000670  023c6b60  00000000 
a3ffeccc: <00dabba0  00000000 
runtime.dumpregs(0xa650ca76)
        /home/paralin/workspaces/m1_plus/host/lib/go/src/runtime/signal_arm.go:27 +0x4ce

goroutine 28 [syscall]:
runtime.cgocall(0xdabc1c, 0x20589c8, 0x29)
        /home/paralin/workspaces/m1_plus/host/lib/go/src/runtime/cgocall.go:128 +0x5c fp=0x20589ac sp=0x2058994 pc=0x127d4
os/user._Cfunc_mygetgrgid_r(0x0, 0x232eb30, 0xa2e00470, 0x400, 0x2145340, 0x0)
        _cgo_gotypes.go:104 +0x38 fp=0x20589c4 sp=0x20589ac pc=0x3c1fd8
os/user.lookupUnixGid.func1.1(0x0, 0x232eb30, 0xa2e00470, 0x400, 0x2145340, 0xa2e00470)
        /home/paralin/workspaces/m1_plus/host/lib/go/src/os/user/cgo_lookup_unix.go:181 +0xfc fp=0x20589e0 sp=0x20589c4 pc=0x3c39f8
os/user.lookupUnixGid.func1(0x8)
        /home/paralin/workspaces/m1_plus/host/lib/go/src/os/user/cgo_lookup_unix.go:181 +0x40 fp=0x20589fc sp=0x20589e0 pc=0x3c3a94
os/user.retryWithBuffer(0x2145348, 0x2058a70, 0x2145348, 0x0)
        /home/paralin/workspaces/m1_plus/host/lib/go/src/os/user/cgo_lookup_unix.go:253 +0x2c fp=0x2058a24 sp=0x20589fc pc=0x3c2f40
os/user.lookupUnixGid(0x0, 0x0, 0x0, 0x0)
        /home/paralin/workspaces/m1_plus/host/lib/go/src/os/user/cgo_lookup_unix.go:177 +0xc8 fp=0x2058a84 sp=0x2058a24 pc=0x3c2ad4
os/user.lookupGroupId(0x10b7b40, 0x1, 0xa, 0x10b7b40, 0x1)
        /home/paralin/workspaces/m1_plus/host/lib/go/src/os/user/cgo_lookup_unix.go:167 +0x54 fp=0x2058a9c sp=0x2058a84 pc=0x3c29e4
os/user.LookupGroupId(0x10b7b40, 0x1, 0x1, 0x0, 0x0)
        /home/paralin/workspaces/m1_plus/host/lib/go/src/os/user/lookup.go:53 +0x24 fp=0x2058ab4 sp=0x2058a9c pc=0x3c1d60
archive/tar.statUnix(0x12c09f0, 0x24f8480, 0x22bbae0, 0xd374c270, 0xe)
        /home/paralin/workspaces/m1_plus/host/lib/go/src/archive/tar/stat_unix.go:45 +0x5ac fp=0x2058be8 sp=0x2058ab4 pc=0x3cca84
archive/tar.FileInfoHeader(0x12c09f0, 0x24f8480, 0x0, 0x0, 0xd374c270, 0xe, 0x1ed0d08)
        /home/paralin/workspaces/m1_plus/host/lib/go/src/archive/tar/common.go:702 +0x384 fp=0x2058cb4 sp=0x2058be8 pc=0x3c60a0
github.com/docker/cli/vendor/github.com/docker/docker/pkg/archive.FileInfoHeader(0x22595b2, 0xa, 0x12c09f0, 0x24f8480, 0x0, 0x0, 0x4a6430, 0xf78538, 0x24d81e0)
        /home/paralin/workspaces/m1_plus/build/docker-cli-v18.06.1-ce/gopath/src/github.com/docker/cli/vendor/github.com/docker/docker/pkg/archive/archive.go:361 +0x34 fp=0x2058d14 sp=0x205
8cb4 pc=0x49ffa0                                                                                                                                                                                                    
github.com/docker/cli/vendor/github.com/docker/docker/pkg/archive.(*tarAppender).addTarFile(0x24adb40, 0x22595a0, 0x1c, 0x22595b2, 0xa, 0x0, 0x0)
        /home/paralin/workspaces/m1_plus/build/docker-cli-v18.06.1-ce/gopath/src/github.com/docker/cli/vendor/github.com/docker/docker/pkg/archive/archive.go:479 +0xa0 fp=0x2058d60 sp=0x205
8d14 pc=0x4a0720                                                                                                                                                                                                    
github.com/docker/cli/vendor/github.com/docker/docker/pkg/archive.TarWithOptions.func1.2(0x22595a0, 0x1c, 0x12c09f0, 0x24f83f0, 0x0, 0x0, 0xe, 0x1ed0d08)
        /home/paralin/workspaces/m1_plus/build/docker-cli-v18.06.1-ce/gopath/src/github.com/docker/cli/vendor/github.com/docker/docker/pkg/archive/archive.go:889 +0x484 fp=0x2058e64 sp=0x20
58d60 pc=0x4a64b8                                                                                                                                                                                                   
path/filepath.walk(0x22595a0, 0x1c, 0x12c09f0, 0x24f83f0, 0x23f2cc0, 0x0, 0x0)
        /home/paralin/workspaces/m1_plus/host/lib/go/src/path/filepath/path.go:358 +0x2f0 fp=0x2058ec8 sp=0x2058e64 pc=0x16ec28
path/filepath.walk(0x2259500, 0x13, 0x12c09f0, 0x24f8360, 0x23f2cc0, 0x0, 0x30)
        /home/paralin/workspaces/m1_plus/host/lib/go/src/path/filepath/path.go:382 +0x204 fp=0x2058f2c sp=0x2058ec8 pc=0x16eb3c
path/filepath.Walk(0x2259500, 0x13, 0x23f2cc0, 0x108490a, 0x1)
        /home/paralin/workspaces/m1_plus/host/lib/go/src/path/filepath/path.go:404 +0xe8 fp=0x2058f58 sp=0x2058f2c pc=0x16ed40
github.com/docker/cli/vendor/github.com/docker/docker/pkg/archive.TarWithOptions.func1(0x24d27d0, 0x12aedc8, 0x232e900, 0x2144f10, 0x2144f00, 0x232e8f0)
        /home/paralin/workspaces/m1_plus/build/docker-cli-v18.06.1-ce/gopath/src/github.com/docker/cli/vendor/github.com/docker/docker/pkg/archive/archive.go:807 +0x238 fp=0x2058fd4 sp=0x20
58f58 pc=0x4a69d0                                                                                                                                                                                                   
runtime.goexit()
        /home/paralin/workspaces/m1_plus/host/lib/go/src/runtime/asm_arm.s:867 +0x4 fp=0x2058fd4 sp=0x2058fd4 pc=0x670dc
created by github.com/docker/cli/vendor/github.com/docker/docker/pkg/archive.TarWithOptions
        /home/paralin/workspaces/m1_plus/build/docker-cli-v18.06.1-ce/gopath/src/github.com/docker/cli/vendor/github.com/docker/docker/pkg/archive/archive.go:749 +0x1e0

goroutine 1 [select]:
net/http.(*persistConn).roundTrip(0x22bb9a0, 0x24ada80, 0x0, 0x0, 0x0)
        /home/paralin/workspaces/m1_plus/host/lib/go/src/net/http/transport.go:2101 +0x400
net/http.(*Transport).roundTrip(0x24b6f20, 0x24d6880, 0x0, 0x0, 0x0)
        /home/paralin/workspaces/m1_plus/host/lib/go/src/net/http/transport.go:465 +0x688
net/http.(*Transport).RoundTrip(0x24b6f20, 0x24d6880, 0x24b6f20, 0x0, 0x0)
        /home/paralin/workspaces/m1_plus/host/lib/go/src/net/http/roundtrip.go:17 +0x24
net/http.send(0x24d6880, 0x12aada8, 0x24b6f20, 0x0, 0x0, 0x0, 0x0, 0x0, 0x21452a8, 0xa6d7af3f, ...)
        /home/paralin/workspaces/m1_plus/host/lib/go/src/net/http/client.go:250 +0x12c
net/http.(*Client).send(0x24ad400, 0x24d6880, 0x0, 0x0, 0x0, 0x0, 0x0, 0x2, 0x10, 0x2247b3f, ...)
        /home/paralin/workspaces/m1_plus/host/lib/go/src/net/http/client.go:174 +0xb8
net/http.(*Client).do(0x24ad400, 0x24d6880, 0x0, 0x0, 0x0)
        /home/paralin/workspaces/m1_plus/host/lib/go/src/net/http/client.go:641 +0x1ec
net/http.(*Client).Do(0x24ad400, 0x24d6880, 0xa, 0x2145268, 0x238bef0)
        /home/paralin/workspaces/m1_plus/host/lib/go/src/net/http/client.go:509 +0x24
github.com/docker/cli/vendor/golang.org/x/net/context/ctxhttp.Do(0x12bd2d0, 0x24ad720, 0x24ad400, 0x24d6800, 0x24b73f0, 0x0, 0xffffffff)
        /home/paralin/workspaces/m1_plus/build/docker-cli-v18.06.1-ce/gopath/src/github.com/docker/cli/vendor/golang.org/x/net/context/ctxhttp/ctxhttp.go:30 +0xa0
github.com/docker/cli/vendor/github.com/docker/docker/client.(*Client).doRequest(0x22471c0, 0x12bd2d0, 0x24ad720, 0x24d6800, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
        /home/paralin/workspaces/m1_plus/build/docker-cli-v18.06.1-ce/gopath/src/github.com/docker/cli/vendor/github.com/docker/docker/client/request.go:132 +0x60
github.com/docker/cli/vendor/github.com/docker/docker/client.(*Client).sendRequest(0x22471c0, 0x12bd2d0, 0x24ad720, 0x10858bf, 0x4, 0x1087985, 0x6, 0x24ad7e0, 0x12a98c0, 0x22478c0, ...)
        /home/paralin/workspaces/m1_plus/build/docker-cli-v18.06.1-ce/gopath/src/github.com/docker/cli/vendor/github.com/docker/docker/client/request.go:122 +0xc4
github.com/docker/cli/vendor/github.com/docker/docker/client.(*Client).postRaw(0x22471c0, 0x12bd2d0, 0x24ad720, 0x1087985, 0x6, 0x24ad7e0, 0x12a98c0, 0x22478c0, 0x255949c, 0x0, ...)
        /home/paralin/workspaces/m1_plus/build/docker-cli-v18.06.1-ce/gopath/src/github.com/docker/cli/vendor/github.com/docker/docker/client/request.go:50 +0x80
github.com/docker/cli/vendor/github.com/docker/docker/client.(*Client).ImageBuild(0x22471c0, 0x12bd2d0, 0x24ad720, 0x12a98c0, 0x22478c0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
        /home/paralin/workspaces/m1_plus/build/docker-cli-v18.06.1-ce/gopath/src/github.com/docker/cli/vendor/github.com/docker/docker/client/image_build.go:35 +0x168
github.com/docker/cli/cli/command/image.runBuild(0x12c6ad8, 0x207c980, 0xbe99df16, 0x1, 0x0, 0x0, 0x2399070, 0x113e8dc, 0x2399090, 0x113e9b4, ...)
        /home/paralin/workspaces/m1_plus/build/docker-cli-v18.06.1-ce/gopath/src/github.com/docker/cli/cli/command/image/build.go:399 +0x70c
github.com/docker/cli/cli/command/image.NewBuildCommand.func1(0x240b540, 0x2144df8, 0x1, 0x1, 0x0, 0x0)
        /home/paralin/workspaces/m1_plus/build/docker-cli-v18.06.1-ce/gopath/src/github.com/docker/cli/cli/command/image/build.go:110 +0x6c
github.com/docker/cli/vendor/github.com/spf13/cobra.(*Command).execute(0x240b540, 0x208e070, 0x1, 0x2, 0x240b540, 0x208e070)
        /home/paralin/workspaces/m1_plus/build/docker-cli-v18.06.1-ce/gopath/src/github.com/docker/cli/vendor/github.com/spf13/cobra/command.go:762 +0x364
github.com/docker/cli/vendor/github.com/spf13/cobra.(*Command).ExecuteC(0x20f2c80, 0x200ff08, 0xee1248, 0x200ff10)
        /home/paralin/workspaces/m1_plus/build/docker-cli-v18.06.1-ce/gopath/src/github.com/docker/cli/vendor/github.com/spf13/cobra/command.go:852 +0x210
github.com/docker/cli/vendor/github.com/spf13/cobra.(*Command).Execute(0x20f2c80, 0x20f2c80, 0x12aafe8)
        /home/paralin/workspaces/m1_plus/build/docker-cli-v18.06.1-ce/gopath/src/github.com/docker/cli/vendor/github.com/spf13/cobra/command.go:800 +0x1c
main.main()
        cmd/docker/docker.go:174 +0x98

goroutine 20 [syscall]:
os/signal.signal_recv(0x0)
        /home/paralin/workspaces/m1_plus/host/lib/go/src/runtime/sigqueue.go:139 +0x130
os/signal.loop()
        /home/paralin/workspaces/m1_plus/host/lib/go/src/os/signal/signal_unix.go:23 +0x14
created by os/signal.init.0
        /home/paralin/workspaces/m1_plus/host/lib/go/src/os/signal/signal_unix.go:29 +0x30

goroutine 23 [chan receive]:
github.com/docker/cli/vendor/github.com/golang/glog.(*loggingT).flushDaemon(0x1ed0f30)
        /home/paralin/workspaces/m1_plus/build/docker-cli-v18.06.1-ce/gopath/src/github.com/docker/cli/vendor/github.com/golang/glog/glog.go:879 +0x70
created by github.com/docker/cli/vendor/github.com/golang/glog.init.0
        /home/paralin/workspaces/m1_plus/build/docker-cli-v18.06.1-ce/gopath/src/github.com/docker/cli/vendor/github.com/golang/glog/glog.go:410 +0x1a0

goroutine 26 [IO wait]:
internal/poll.runtime_pollWait(0xa4ccffc0, 0x72, 0x9dfc8)
        /home/paralin/workspaces/m1_plus/host/lib/go/src/runtime/netpoll.go:173 +0x44
internal/poll.(*pollDesc).wait(0x24d26a4, 0x72, 0xffffff00, 0x12ac9b0, 0x1e5f15c)
        /home/paralin/workspaces/m1_plus/host/lib/go/src/internal/poll/fd_poll_runtime.go:85 +0x7c
internal/poll.(*pollDesc).waitRead(0x24d26a4, 0x24f2000, 0x1000, 0x1000)
        /home/paralin/workspaces/m1_plus/host/lib/go/src/internal/poll/fd_poll_runtime.go:90 +0x2c
internal/poll.(*FD).Read(0x24d2690, 0x24f2000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
        /home/paralin/workspaces/m1_plus/host/lib/go/src/internal/poll/fd_unix.go:169 +0x14c
net.(*netFD).Read(0x24d2690, 0x24f2000, 0x1000, 0x1000, 0x20000e0, 0x4, 0x148c0)
        /home/paralin/workspaces/m1_plus/host/lib/go/src/net/fd_unix.go:202 +0x38
net.(*conn).Read(0x2144e88, 0x24f2000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
        /home/paralin/workspaces/m1_plus/host/lib/go/src/net/net.go:177 +0x58
net/http.(*persistConn).Read(0x22bb9a0, 0x24f2000, 0x1000, 0x1000, 0x2af2a0, 0x2078000, 0x204d620)
        /home/paralin/workspaces/m1_plus/host/lib/go/src/net/http/transport.go:1497 +0x170
bufio.(*Reader).fill(0x23f2ab0)
        /home/paralin/workspaces/m1_plus/host/lib/go/src/bufio/bufio.go:100 +0x10c
bufio.(*Reader).Peek(0x23f2ab0, 0x1, 0x0, 0x0, 0x1, 0x2279680, 0x0)
        /home/paralin/workspaces/m1_plus/host/lib/go/src/bufio/bufio.go:132 +0x2c
net/http.(*persistConn).readLoop(0x22bb9a0)
        /home/paralin/workspaces/m1_plus/host/lib/go/src/net/http/transport.go:1645 +0x164
created by net/http.(*Transport).dialConn
        /home/paralin/workspaces/m1_plus/host/lib/go/src/net/http/transport.go:1338 +0x7f8

goroutine 27 [select]:
io.(*pipe).Read(0x23f2b40, 0x24c0000, 0x8000, 0x8000, 0x2026300, 0x19501, 0x1ee5001)
        /home/paralin/workspaces/m1_plus/host/lib/go/src/io/pipe.go:50 +0xac
io.(*PipeReader).Read(0x2144f08, 0x24c0000, 0x8000, 0x8000, 0x1ee5020, 0x24c0000, 0x0)
        /home/paralin/workspaces/m1_plus/host/lib/go/src/io/pipe.go:127 +0x38
github.com/docker/cli/vendor/github.com/docker/docker/pkg/progress.(*Reader).Read(0x22478c0, 0x24c0000, 0x8000, 0x8000, 0x24ddc8, 0x7c6d0, 0x8000)
        /home/paralin/workspaces/m1_plus/build/docker-cli-v18.06.1-ce/gopath/src/github.com/docker/cli/vendor/github.com/docker/docker/pkg/progress/progressreader.go:35 +0x40
net/http.transferBodyReader.Read(0x24d2910, 0x24c0000, 0x8000, 0x8000, 0x8000, 0x8000, 0xf20d01)
        /home/paralin/workspaces/m1_plus/host/lib/go/src/net/http/transfer.go:62 +0x40
io.copyBuffer(0xa4cdd250, 0x21452f0, 0x12ac890, 0x24d2910, 0x24c0000, 0x8000, 0x8000, 0xf20d10, 0xf4b780, 0x100, ...)
        /home/paralin/workspaces/m1_plus/host/lib/go/src/io/io.go:402 +0xd8
io.Copy(0xa4cdd250, 0x21452f0, 0x12ac890, 0x24d2910, 0x21452f0, 0x24f3000, 0x195, 0x1000)
        /home/paralin/workspaces/m1_plus/host/lib/go/src/io/io.go:364 +0x48
net/http.(*transferWriter).writeBody(0x24d2910, 0x12a9158, 0x24ad580, 0x2, 0x2)
        /home/paralin/workspaces/m1_plus/host/lib/go/src/net/http/transfer.go:355 +0xec
net/http.(*Request).write(0x24d6880, 0x12a9158, 0x24ad580, 0x0, 0x0, 0x0, 0x0, 0x0)
        /home/paralin/workspaces/m1_plus/host/lib/go/src/net/http/request.go:645 +0x4d8
net/http.(*persistConn).writeLoop(0x22bb9a0)
        /home/paralin/workspaces/m1_plus/host/lib/go/src/net/http/transport.go:1888 +0x134
created by net/http.(*Transport).dialConn
        /home/paralin/workspaces/m1_plus/host/lib/go/src/net/http/transport.go:1339 +0x814

trap    0x6
error   0x0
oldmask 0x0
r0      0x0
r1      0x20000000
r2      0x72
r3      0xa2e00471
r4      0x0
r5      0xa2e0086f
r6      0xfffffc01
r7      0xa2e0086f
r8      0xa2c00470
r9      0x7ffffffe
r10     0x2000d20
fp      0x3ff
ip      0xa43b3064
sp      0xa3ffeccc
lr      0xa439f8a8
pc      0x4f64e
cpsr    0x400f0030
fault   0xe4
#+END_SRC

[paralin]

• Reproduced on go 1.11.1

Here is the /proc/cpuinfo:

# cat /proc/cpuinfo 
processor       : 0
model name      : ARMv7 Processor rev 4 (v7l)
BogoMIPS        : 50.52
Features        : half thumb fastmult vfp edsp neon vfpv3 tls vfpv4 idiva idivt vfpd32 lpae evtstrm 
CPU implementer : 0x41
CPU architecture: 7
CPU variant     : 0x0
CPU part        : 0xc07
CPU revision    : 4

processor       : 1
model name      : ARMv7 Processor rev 4 (v7l)
BogoMIPS        : 50.52
Features        : half thumb fastmult vfp edsp neon vfpv3 tls vfpv4 idiva idivt vfpd32 lpae evtstrm 
CPU implementer : 0x41
CPU architecture: 7
CPU variant     : 0x0
CPU part        : 0xc07
CPU revision    : 4

Hardware        : Allwinner sun7i (A20) Family
Revision        : 0000
Serial          : 1651661409c28618

[paralin]

Cross compiler information (CC and CXX)

# ./arm-linux-gnueabihf-g++ -v
Using built-in specs.
COLLECT_GCC=/home/paralin/Documents/synrobo/skiff/workspaces/m1_plus/host/opt/ext-toolchain/bin/arm-linux-gnueabihf-g++
COLLECT_LTO_WRAPPER=/home/paralin/Documents/synrobo/skiff/workspaces/m1_plus/host/opt/ext-toolchain/bin/../libexec/gcc/arm-linux-gnueabihf/7.3.1/lto-wrapper
Target: arm-linux-gnueabihf
Configured with: '/home/tcwg-buildslave/workspace/tcwg-make-release/builder_arch/amd64/label/tcwg-x86_64-build/target/arm-linux-gnueabihf/snapshots/gcc.git~linaro-7.3-2018.05/configure' SHELL=/bin/bash --with-mpc=/home/tcwg-buildslave/workspace/tcwg-make-release/builder_arch/amd64/label/tcwg-x86_64-build/target/arm-linux-gnueabihf/_build/builds/destdir/x86_64-unknown-linux-gnu --with-mpfr=/home/tcwg-buildslave/workspace/tcwg-make-release/builder_arch/amd64/label/tcwg-x86_64-build/target/arm-linux-gnueabihf/_build/builds/destdir/x86_64-unknown-linux-gnu --with-gmp=/home/tcwg-buildslave/workspace/tcwg-make-release/builder_arch/amd64/label/tcwg-x86_64-build/target/arm-linux-gnueabihf/_build/builds/destdir/x86_64-unknown-linux-gnu --with-gnu-as --with-gnu-ld --disable-libmudflap --enable-lto --enable-shared --without-included-gettext --enable-nls --with-system-zlib --disable-sjlj-exceptions --enable-gnu-unique-object --enable-linker-build-id --disable-libstdcxx-pch --enable-c99 --enable-clocale=gnu --enable-libstdcxx-debug --enable-long-long --with-cloog=no --with-ppl=no --with-isl=no --disable-multilib --with-float=hard --with-fpu=vfpv3-d16 --with-mode=thumb --with-tune=cortex-a9 --with-arch=armv7-a --enable-threads=posix --enable-multiarch --enable-libstdcxx-time=yes --enable-gnu-indirect-function --with-build-sysroot=/home/tcwg-buildslave/workspace/tcwg-make-release/builder_arch/amd64/label/tcwg-x86_64-build/target/arm-linux-gnueabihf/_build/sysroots/arm-linux-gnueabihf --with-sysroot=/home/tcwg-buildslave/workspace/tcwg-make-release/builder_arch/amd64/label/tcwg-x86_64-build/target/arm-linux-gnueabihf/_build/builds/destdir/x86_64-unknown-linux-gnu/arm-linux-gnueabihf/libc --enable-checking=release --disable-bootstrap --enable-languages=c,c++,fortran,lto --build=x86_64-unknown-linux-gnu --host=x86_64-unknown-linux-gnu --target=arm-linux-gnueabihf --prefix=/home/tcwg-buildslave/workspace/tcwg-make-release/builder_arch/amd64/label/tcwg-x86_64-build/target/arm-linux-gnueabihf/_build/builds/destdir/x86_64-unknown-linux-gnu
Thread model: posix
gcc version 7.3.1 20180425 [linaro-7.3-2018.05 revision d29120a424ecfbc167ef90065c0eeb7f91977701] (Linaro GCC 7.3-2018.05) 

Using built-in specs.
COLLECT_GCC=/home/paralin/Documents/synrobo/skiff/workspaces/m1_plus/host/opt/ext-toolchain/bin/arm-linux-gnueabihf-gcc
COLLECT_LTO_WRAPPER=/home/paralin/Documents/synrobo/skiff/workspaces/m1_plus/host/opt/ext-toolchain/bin/../libexec/gcc/arm-linux-gnueabihf/7.3.1/lto-wrapper
Target: arm-linux-gnueabihf
Configured with: '/home/tcwg-buildslave/workspace/tcwg-make-release/builder_arch/amd64/label/tcwg-x86_64-build/target/arm-linux-gnueabihf/snapshots/gcc.git~linaro-7.3-2018.05/configure' SHELL=/bin/bash --with-mpc=/home/tcwg-buildslave/workspace/tcwg-make-release/builder_arch/amd64/label/tcwg-x86_64-build/target/arm-linux-gnueabihf/_build/builds/destdir/x86_64-unknown-linux-gnu --with-mpfr=/home/tcwg-buildslave/workspace/tcwg-make-release/builder_arch/amd64/label/tcwg-x86_64-build/target/arm-linux-gnueabihf/_build/builds/destdir/x86_64-unknown-linux-gnu --with-gmp=/home/tcwg-buildslave/workspace/tcwg-make-release/builder_arch/amd64/label/tcwg-x86_64-build/target/arm-linux-gnueabihf/_build/builds/destdir/x86_64-unknown-linux-gnu --with-gnu-as --with-gnu-ld --disable-libmudflap --enable-lto --enable-shared --without-included-gettext --enable-nls --with-system-zlib --disable-sjlj-exceptions --enable-gnu-unique-object --enable-linker-build-id --disable-libstdcxx-pch --enable-c99 --enable-clocale=gnu --enable-libstdcxx-debug --enable-long-long --with-cloog=no --with-ppl=no --with-isl=no --disable-multilib --with-float=hard --with-fpu=vfpv3-d16 --with-mode=thumb --with-tune=cortex-a9 --with-arch=armv7-a --enable-threads=posix --enable-multiarch --enable-libstdcxx-time=yes --enable-gnu-indirect-function --with-build-sysroot=/home/tcwg-buildslave/workspace/tcwg-make-release/builder_arch/amd64/label/tcwg-x86_64-build/target/arm-linux-gnueabihf/_build/sysroots/arm-linux-gnueabihf --with-sysroot=/home/tcwg-buildslave/workspace/tcwg-make-release/builder_arch/amd64/label/tcwg-x86_64-build/target/arm-linux-gnueabihf/_build/builds/destdir/x86_64-unknown-linux-gnu/arm-linux-gnueabihf/libc --enable-checking=release --disable-bootstrap --enable-languages=c,c++,fortran,lto --build=x86_64-unknown-linux-gnu --host=x86_64-unknown-linux-gnu --target=arm-linux-gnueabihf --prefix=/home/tcwg-buildslave/workspace/tcwg-make-release/builder_arch/amd64/label/tcwg-x86_64-build/target/arm-linux-gnueabihf/_build/builds/destdir/x86_64-unknown-linux-gnu
Thread model: posix
gcc version 7.3.1 20180425 [linaro-7.3-2018.05 revision d29120a424ecfbc167ef90065c0eeb7f91977701] (Linaro GCC 7.3-2018.05) 

[paralin]

Go is compiled with:

cd /home/paralin/Documents/synrobo/skiff/workspaces/m1_plus/build/host-go-1.10.5/src && GOROOT_BOOTSTRAP=/home/paralin/Documents/synrobo/skiff/workspaces/m1_plus/host/lib/go-1.4.3 GOROOT_FINAL=/home/paralin/Docume
nts/synrobo/skiff/workspaces/m1_plus/host/lib/go GOROOT="/home/paralin/Documents/synrobo/skiff/workspaces/m1_plus/build/host-go-1.10.5" GOBIN="/home/paralin/Documents/synrobo/skiff/workspaces/m1_plus/build/host-go
-1.10.5/bin" GOARCH=arm GOARM=7 GOOS=linux CC=/usr/bin/gcc CXX=/usr/bin/g++ GO_ASSUME_CROSSCOMPILING=1 CC_FOR_TARGET="/home/paralin/Documents/synrobo/skiff/workspaces/m1_plus/host/bin/arm-linux-gnueabihf-gcc" CXX_
FOR_TARGET="/home/paralin/Documents/synrobo/skiff/workspaces/m1_plus/host/bin/arm-linux-gnueabihf-g++" CGO_ENABLED=1 ./make.bash                                                                                    

[randall77]

Can you disassemble the binary and show us the instructions around PC=0x4f64e?
Possibly a dup of #28431 . Can you try tip and/or patch the associated CL in?

[paralin]

Reproduced against go 1.10.3 .... Really starting to look like this may not be a Go specific issue, but I don't know where else to start looking yet (maybe the toolchain?)

This build, I ensured that CC_FOR_TARGET=$(CC) and CXX_FOR_TARGET=$(CXX) when cross-compiling.

SIGILL: illegal instruction
PC=0x5152c m=3 sigcode=1

goroutine 0 [idle]:
runtime: unexpected return pc for runtime.sigInstallGoHandler called from 0xb459f89f
stack: frame={sp:0xb670cbc4, fp:0xb670cbd0} stack=[0xb5f0d0dc,0xb670ccdc)
b670cb44:  b4200518  0000000a  b44fd503  00000218 
b670cb54:  b2e00470  000001e6  b4200519  00000000 
b670cb64:  b2e005e2  00000000  00000001  00000000 
b670cb74:  b4200518  fbad2488  b2e00470  7ffffffe 
b670cb84:  b4200518  000003ff  b44f4ca5  00000001 
b670cb94:  00000000  0000001c  b2e00470  fbad2488 
b670cba4:  b44fb171  00000001  00000000  00000160 
b670cbb4:  00000400  b4200518  00000400  b4200917 
b670cbc4: <b459f89f  b45a1b10  00000001 >00000000 
b670cbd4:  00000400  00000400  12342020  b4596a48 
b670cbe4:  00000001  12342020  b4200518  00000400 
b670cbf4:  b670d7e4  00000000  01288d04  0000001c 
b670cc04:  b459fb1f  b670d7e4  00000000  b4200518 
b670cc14:  b2e00470  0000001c  b459fad5  12342020 
b670cc24:  00000000  00000400  b670d7c0  b4200518 
b670cc34:  00dd07c3  b670d7e4  00000081  00000048 
b670cc44:  00000400  01adabc8  00000000 
runtime.sigInstallGoHandler(0x400, 0x400)
        /home/paralin/Documents/synrobo/skiff/workspaces/m1_plus/host/lib/go/src/runtime/signal_unix.go:135 +0xa0

goroutine 27 [syscall]:
runtime.cgocall(0xdb17a0, 0x11e5a970, 0x3a0001)
        /home/paralin/Documents/synrobo/skiff/workspaces/m1_plus/host/lib/go/src/runtime/cgocall.go:128 +0x64 fp=0x11e5a94c sp=0x11e5a934 pc=0x120b4
os/user._Cfunc_mygetgrgid_r(0x0, 0x12342020, 0xb4200518, 0x400, 0x12264048, 0x0)
        _cgo_gotypes.go:104 +0x38 fp=0x11e5a96c sp=0x11e5a94c pc=0x3a66bc
os/user.lookupUnixGid.func1.1(0x0, 0x12342020, 0xb4200518, 0x400, 0x12264048, 0xb4200518)
        /home/paralin/Documents/synrobo/skiff/workspaces/m1_plus/host/lib/go/src/os/user/cgo_lookup_unix.go:181 +0x104 fp=0x11e5a990 sp=0x11e5a96c pc=0x3a81b4
os/user.lookupUnixGid.func1(0x8)
        /home/paralin/Documents/synrobo/skiff/workspaces/m1_plus/host/lib/go/src/os/user/cgo_lookup_unix.go:181 +0x40 fp=0x11e5a9ac sp=0x11e5a990 pc=0x3a8260
os/user.retryWithBuffer(0x12264050, 0x11e5aa28, 0x12264050, 0x1ac76f8)
        /home/paralin/Documents/synrobo/skiff/workspaces/m1_plus/host/lib/go/src/os/user/cgo_lookup_unix.go:253 +0x2c fp=0x11e5a9d8 sp=0x11e5a9ac pc=0x3a76a4
os/user.lookupUnixGid(0x0, 0x0, 0x0, 0x0)
        /home/paralin/Documents/synrobo/skiff/workspaces/m1_plus/host/lib/go/src/os/user/cgo_lookup_unix.go:177 +0xc4 fp=0x11e5aa3c sp=0x11e5a9d8 pc=0x3a720c
os/user.lookupGroupId(0x112e149, 0x1, 0xa, 0x112e149, 0x1)
        /home/paralin/Documents/synrobo/skiff/workspaces/m1_plus/host/lib/go/src/os/user/cgo_lookup_unix.go:167 +0x54 fp=0x11e5aa54 sp=0x11e5aa3c pc=0x3a7120
os/user.LookupGroupId(0x112e149, 0x1, 0x1, 0x0, 0x0)
        /home/paralin/Documents/synrobo/skiff/workspaces/m1_plus/host/lib/go/src/os/user/lookup.go:53 +0x24 fp=0x11e5aa6c sp=0x11e5aa54 pc=0x3a6444
archive/tar.statUnix(0x1255270, 0x12002240, 0x11ea00a0, 0xd374c270, 0xe)
        /home/paralin/Documents/synrobo/skiff/workspaces/m1_plus/host/lib/go/src/archive/tar/stat_unix.go:45 +0x5c4 fp=0x11e5aba8 sp=0x11e5aa6c pc=0x3b1564
archive/tar.FileInfoHeader(0x1255270, 0x12002240, 0x0, 0x0, 0xe, 0x1ac7bf8, 0xa831c)
        /home/paralin/Documents/synrobo/skiff/workspaces/m1_plus/host/lib/go/src/archive/tar/common.go:699 +0x388 fp=0x11e5ac84 sp=0x11e5aba8 pc=0x3aa980
github.com/docker/cli/vendor/github.com/docker/docker/pkg/archive.FileInfoHeader(0x1231a0f2, 0xa, 0x1255270, 0x12002240, 0x0, 0x0, 0x48f4a0, 0xf89818, 0x11f08480)
        /home/paralin/Documents/synrobo/skiff/workspaces/m1_plus/build/docker-cli-v18.06.1-ce/gopath/src/github.com/docker/cli/vendor/github.com/docker/docker/pkg/archive/archive.go:361 +0x34 fp=0x11e5ace4 sp=0x11
e5ac84 pc=0x488cec                                                                                                                                                                                                  
github.com/docker/cli/vendor/github.com/docker/docker/pkg/archive.(*tarAppender).addTarFile(0x12324580, 0x1231a0e0, 0x1c, 0x1231a0f2, 0xa, 0x0, 0x0)
        /home/paralin/Documents/synrobo/skiff/workspaces/m1_plus/build/docker-cli-v18.06.1-ce/gopath/src/github.com/docker/cli/vendor/github.com/docker/docker/pkg/archive/archive.go:479 +0x9c fp=0x11e5ad3c sp=0x11
e5ace4 pc=0x4894c8                                                                                                                                                                                                  
github.com/docker/cli/vendor/github.com/docker/docker/pkg/archive.TarWithOptions.func1.2(0x1231a0e0, 0x1c, 0x1255270, 0x120021b0, 0x0, 0x0, 0x1ac7bf8, 0xa831c)
        /home/paralin/Documents/synrobo/skiff/workspaces/m1_plus/build/docker-cli-v18.06.1-ce/gopath/src/github.com/docker/cli/vendor/github.com/docker/docker/pkg/archive/archive.go:889 +0x478 fp=0x11e5ae40 sp=0x1
1e5ad3c pc=0x48f528                                                                                                                                                                                                 
path/filepath.walk(0x1231a0e0, 0x1c, 0x1255270, 0x120021b0, 0x11e9c390, 0x0, 0x0)
        /home/paralin/Documents/synrobo/skiff/workspaces/m1_plus/host/lib/go/src/path/filepath/path.go:357 +0x2dc fp=0x11e5aea4 sp=0x11e5ae40 pc=0x161ca8
path/filepath.walk(0x1231a040, 0x13, 0x1255270, 0x12002120, 0x11e9c390, 0x0, 0x30)
        /home/paralin/Documents/synrobo/skiff/workspaces/m1_plus/host/lib/go/src/path/filepath/path.go:381 +0x1f8 fp=0x11e5af08 sp=0x11e5aea4 pc=0x161bc4
path/filepath.Walk(0x1231a040, 0x13, 0x11e9c390, 0x109d872, 0x1)
        /home/paralin/Documents/synrobo/skiff/workspaces/m1_plus/host/lib/go/src/path/filepath/path.go:403 +0xe0 fp=0x11e5af34 sp=0x11e5af08 pc=0x161db8
github.com/docker/cli/vendor/github.com/docker/docker/pkg/archive.TarWithOptions.func1(0x122bc7d0, 0x12435e8, 0x1217cde0, 0x122f00d8, 0x122f00a0, 0x1217cdd0)
        /home/paralin/Documents/synrobo/skiff/workspaces/m1_plus/build/docker-cli-v18.06.1-ce/gopath/src/github.com/docker/cli/vendor/github.com/docker/docker/pkg/archive/archive.go:807 +0x244 fp=0x11e5afd4 sp=0x1
1e5af34 pc=0x48fa50                                                                                                                                                                                                 
runtime.goexit()
        /home/paralin/Documents/synrobo/skiff/workspaces/m1_plus/host/lib/go/src/runtime/asm_arm.s:1015 +0x4 fp=0x11e5afd4 sp=0x11e5afd4 pc=0x66ed0
created by github.com/docker/cli/vendor/github.com/docker/docker/pkg/archive.TarWithOptions
        /home/paralin/Documents/synrobo/skiff/workspaces/m1_plus/build/docker-cli-v18.06.1-ce/gopath/src/github.com/docker/cli/vendor/github.com/docker/docker/pkg/archive/archive.go:749 +0x1e8

goroutine 1 [select]:
net/http.(*persistConn).roundTrip(0x120c5a40, 0x122fc1a0, 0x0, 0x0, 0x0)
        /home/paralin/Documents/synrobo/skiff/workspaces/m1_plus/host/lib/go/src/net/http/transport.go:2033 +0x464
net/http.(*Transport).RoundTrip(0x120c59a0, 0x122be300, 0x120c59a0, 0x0, 0x0)
        /home/paralin/Documents/synrobo/skiff/workspaces/m1_plus/host/lib/go/src/net/http/transport.go:422 +0x628
net/http.send(0x122be300, 0x123f5f8, 0x120c59a0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x122f0be8, 0xf8, ...)
        /home/paralin/Documents/synrobo/skiff/workspaces/m1_plus/host/lib/go/src/net/http/client.go:252 +0x130
net/http.(*Client).send(0x12119b40, 0x122be300, 0x0, 0x0, 0x0, 0x0, 0x0, 0x2, 0x1, 0x12115550, ...)
        /home/paralin/Documents/synrobo/skiff/workspaces/m1_plus/host/lib/go/src/net/http/client.go:176 +0xb8
net/http.(*Client).Do(0x12119b40, 0x122be300, 0x12119e40, 0x122be300, 0xf08d60)
        /home/paralin/Documents/synrobo/skiff/workspaces/m1_plus/host/lib/go/src/net/http/client.go:615 +0x1d0
github.com/docker/cli/vendor/golang.org/x/net/context/ctxhttp.Do(0x1251b50, 0x12119e40, 0x12119b40, 0x122be280, 0xffffffff, 0x277ba8, 0xfe7440)
        /home/paralin/Documents/synrobo/skiff/workspaces/m1_plus/build/docker-cli-v18.06.1-ce/gopath/src/github.com/docker/cli/vendor/golang.org/x/net/context/ctxhttp/ctxhttp.go:30 +0x50
github.com/docker/cli/vendor/github.com/docker/docker/client.(*Client).doRequest(0x120c06c0, 0x1251b50, 0x12119e40, 0x122be280, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
        /home/paralin/Documents/synrobo/skiff/workspaces/m1_plus/build/docker-cli-v18.06.1-ce/gopath/src/github.com/docker/cli/vendor/github.com/docker/docker/client/request.go:132 +0x5c
github.com/docker/cli/vendor/github.com/docker/docker/client.(*Client).sendRequest(0x120c06c0, 0x1251b50, 0x12119e40, 0x109e822, 0x4, 0x10a08f0, 0x6, 0x12119f00, 0x123e0b0, 0x120c0bc0, ...)
        /home/paralin/Documents/synrobo/skiff/workspaces/m1_plus/build/docker-cli-v18.06.1-ce/gopath/src/github.com/docker/cli/vendor/github.com/docker/docker/client/request.go:122 +0xc4
github.com/docker/cli/vendor/github.com/docker/docker/client.(*Client).postRaw(0x120c06c0, 0x1251b50, 0x12119e40, 0x10a08f0, 0x6, 0x12119f00, 0x123e0b0, 0x120c0bc0, 0x1222f478, 0x0, ...)
        /home/paralin/Documents/synrobo/skiff/workspaces/m1_plus/build/docker-cli-v18.06.1-ce/gopath/src/github.com/docker/cli/vendor/github.com/docker/docker/client/request.go:50 +0x80
github.com/docker/cli/vendor/github.com/docker/docker/client.(*Client).ImageBuild(0x120c06c0, 0x1251b50, 0x12119e40, 0x123e0b0, 0x120c0bc0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
        /home/paralin/Documents/synrobo/skiff/workspaces/m1_plus/build/docker-cli-v18.06.1-ce/gopath/src/github.com/docker/cli/vendor/github.com/docker/docker/client/image_build.go:35 +0x168
github.com/docker/cli/cli/command/image.runBuild(0x125b2d8, 0x12045dc0, 0xbefaaf16, 0x1, 0x0, 0x0, 0x11ef17d0, 0x11570a8, 0x11ef17f0, 0x1157180, ...)
        /home/paralin/Documents/synrobo/skiff/workspaces/m1_plus/build/docker-cli-v18.06.1-ce/gopath/src/github.com/docker/cli/cli/command/image/build.go:399 +0x6d4
github.com/docker/cli/cli/command/image.NewBuildCommand.func1(0x11ea5400, 0x122f08e0, 0x1, 0x1, 0x0, 0x0)
        /home/paralin/Documents/synrobo/skiff/workspaces/m1_plus/build/docker-cli-v18.06.1-ce/gopath/src/github.com/docker/cli/cli/command/image/build.go:110 +0x6c
github.com/docker/cli/vendor/github.com/spf13/cobra.(*Command).execute(0x11ea5400, 0x11e360b0, 0x1, 0x2, 0x11ea5400, 0x11e360b0)
        /home/paralin/Documents/synrobo/skiff/workspaces/m1_plus/build/docker-cli-v18.06.1-ce/gopath/src/github.com/docker/cli/vendor/github.com/spf13/cobra/command.go:762 +0x360
github.com/docker/cli/vendor/github.com/spf13/cobra.(*Command).ExecuteC(0x120c7040, 0x122c19b8, 0xef3288, 0x122c19c0)
        /home/paralin/Documents/synrobo/skiff/workspaces/m1_plus/build/docker-cli-v18.06.1-ce/gopath/src/github.com/docker/cli/vendor/github.com/spf13/cobra/command.go:852 +0x20c
github.com/docker/cli/vendor/github.com/spf13/cobra.(*Command).Execute(0x120c7040, 0x120c7040, 0x123f820)
        /home/paralin/Documents/synrobo/skiff/workspaces/m1_plus/build/docker-cli-v18.06.1-ce/gopath/src/github.com/docker/cli/vendor/github.com/spf13/cobra/command.go:800 +0x1c
main.main()
        docker-cli-v18.06.1-ce/gopath/src/github.com/docker/cli/cmd/docker/docker.go:174 +0x98

goroutine 4 [syscall]:
os/signal.signal_recv(0x0)
        /home/paralin/Documents/synrobo/skiff/workspaces/m1_plus/host/lib/go/src/runtime/sigqueue.go:139 +0x130
os/signal.loop()
        /home/paralin/Documents/synrobo/skiff/workspaces/m1_plus/host/lib/go/src/os/signal/signal_unix.go:22 +0x14
created by os/signal.init.0
        /home/paralin/Documents/synrobo/skiff/workspaces/m1_plus/host/lib/go/src/os/signal/signal_unix.go:28 +0x30

goroutine 22 [chan receive]:
github.com/docker/cli/vendor/github.com/golang/glog.(*loggingT).flushDaemon(0x1ac7e18)
        /home/paralin/Documents/synrobo/skiff/workspaces/m1_plus/build/docker-cli-v18.06.1-ce/gopath/src/github.com/docker/cli/vendor/github.com/golang/glog/glog.go:879 +0x70
created by github.com/docker/cli/vendor/github.com/golang/glog.init.0
        /home/paralin/Documents/synrobo/skiff/workspaces/m1_plus/build/docker-cli-v18.06.1-ce/gopath/src/github.com/docker/cli/vendor/github.com/golang/glog/glog.go:410 +0x1a0

goroutine 25 [IO wait]:
internal/poll.runtime_pollWait(0xb4ecffc0, 0x72, 0x122ea000)
        /home/paralin/Documents/synrobo/skiff/workspaces/m1_plus/host/lib/go/src/runtime/netpoll.go:173 +0x44
internal/poll.(*pollDesc).wait(0x122bc6a4, 0x72, 0xffffff00, 0x1241200, 0x1a56270)
        /home/paralin/Documents/synrobo/skiff/workspaces/m1_plus/host/lib/go/src/internal/poll/fd_poll_runtime.go:85 +0x8c
internal/poll.(*pollDesc).waitRead(0x122bc6a4, 0x122ea000, 0x1000, 0x1000)
        /home/paralin/Documents/synrobo/skiff/workspaces/m1_plus/host/lib/go/src/internal/poll/fd_poll_runtime.go:90 +0x2c
internal/poll.(*FD).Read(0x122bc690, 0x122ea000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
        /home/paralin/Documents/synrobo/skiff/workspaces/m1_plus/host/lib/go/src/internal/poll/fd_unix.go:157 +0x148
net.(*netFD).Read(0x122bc690, 0x122ea000, 0x1000, 0x1000, 0x3e328, 0x14548, 0x11e5bcdc)
        /home/paralin/Documents/synrobo/skiff/workspaces/m1_plus/host/lib/go/src/net/fd_unix.go:202 +0x38
net.(*conn).Read(0x122f0008, 0x122ea000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
        /home/paralin/Documents/synrobo/skiff/workspaces/m1_plus/host/lib/go/src/net/net.go:176 +0x58
net/http.(*persistConn).Read(0x120c5a40, 0x122ea000, 0x1000, 0x1000, 0x11ffd640, 0x11f960b0, 0x62390)
        /home/paralin/Documents/synrobo/skiff/workspaces/m1_plus/host/lib/go/src/net/http/transport.go:1453 +0x174
bufio.(*Reader).fill(0x121ec000)
        /home/paralin/Documents/synrobo/skiff/workspaces/m1_plus/host/lib/go/src/bufio/bufio.go:100 +0x104
bufio.(*Reader).Peek(0x121ec000, 0x1, 0x0, 0x0, 0x0, 0x11f96000, 0x0)
        /home/paralin/Documents/synrobo/skiff/workspaces/m1_plus/host/lib/go/src/bufio/bufio.go:132 +0x28
net/http.(*persistConn).readLoop(0x120c5a40)
        /home/paralin/Documents/synrobo/skiff/workspaces/m1_plus/host/lib/go/src/net/http/transport.go:1601 +0x14c
created by net/http.(*Transport).dialConn
        /home/paralin/Documents/synrobo/skiff/workspaces/m1_plus/host/lib/go/src/net/http/transport.go:1237 +0x7b0

goroutine 26 [select]:
io.(*pipe).Read(0x121ec120, 0x12326000, 0x8000, 0x8000, 0x0, 0x1adbf00, 0x1)
        /home/paralin/Documents/synrobo/skiff/workspaces/m1_plus/host/lib/go/src/io/pipe.go:50 +0xe0
io.(*PipeReader).Read(0x122f00b0, 0x12326000, 0x8000, 0x8000, 0x8000, 0xe31078, 0x12326000)
        /home/paralin/Documents/synrobo/skiff/workspaces/m1_plus/host/lib/go/src/io/pipe.go:127 +0x38
github.com/docker/cli/vendor/github.com/docker/docker/pkg/progress.(*Reader).Read(0x120c0bc0, 0x12326000, 0x8000, 0x8000, 0x1dce0, 0x7c024, 0x8000)
        /home/paralin/Documents/synrobo/skiff/workspaces/m1_plus/build/docker-cli-v18.06.1-ce/gopath/src/github.com/docker/cli/vendor/github.com/docker/docker/pkg/progress/progressreader.go:35 +0x40
net/http.transferBodyReader.Read(0x122bc910, 0x12326000, 0x8000, 0x8000, 0x8000, 0x8000, 0x1adbf70)
        /home/paralin/Documents/synrobo/skiff/workspaces/m1_plus/host/lib/go/src/net/http/transfer.go:60 +0x40
io.copyBuffer(0xb4693400, 0x122f0c30, 0x12410e0, 0x122bc910, 0x12326000, 0x8000, 0x8000, 0xf326d0, 0xf5d0a8, 0x1ed00, ...)
        /home/paralin/Documents/synrobo/skiff/workspaces/m1_plus/host/lib/go/src/io/io.go:400 +0x14c
io.Copy(0xb4693400, 0x122f0c30, 0x12410e0, 0x122bc910, 0x122f0c30, 0x120c0e54, 0x28abf8, 0x8)
        /home/paralin/Documents/synrobo/skiff/workspaces/m1_plus/host/lib/go/src/io/io.go:362 +0x48
net/http.(*transferWriter).WriteBody(0x122bc910, 0x123d948, 0x12119cc0, 0x2, 0x2)
        /home/paralin/Documents/synrobo/skiff/workspaces/m1_plus/host/lib/go/src/net/http/transfer.go:330 +0xf0
net/http.(*Request).write(0x122be300, 0x123d948, 0x12119cc0, 0x0, 0x0, 0x0, 0x0, 0x0)
        /home/paralin/Documents/synrobo/skiff/workspaces/m1_plus/host/lib/go/src/net/http/request.go:622 +0x4a8
net/http.(*persistConn).writeLoop(0x120c5a40)
        /home/paralin/Documents/synrobo/skiff/workspaces/m1_plus/host/lib/go/src/net/http/transport.go:1825 +0x168
created by net/http.(*Transport).dialConn
        /home/paralin/Documents/synrobo/skiff/workspaces/m1_plus/host/lib/go/src/net/http/transport.go:1238 +0x7cc

trap    0x6
error   0x0
oldmask 0x0
r0      0x0
r1      0x0
r2      0x72
r3      0xb4200519
r4      0x0
r5      0xb4200917
r6      0xfffffc01
r7      0x3fe
r8      0xb2e00470
r9      0x7ffffffe
r10     0x11e007e0
fp      0x3ff
ip      0xb45b3064
sp      0xb670cbc4
lr      0xb459f8a8
pc      0x5152c
cpsr    0x200f0030
fault   0xe4

The disassembly around 5152c:

   5150a:	e58d      	b.n	51028 <_start@@Base+0x3ec88>
   5150c:	3004      	adds	r0, #4
   5150e:	e58d      	b.n	5102c <_start@@Base+0x3ec8c>
   51510:	c008      	stmia	r0!, {r3}
   51512:	e58d      	b.n	51030 <_start@@Base+0x3ec90>
   51514:	48e3      	ldr	r0, [pc, #908]	; (518a4 <_start@@Base+0x3f504>)
   51516:	ebff 00b0 			; <UNDEFINED> instruction: 0xebff00b0
   5151a:	e59d      	b.n	51058 <_start@@Base+0x3ecb8>
   5151c:	0004      	movs	r4, r0
   5151e:	e58d      	b.n	5103c <_start@@Base+0x3ec9c>
   51520:	0064      	lsls	r4, r4, #1
   51522:	e59d      	b.n	51060 <_start@@Base+0x3ecc0>
   51524:	0008      	movs	r0, r1
   51526:	e58d      	b.n	51044 <_start@@Base+0x3eca4>
   51528:	48de      	ldr	r0, [pc, #888]	; (518a4 <_start@@Base+0x3f504>)
   5152a:	ebff 00b8 			; <UNDEFINED> instruction: 0xebff00b8
   5152e:	e59d      	b.n	5106c <_start@@Base+0x3eccc>
   51530:	0004      	movs	r4, r0
   51532:	e58d      	b.n	51050 <_start@@Base+0x3ecb0>
   51534:	0000      	movs	r0, r0
   51536:	e3a0      	b.n	51c7a <_start@@Base+0x3f8da>
   51538:	0008      	movs	r0, r1
   5153a:	e58d      	b.n	51058 <_start@@Base+0x3ecb8>
   5153c:	48d9      	ldr	r0, [pc, #868]	; (518a4 <_start@@Base+0x3f504>)
   5153e:	ebff 00b4 			; <UNDEFINED> instruction: 0xebff00b4
   51542:	e59d      	b.n	51080 <_start@@Base+0x3ece0>
   51544:	0004      	movs	r4, r0
   51546:	e58d      	b.n	51064 <_start@@Base+0x3ecc4>
   51548:	0000      	movs	r0, r0
   5154a:	e3a0      	b.n	51c8e <_start@@Base+0x3f8ee>
   5154c:	0008      	movs	r0, r1
   5154e:	e58d      	b.n	5106c <_start@@Base+0x3eccc>
   51550:	48d4      	ldr	r0, [pc, #848]	; (518a4 <_start@@Base+0x3f504>)
   51552:	ebff 0000 			; <UNDEFINED> instruction: 0xebff0000
   51556:	e3a0      	b.n	51c9a <_start@@Base+0x3f8fa>
   51558:	1040      	asrs	r0, r0, #1
   5155a:	e59d      	b.n	51098 <_start@@Base+0x3ecf8>
   5155c:	20a8      	movs	r0, #168	; 0xa8
   5155e:	e59d      	b.n	5109c <_start@@Base+0x3ecfc>
   51560:	5068      	str	r0, [r5, r1]
   51562:	e59d      	b.n	510a0 <_start@@Base+0x3ed00>

Done using the toolchain objdump.

I see UNDEFINED instruction here a lot, are these causing the SIGILL?

Will try with the CL patched in.

@randall77

[randall77]

Looks like you disassembled in thumb mode. We generate only standard arm code. Could you figure out how to disassemble in standard mode?
Or, this could be the root of the issue. Maybe your C-compiled code is thumb and Go is standard? That would probably break everything.

[paralin]

@randall77 In Buildroot we compile with thumb2 enabled on ARM. Would this break everything if the thumb mode was disabled when Go was calling the CC, or also if anything in the shared libraries used was in thumb2?

[paralin]

@randall77 That CL doesn't apply to this scenario, it's for arm64 (this is arm) and for android (this is GOOS=linux)

[randall77]

@randall77 In Buildroot we compile with thumb2 enabled on ARM. Would this break everything if the thumb mode was disabled when Go was calling the CC, or also if anything in the shared libraries used was in thumb2?

I don't know much about arm <-> thumb transitions, just a suspicion based on the fact that the disassembler chose thumb.
It would be a good experiment at least to disable thumb everywhere you can, and see if that helps.

@randall77 That CL doesn't apply to this scenario, it's for arm64 (this is arm) and for android (this is GOOS=linux)

Ok. It's possibly a related issue, as we do use some feature detection to enable instructions on arm as well. I think we do it for the hardware divide instruction, at least. But that seems an unlikely culprit at the moment.

[paralin]

Crash at PC=0x127d4 (cgocall.go:128) go 1.11.2

   127b4:	e58010bc 	str	r1, [r0, #188]	; 0xbc
   127b8:	e59010c0 	ldr	r1, [r0, #192]	; 0xc0
   127bc:	e2811001 	add	r1, r1, #1
   127c0:	e58010c0 	str	r1, [r0, #192]	; 0xc0
   127c4:	e59010c8 	ldr	r1, [r0, #200]	; 0xc8
   127c8:	e3a02000 	mov	r2, #0
   127cc:	e5812000 	str	r2, [r1]
   127d0:	eb00c4b5 	bl	0x43aac
   127d4:	e3a00001 	mov	r0, #1
   127d8:	e59d1014 	ldr	r1, [sp, #20]
   127dc:	e5c100a5 	strb	r0, [r1, #165]	; 0xa5
   127e0:	e59d001c 	ldr	r0, [sp, #28]
   127e4:	e58d0004 	str	r0, [sp, #4]
   127e8:	e59d2020 	ldr	r2, [sp, #32]
   127ec:	e58d2008 	str	r2, [sp, #8]
   127f0:	eb015196 	bl	0x66e50
   127f4:	e59d000c 	ldr	r0, [sp, #12]
   127f8:	e58d0010 	str	r0, [sp, #16]

Maybe bl 0x43aac ?

[paralin]

_cgo_gotypes.go:104 +0x38 fp=0x20589c4 sp=0x20589ac pc=0x3c1fd8

  3c1fb8:	e28d001c 	add	r0, sp, #28
  3c1fbc:	e58d0014 	str	r0, [sp, #20]
  3c1fc0:	e59fb0c4 	ldr	fp, [pc, #196]	; 0x3c208c
  3c1fc4:	e59b0000 	ldr	r0, [fp]
  3c1fc8:	e58d0004 	str	r0, [sp, #4]
  3c1fcc:	e59d0014 	ldr	r0, [sp, #20]
  3c1fd0:	e58d0008 	str	r0, [sp, #8]
  3c1fd4:	ebf141e7 	bl	0x12778
  3c1fd8:	e59fb0b0 	ldr	fp, [pc, #176]	; 0x3c2090
  3c1fdc:	e5db0000 	ldrb	r0, [fp]
  3c1fe0:	e3500000 	cmp	r0, #0
  3c1fe4:	1a000000 	bne	0x3c1fec
  3c1fe8:	e49df018 	ldr	pc, [sp], #24
  3c1fec:	e59d001c 	ldr	r0, [sp, #28]
  3c1ff0:	e59f109c 	ldr	r1, [pc, #156]	; 0x3c2094
  3c1ff4:	e58d1004 	str	r1, [sp, #4]
  3c1ff8:	e58d0008 	str	r0, [sp, #8]
  3c1ffc:	ebf159d4 	bl	0x18754

3c1fd8: e59fb0b0 ldr fp, [pc, #176] ; 0x3c2090

[paralin]

Okay, disabling static compilation of the binary fixes the issue. I'm going to go ahead and close this, it's probably related to mixing thumb2 and standard instructions with static compilation. Going to give up on having a static docker client for now.