cmd/compile: SIGILL: illegal instruction on linux/arm

[petemoore]

What version of Go are you using (go version)?

$ go version
go version go1.13.7 linux/arm

Does this issue reproduce with the latest release?

I suspect it is intermittent, since it only occurred once, so I haven't tested on go 1.14. However this was in go 1.13.7. It occurred in our CI for a particular commit, so it may be reproducible with just this commit, or perhaps it is just an intermittent failure. It only occurred on one platform, although we build for several.

What operating system and processor architecture are you using (go env)?

go env Output

$ go env
GO111MODULE=""
GOARCH="arm"
GOBIN=""
GOCACHE="/home/genericworker/.cache/go-build"
GOENV="/home/genericworker/.config/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="arm"
GOHOSTOS="linux"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/home/genericworker/tasks/task_1582823294/gopath1.13.7"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/home/genericworker/tasks/task_1582823294/go1.13.7/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/home/genericworker/tasks/task_1582823294/go1.13.7/go/pkg/tool/linux_arm"
GCCGO="gccgo"
GOARM="6"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="0"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -marm -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build077222367=/tmp/go-build -gno-record-gcc-switches"

What did you do?

I ran this task.

It performed the following steps:

function b64 {
  [ "$(uname -s)" != "Darwin" ] || base64 -D
  [ "$(uname -s)" != "Linux" ]  || base64 -d
}
chmod a+x bin/taskcluster-proxy
# go test: -race and -msan are only supported on linux/amd64, freebsd/amd64, darwin/amd64 and windows/amd64
if [ "$(uname -m)" == "x86_64" ]; then
  RACE=-race
  CGO_ENABLED_TESTS=1
  # See https://github.com/golang/go/issues/27089#issuecomment-415329050
  VET=-vet=off
else
  RACE=
  CGO_ENABLED_TESTS=0
  VET=
fi
export CGO_ENABLED=0
export GOROOT="$(pwd)/go1.13.7/go"
export GOPATH="$(pwd)/gopath1.13.7"
export PATH="${GOPATH}/bin:${GOROOT}/bin:$(pwd)/bin:${PATH}"
git --version
go version
go env
curl -s "${TASKCLUSTER_PROXY_URL}/secrets/v1/secret/project/taskcluster/testing/generic-worker/ci-creds" | sed -n 's/.*"b64_encoded_credentials_script": "\(.*\)".*/\1/p' | b64 > tc-creds.sh
source tc-creds.sh
if [ ! -d taskcluster/.git ]; then
  rm -rf taskcluster
  git clone "${GITHUB_CLONE_URL}" taskcluster
fi
cd taskcluster
git fetch "${GITHUB_CLONE_URL}" "+${GITHUB_SHA}:refs/heads/X${TASK_ID}"
git checkout -f "X${TASK_ID}"
git reset --hard "${GITHUB_SHA}"
git clean -fdx
git checkout -B tmp -t "X${TASK_ID}"
go get -v -u github.com/taskcluster/livelog
cd workers/generic-worker
# go.mod and go.sum will be affected by above go get commands, so
# tidy them before checking for changes. Not needed in go 1.14:
# https://github.com/golang/go/issues/30515#issuecomment-581984371
# TODO: use go get -modfile instead when running go 1.14, and remove
# go mod tidy command
go mod tidy
git status
# output of wc command can contain spaces on darwin, so no quotes around expression
test $(git status --porcelain | wc -l) == 0
go install -tags "${ENGINE}" -v -ldflags "-X main.revision=${GITHUB_SHA}" ./...
go vet -tags "${ENGINE}" ./...
if [ "${ENGINE}" == "multiuser" ]; then
  cp "${TASK_USER_CREDENTIALS}" next-task-user.json
  # IMPORTANT - run go test with GW_TESTS_RUN_AS_CURRENT_USER=true *before* running it without
  # otherwise tests that call `go run ....` will write go object files to .cache as root
  GW_TESTS_RUN_AS_CURRENT_USER=true GORACE=history_size=7 CGO_ENABLED=1 go test -tags "${ENGINE}" -timeout 45m -ldflags "-X github.com/taskcluster/taskcluster/v25/workers/generic-worker.revision=${GITHUB_SHA}" -v ${RACE} ${VET}
fi
GORACE=history_size=7 CGO_ENABLED=${CGO_ENABLED_TESTS} go test -tags "${ENGINE}" -timeout 45m -ldflags "-X github.com/taskcluster/taskcluster/v25/workers/generic-worker.revision=${GITHUB_SHA}" -v ${RACE} ${VET} ./...
../../../golangci-lint/golangci-lint-1.23.6-*/golangci-lint run --build-tags "${ENGINE}"

Please note the env variables referenced etc are included in the linked task definition, if required.

What did you expect to see?

Successful compilation. Note, the build was from this commit of our repository, which includes some misformed build constraints (such as a trailing comma).

What did you see instead?

The task produced these logs:

+ go install -tags simple -v -ldflags '-X main.revision=5230896c11537d0ada1df54c1fb8174f49740d01' ./...
go: finding github.com/ghodss/yaml v1.0.0
go: finding github.com/taskcluster/shell v0.0.0-20191115171910-c688067f12d3
go: finding github.com/kr/text v0.1.0
go: finding github.com/gorilla/websocket v1.4.1
go: finding github.com/taskcluster/stateless-dns-go v1.0.6
go: finding github.com/peterbourgon/mergemap v0.0.0-20130613134717-e21c03b7a721
go: finding github.com/taskcluster/slugid-go v1.1.0
go: finding github.com/taskcluster/websocktunnel v2.0.0+incompatible
go: finding golang.org/x/tools v0.0.0-20200219161401-5fb17a1e7b9b
go: finding github.com/pborman/uuid v1.2.0
go: finding github.com/go-yaml/yaml v0.0.0-20181115110504-51d6538a90f8
go: finding github.com/docopt/docopt-go v0.0.0-20180111231733-ee0de3bc6815
go: finding github.com/elastic/go-sysinfo v1.3.0
go: finding github.com/aws/aws-sdk-go v1.28.9
go: finding github.com/getsentry/raven-go v0.2.0
go: finding github.com/fatih/camelcase v1.0.0
go: finding github.com/mholt/archiver v2.1.0+incompatible
go: finding github.com/dchest/uniuri v0.0.0-20160212164326-8902c56451e9
go: finding github.com/dgrijalva/jwt-go v3.2.0+incompatible
go: finding github.com/taskcluster/httpbackoff/v3 v3.0.0
go: finding github.com/pkg/errors v0.8.1
go: finding howett.net/plist v0.0.0-20181124034731-591f970eefbb
go: finding github.com/taskcluster/taskcluster-lib-urls v13.0.0+incompatible
go: finding github.com/cenkalti/backoff/v3 v3.0.0
go: finding github.com/jmespath/go-jmespath v0.0.0-20180206201540-c2b33e8439af
go: finding github.com/joeshaw/multierror v0.0.0-20140124173710-69b34d4ec901
go: finding github.com/google/uuid v1.0.0
go: finding github.com/dsnet/compress v0.0.1
go: finding github.com/golang/snappy v0.0.1
go: finding github.com/tent/hawk-go v0.0.0-20161026210932-d341ea318957
go: finding github.com/prometheus/procfs v0.0.0-20190425082905-87a4384529e0
go: finding github.com/nwaples/rardecode v1.0.0
go: finding github.com/pierrec/lz4 v2.4.1+incompatible
go: finding github.com/certifi/gocertifi v0.0.0-20200104152315-a6d78f326758
go: finding github.com/ulikunitz/xz v0.5.6
go: finding golang.org/x/mod v0.1.1-0.20191105210325-c90efee705ee
go: finding golang.org/x/xerrors v0.0.0-20191011141410-1b5146add898
go: finding github.com/xeipuuv/gojsonschema v1.1.0
go: finding golang.org/x/crypto v0.0.0-20191011191535-87dc89f01550
go: finding github.com/xeipuuv/gojsonreference v0.0.0-20180127040603-bd5ef7bd5415
go: finding github.com/xeipuuv/gojsonpointer v0.0.0-20190905194746-02993c407bfb
github.com/elastic/go-sysinfo/providers/windows
github.com/dsnet/compress/bzip2/internal/sais
github.com/dsnet/compress/internal
github.com/ulikunitz/xz/internal/hash
github.com/fatih/camelcase
golang.org/x/mod/semver
golang.org/x/xerrors/internal
github.com/dsnet/compress/internal/errors
github.com/pierrec/lz4/internal/xxh32
github.com/golang/snappy
github.com/docopt/docopt-go
github.com/pkg/errors
howett.net/plist
github.com/elastic/go-sysinfo/types
github.com/elastic/go-sysinfo/providers/shared
github.com/elastic/go-sysinfo/internal/registry
github.com/joeshaw/multierror
github.com/prometheus/procfs
github.com/certifi/gocertifi
github.com/dsnet/compress
github.com/getsentry/raven-go
github.com/dsnet/compress/internal/prefix
github.com/elastic/go-sysinfo/providers/linux
github.com/elastic/go-sysinfo/providers/darwin
github.com/dsnet/compress/bzip2
github.com/nwaples/rardecode
github.com/pierrec/lz4
github.com/ulikunitz/xz/internal/xlog
github.com/elastic/go-sysinfo
github.com/cenkalti/backoff/v3
github.com/ulikunitz/xz/lzma
github.com/taskcluster/shell
github.com/google/uuid
github.com/taskcluster/httpbackoff/v3
github.com/taskcluster/taskcluster-lib-urls
github.com/pborman/uuid
github.com/taskcluster/taskcluster/v25/tools/jsonschema2go/text
github.com/tent/hawk-go
github.com/taskcluster/slugid-go/slugid
github.com/taskcluster/taskcluster/v25/tools/taskcluster-worker-runner/protocol
github.com/gorilla/websocket
github.com/taskcluster/stateless-dns-go/hostname
github.com/taskcluster/taskcluster/v25/clients/client-go
github.com/dgrijalva/jwt-go
github.com/ulikunitz/xz
github.com/taskcluster/taskcluster/v25/clients/client-go/tcpurgecache
github.com/taskcluster/taskcluster/v25/clients/client-go/tcqueue
github.com/taskcluster/taskcluster/v25/clients/client-go/tcsecrets
github.com/mholt/archiver
github.com/taskcluster/taskcluster/v25/clients/client-go/tcworkermanager
github.com/taskcluster/taskcluster/v25/clients/client-go/tcauth
github.com/taskcluster/websocktunnel/util
github.com/taskcluster/taskcluster/v25/workers/generic-worker/host
github.com/taskcluster/taskcluster/v25/workers/generic-worker/fileutil
github.com/taskcluster/websocktunnel/wsmux
github.com/peterbourgon/mergemap
github.com/taskcluster/taskcluster/v25/workers/generic-worker/livelog
github.com/taskcluster/taskcluster/v25/workers/generic-worker/process
github.com/taskcluster/taskcluster/v25/internal/scopes
github.com/taskcluster/websocktunnel/client
github.com/taskcluster/taskcluster/v25/workers/generic-worker/gwconfig
github.com/dchest/uniuri
github.com/taskcluster/taskcluster/v25/workers/generic-worker/gdm3
github.com/taskcluster/taskcluster/v25/workers/generic-worker/tcproxy
github.com/taskcluster/taskcluster/v25/workers/generic-worker/runtime
github.com/taskcluster/taskcluster/v25/workers/generic-worker/expose
github.com/xeipuuv/gojsonpointer
golang.org/x/crypto/ed25519
github.com/kr/text
gopkg.in/yaml.v2
github.com/taskcluster/taskcluster/v25/internal/jsontest
golang.org/x/xerrors
github.com/xeipuuv/gojsonreference
github.com/xeipuuv/gojsonschema
golang.org/x/tools/go/ast/astutil
golang.org/x/mod/module
golang.org/x/tools/internal/fastwalk
golang.org/x/tools/internal/gopathwalk
github.com/aws/aws-sdk-go/aws/awserr
golang.org/x/tools/internal/imports
github.com/aws/aws-sdk-go/internal/ini
github.com/aws/aws-sdk-go/internal/shareddefaults
github.com/aws/aws-sdk-go/aws/credentials
github.com/aws/aws-sdk-go/aws/endpoints
github.com/ghodss/yaml
github.com/taskcluster/taskcluster/v25/tools/jsonschema2go
github.com/taskcluster/taskcluster/v25/workers/generic-worker
golang.org/x/tools/imports
github.com/taskcluster/taskcluster/v25/workers/generic-worker/gw-decision-task
github.com/taskcluster/taskcluster/v25/workers/generic-worker/gw-codegen
github.com/aws/aws-sdk-go/internal/sdkio
github.com/aws/aws-sdk-go/aws/client/metadata
github.com/aws/aws-sdk-go/aws
github.com/jmespath/go-jmespath
github.com/aws/aws-sdk-go/aws/awsutil
github.com/aws/aws-sdk-go/internal/sdkrand
github.com/aws/aws-sdk-go/internal/strings
github.com/aws/aws-sdk-go/aws/credentials/processcreds
github.com/aws/aws-sdk-go/internal/sdkmath
github.com/aws/aws-sdk-go/aws/request
github.com/aws/aws-sdk-go/internal/sdkuri
github.com/taskcluster/taskcluster/v25/workers/generic-worker/kc
github.com/taskcluster/taskcluster/v25/workers/generic-worker/resolvetask
github.com/taskcluster/taskcluster/v25/workers/generic-worker/server-logs
github.com/aws/aws-sdk-go/aws/client
github.com/aws/aws-sdk-go/aws/corehandlers
github.com/aws/aws-sdk-go/private/protocol
github.com/aws/aws-sdk-go/aws/csm
github.com/aws/aws-sdk-go/private/protocol/rest
github.com/aws/aws-sdk-go/private/protocol/query/queryutil
github.com/aws/aws-sdk-go/private/protocol/xml/xmlutil
github.com/aws/aws-sdk-go/aws/signer/v4
github.com/aws/aws-sdk-go/private/protocol/query
github.com/aws/aws-sdk-go/aws/ec2metadata
github.com/aws/aws-sdk-go/service/sts
github.com/aws/aws-sdk-go/aws/credentials/ec2rolecreds
github.com/aws/aws-sdk-go/private/protocol/json/jsonutil
github.com/aws/aws-sdk-go/service/sts/stsiface
github.com/aws/aws-sdk-go/aws/credentials/endpointcreds
github.com/aws/aws-sdk-go/aws/credentials/stscreds
github.com/aws/aws-sdk-go/private/protocol/ec2query
github.com/aws/aws-sdk-go/aws/defaults
github.com/taskcluster/taskcluster/v25/workers/generic-worker/testutil
github.com/taskcluster/taskcluster/v25/workers/generic-worker/update-worker-pool
github.com/aws/aws-sdk-go/aws/session
github.com/aws/aws-sdk-go/service/ec2
github.com/taskcluster/taskcluster/v25/workers/generic-worker/win32
github.com/taskcluster/taskcluster/v25/workers/generic-worker/yamltojson
# github.com/aws/aws-sdk-go/service/ec2
SIGILL: illegal instruction
PC=0x242f4 m=2 sigcode=1

goroutine 35 [running]:
runtime.heapBitsSetType(0x10ef24c0, 0x20, 0x20, 0x9516b8)
	/usr/local/go/src/runtime/mbitmap.go:1276 +0x634 fp=0xb9939d4 sp=0xb99394c pc=0x242f4
runtime.mallocgc(0x20, 0x9516b8, 0x9b3401, 0x0)
	/usr/local/go/src/runtime/malloc.go:1052 +0x508 fp=0xb993a38 sp=0xb9939d4 pc=0x1b150
runtime.growslice(0x9516b8, 0x0, 0x0, 0x0, 0x1, 0x9, 0x9b3500, 0x1)
	/usr/local/go/src/runtime/slice.go:181 +0x190 fp=0xb993a68 sp=0xb993a38 pc=0x55cc4
cmd/compile/internal/gc.(*Liveness).epilogue(0xc180790)
	/usr/local/go/src/cmd/compile/internal/gc/plive.go:943 +0xfac fp=0xb993c28 sp=0xb993a68 pc=0x812be4
cmd/compile/internal/gc.liveness(0x10e9bfa0, 0xc1804d0, 0x2ff3480, 0x9b3408, 0x0, 0x0)
	/usr/local/go/src/cmd/compile/internal/gc/plive.go:1399 +0x90 fp=0xb993c74 sp=0xb993c28 pc=0x815214
cmd/compile/internal/gc.genssa(0xc1804d0, 0x2ff3480)
	/usr/local/go/src/cmd/compile/internal/gc/ssa.go:5286 +0x58 fp=0xb993f1c sp=0xb993c74 pc=0x860958
cmd/compile/internal/gc.compileSSA(0x79b2180, 0x2)
	/usr/local/go/src/cmd/compile/internal/gc/pgen.go:308 +0x2e4 fp=0xb993fc8 sp=0xb993f1c pc=0x80821c
cmd/compile/internal/gc.compileFunctions.func2(0x9cc3900, 0x8305eb0, 0x2)
	/usr/local/go/src/cmd/compile/internal/gc/pgen.go:363 +0x38 fp=0xb993fdc sp=0xb993fc8 pc=0x8b901c
runtime.goexit()
	/usr/local/go/src/runtime/asm_arm.s:868 +0x4 fp=0xb993fdc sp=0xb993fdc pc=0x6d20c
created by cmd/compile/internal/gc.compileFunctions
	/usr/local/go/src/cmd/compile/internal/gc/pgen.go:361 +0x100

goroutine 1 [runnable]:
cmd/compile/internal/gc.compileFunctions()
	/usr/local/go/src/cmd/compile/internal/gc/pgen.go:369 +0x158
cmd/compile/internal/gc.Main(0x9b314c)
	/usr/local/go/src/cmd/compile/internal/gc/main.go:695 +0x3534
main.main()
	/usr/local/go/src/cmd/compile/main.go:51 +0x98

goroutine 33 [runnable]:
cmd/compile/internal/gc.bvec.Clear(0x2, 0x100eedf0, 0x1, 0x1)
	/usr/local/go/src/cmd/compile/internal/gc/bv.go:199 +0x40
cmd/compile/internal/gc.(*Liveness).emit(0xc135e40, 0xa0, 0xb422130, 0x2)
	/usr/local/go/src/cmd/compile/internal/gc/plive.go:1367 +0x5ac
cmd/compile/internal/gc.liveness(0x10ecc4e0, 0xc135b80, 0x2fd0cc0, 0x9b3408, 0x0, 0x0)
	/usr/local/go/src/cmd/compile/internal/gc/plive.go:1430 +0x29c
cmd/compile/internal/gc.genssa(0xc135b80, 0x2fd0cc0)
	/usr/local/go/src/cmd/compile/internal/gc/ssa.go:5286 +0x58
cmd/compile/internal/gc.compileSSA(0x7852600, 0x0)
	/usr/local/go/src/cmd/compile/internal/gc/pgen.go:308 +0x2e4
cmd/compile/internal/gc.compileFunctions.func2(0x9cc3900, 0x8305eb0, 0x0)
	/usr/local/go/src/cmd/compile/internal/gc/pgen.go:363 +0x38
created by cmd/compile/internal/gc.compileFunctions
	/usr/local/go/src/cmd/compile/internal/gc/pgen.go:361 +0x100

goroutine 34 [running]:
	goroutine running on other thread; stack unavailable
created by cmd/compile/internal/gc.compileFunctions
	/usr/local/go/src/cmd/compile/internal/gc/pgen.go:361 +0x100

goroutine 36 [running]:
	goroutine running on other thread; stack unavailable
created by cmd/compile/internal/gc.compileFunctions
	/usr/local/go/src/cmd/compile/internal/gc/pgen.go:361 +0x100

trap    0x6
error   0x0
oldmask 0x0
r0      0x0
r1      0x0
r2      0x19
r3      0x1
r4      0x210842
r5      0x14
r6      0x0
r7      0x9516b8
r8      0x19
r9      0x20
r10     0x2c5e1c0
fp      0x1
ip      0x10ef24df
sp      0xb99394c
lr      0x23e44
pc      0x242f4
cpsr    0x60000010
fault   0x0

[ianlancetaylor]

This does look like a bug running the compiler itself. Is it easy to recreate? If you can recreate it under gdb, please run until the signal is received and use x/i $pc to show us the illegal instruction. Thanks.

[randall77]

It's very strange that this is intermittent. That code (heapBitsSetType), and that line in particular, gets executed all the time.
The line it occurs on shouldn't have any wonky instructions - it is just if p != endp.
I suspect a machine flake.

[randall77]

I built the compiler on linux/arm GOARM=6 for Go 1.13.7. The instruction looks fine to me:

$ cd go/src
$ GOARM=6 ./make.bash
$ go tool objdump ../pkg/tool/linux_arm/compile | grep mbitmap.go:1276
  mbitmap.go:1276	0x24188			e1a0e003		MOVW R3, R14					
  mbitmap.go:1276	0x2418c			e3a03001		MOVW $1, R3					
  mbitmap.go:1276	0x241b0			e151000e		CMP R14, R1					
  mbitmap.go:1276	0x241b4			0a000018		B.EQ 0x2421c					
  mbitmap.go:1276	0x24214			e1a0300e		MOVW R14, R3					
  mbitmap.go:1276	0x24264			e1a03007		MOVW R7, R3					
  mbitmap.go:1276	0x2428c			e1a0300e		MOVW R14, R3					
  mbitmap.go:1276	0x242b0			e58d107c		MOVW R1, 0x7c(R13)				
  mbitmap.go:1276	0x242b4			e58de070		MOVW R14, 0x70(R13)				
  mbitmap.go:1276	0x242e0			e59d107c		MOVW 0x7c(R13), R1				
  mbitmap.go:1276	0x242ec			e59de070		MOVW 0x70(R13), R14				
  mbitmap.go:1276	0x242f4			e1a0e000		MOVW R0, R14			<- this one
  mbitmap.go:1276	0x24350			e1a0100e		MOVW R14, R1					
  mbitmap.go:1276	0x24354			e3a04000		MOVW $0, R4		

Nothing looks like an invalid instruction.

We should really print the bytes at the failing PC if we get a SIGILL.

I'm going to close, as I don't see any way this could be anything other than a flake. Please reopen if you disagree.

[gopherbot]

Change https://golang.org/cl/221431 mentions this issue: runtime: print instruction bytes when reporting a SIGILL

[petemoore]

@randall77 Many thanks for adding the debug! ❤️

Indeed I haven't been able to reproduce the issue, so it most certainly was a flake as you suggested, but this extra debug is great.