Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/compile: SIGILL: illegal instruction on linux/arm #37513

petemoore opened this issue Feb 27, 2020 · 5 comments

cmd/compile: SIGILL: illegal instruction on linux/arm #37513

petemoore opened this issue Feb 27, 2020 · 5 comments


Copy link

@petemoore petemoore commented Feb 27, 2020

What version of Go are you using (go version)?

$ go version
go version go1.13.7 linux/arm

Does this issue reproduce with the latest release?

I suspect it is intermittent, since it only occurred once, so I haven't tested on go 1.14. However this was in go 1.13.7. It occurred in our CI for a particular commit, so it may be reproducible with just this commit, or perhaps it is just an intermittent failure. It only occurred on one platform, although we build for several.

What operating system and processor architecture are you using (go env)?

go env Output
$ go env
GOGCCFLAGS="-fPIC -marm -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build077222367=/tmp/go-build -gno-record-gcc-switches"

What did you do?

I ran this task.

It performed the following steps:

function b64 {
  [ "$(uname -s)" != "Darwin" ] || base64 -D
  [ "$(uname -s)" != "Linux" ]  || base64 -d
chmod a+x bin/taskcluster-proxy
# go test: -race and -msan are only supported on linux/amd64, freebsd/amd64, darwin/amd64 and windows/amd64
if [ "$(uname -m)" == "x86_64" ]; then
  # See
export CGO_ENABLED=0
export GOROOT="$(pwd)/go1.13.7/go"
export GOPATH="$(pwd)/gopath1.13.7"
export PATH="${GOPATH}/bin:${GOROOT}/bin:$(pwd)/bin:${PATH}"
git --version
go version
go env
curl -s "${TASKCLUSTER_PROXY_URL}/secrets/v1/secret/project/taskcluster/testing/generic-worker/ci-creds" | sed -n 's/.*"b64_encoded_credentials_script": "\(.*\)".*/\1/p' | b64 >
if [ ! -d taskcluster/.git ]; then
  rm -rf taskcluster
  git clone "${GITHUB_CLONE_URL}" taskcluster
cd taskcluster
git fetch "${GITHUB_CLONE_URL}" "+${GITHUB_SHA}:refs/heads/X${TASK_ID}"
git checkout -f "X${TASK_ID}"
git reset --hard "${GITHUB_SHA}"
git clean -fdx
git checkout -B tmp -t "X${TASK_ID}"
go get -v -u
cd workers/generic-worker
# go.mod and go.sum will be affected by above go get commands, so
# tidy them before checking for changes. Not needed in go 1.14:
# TODO: use go get -modfile instead when running go 1.14, and remove
# go mod tidy command
go mod tidy
git status
# output of wc command can contain spaces on darwin, so no quotes around expression
test $(git status --porcelain | wc -l) == 0
go install -tags "${ENGINE}" -v -ldflags "-X main.revision=${GITHUB_SHA}" ./...
go vet -tags "${ENGINE}" ./...
if [ "${ENGINE}" == "multiuser" ]; then
  cp "${TASK_USER_CREDENTIALS}" next-task-user.json
  # IMPORTANT - run go test with GW_TESTS_RUN_AS_CURRENT_USER=true *before* running it without
  # otherwise tests that call `go run ....` will write go object files to .cache as root
  GW_TESTS_RUN_AS_CURRENT_USER=true GORACE=history_size=7 CGO_ENABLED=1 go test -tags "${ENGINE}" -timeout 45m -ldflags "-X${GITHUB_SHA}" -v ${RACE} ${VET}
GORACE=history_size=7 CGO_ENABLED=${CGO_ENABLED_TESTS} go test -tags "${ENGINE}" -timeout 45m -ldflags "-X${GITHUB_SHA}" -v ${RACE} ${VET} ./...
../../../golangci-lint/golangci-lint-1.23.6-*/golangci-lint run --build-tags "${ENGINE}"

Please note the env variables referenced etc are included in the linked task definition, if required.

What did you expect to see?

Successful compilation. Note, the build was from this commit of our repository, which includes some misformed build constraints (such as a trailing comma).

What did you see instead?

The task produced these logs:

+ go install -tags simple -v -ldflags '-X main.revision=5230896c11537d0ada1df54c1fb8174f49740d01' ./...
go: finding v1.0.0
go: finding v0.0.0-20191115171910-c688067f12d3
go: finding v0.1.0
go: finding v1.4.1
go: finding v1.0.6
go: finding v0.0.0-20130613134717-e21c03b7a721
go: finding v1.1.0
go: finding v2.0.0+incompatible
go: finding v0.0.0-20200219161401-5fb17a1e7b9b
go: finding v1.2.0
go: finding v0.0.0-20181115110504-51d6538a90f8
go: finding v0.0.0-20180111231733-ee0de3bc6815
go: finding v1.3.0
go: finding v1.28.9
go: finding v0.2.0
go: finding v1.0.0
go: finding v2.1.0+incompatible
go: finding v0.0.0-20160212164326-8902c56451e9
go: finding v3.2.0+incompatible
go: finding v3.0.0
go: finding v0.8.1
go: finding v0.0.0-20181124034731-591f970eefbb
go: finding v13.0.0+incompatible
go: finding v3.0.0
go: finding v0.0.0-20180206201540-c2b33e8439af
go: finding v0.0.0-20140124173710-69b34d4ec901
go: finding v1.0.0
go: finding v0.0.1
go: finding v0.0.1
go: finding v0.0.0-20161026210932-d341ea318957
go: finding v0.0.0-20190425082905-87a4384529e0
go: finding v1.0.0
go: finding v2.4.1+incompatible
go: finding v0.0.0-20200104152315-a6d78f326758
go: finding v0.5.6
go: finding v0.1.1-0.20191105210325-c90efee705ee
go: finding v0.0.0-20191011141410-1b5146add898
go: finding v1.1.0
go: finding v0.0.0-20191011191535-87dc89f01550
go: finding v0.0.0-20180127040603-bd5ef7bd5415
go: finding v0.0.0-20190905194746-02993c407bfb
SIGILL: illegal instruction
PC=0x242f4 m=2 sigcode=1

goroutine 35 [running]:
runtime.heapBitsSetType(0x10ef24c0, 0x20, 0x20, 0x9516b8)
	/usr/local/go/src/runtime/mbitmap.go:1276 +0x634 fp=0xb9939d4 sp=0xb99394c pc=0x242f4
runtime.mallocgc(0x20, 0x9516b8, 0x9b3401, 0x0)
	/usr/local/go/src/runtime/malloc.go:1052 +0x508 fp=0xb993a38 sp=0xb9939d4 pc=0x1b150
runtime.growslice(0x9516b8, 0x0, 0x0, 0x0, 0x1, 0x9, 0x9b3500, 0x1)
	/usr/local/go/src/runtime/slice.go:181 +0x190 fp=0xb993a68 sp=0xb993a38 pc=0x55cc4
	/usr/local/go/src/cmd/compile/internal/gc/plive.go:943 +0xfac fp=0xb993c28 sp=0xb993a68 pc=0x812be4
cmd/compile/internal/gc.liveness(0x10e9bfa0, 0xc1804d0, 0x2ff3480, 0x9b3408, 0x0, 0x0)
	/usr/local/go/src/cmd/compile/internal/gc/plive.go:1399 +0x90 fp=0xb993c74 sp=0xb993c28 pc=0x815214
cmd/compile/internal/gc.genssa(0xc1804d0, 0x2ff3480)
	/usr/local/go/src/cmd/compile/internal/gc/ssa.go:5286 +0x58 fp=0xb993f1c sp=0xb993c74 pc=0x860958
cmd/compile/internal/gc.compileSSA(0x79b2180, 0x2)
	/usr/local/go/src/cmd/compile/internal/gc/pgen.go:308 +0x2e4 fp=0xb993fc8 sp=0xb993f1c pc=0x80821c
cmd/compile/internal/gc.compileFunctions.func2(0x9cc3900, 0x8305eb0, 0x2)
	/usr/local/go/src/cmd/compile/internal/gc/pgen.go:363 +0x38 fp=0xb993fdc sp=0xb993fc8 pc=0x8b901c
	/usr/local/go/src/runtime/asm_arm.s:868 +0x4 fp=0xb993fdc sp=0xb993fdc pc=0x6d20c
created by cmd/compile/internal/gc.compileFunctions
	/usr/local/go/src/cmd/compile/internal/gc/pgen.go:361 +0x100

goroutine 1 [runnable]:
	/usr/local/go/src/cmd/compile/internal/gc/pgen.go:369 +0x158
	/usr/local/go/src/cmd/compile/internal/gc/main.go:695 +0x3534
	/usr/local/go/src/cmd/compile/main.go:51 +0x98

goroutine 33 [runnable]:
cmd/compile/internal/gc.bvec.Clear(0x2, 0x100eedf0, 0x1, 0x1)
	/usr/local/go/src/cmd/compile/internal/gc/bv.go:199 +0x40
cmd/compile/internal/gc.(*Liveness).emit(0xc135e40, 0xa0, 0xb422130, 0x2)
	/usr/local/go/src/cmd/compile/internal/gc/plive.go:1367 +0x5ac
cmd/compile/internal/gc.liveness(0x10ecc4e0, 0xc135b80, 0x2fd0cc0, 0x9b3408, 0x0, 0x0)
	/usr/local/go/src/cmd/compile/internal/gc/plive.go:1430 +0x29c
cmd/compile/internal/gc.genssa(0xc135b80, 0x2fd0cc0)
	/usr/local/go/src/cmd/compile/internal/gc/ssa.go:5286 +0x58
cmd/compile/internal/gc.compileSSA(0x7852600, 0x0)
	/usr/local/go/src/cmd/compile/internal/gc/pgen.go:308 +0x2e4
cmd/compile/internal/gc.compileFunctions.func2(0x9cc3900, 0x8305eb0, 0x0)
	/usr/local/go/src/cmd/compile/internal/gc/pgen.go:363 +0x38
created by cmd/compile/internal/gc.compileFunctions
	/usr/local/go/src/cmd/compile/internal/gc/pgen.go:361 +0x100

goroutine 34 [running]:
	goroutine running on other thread; stack unavailable
created by cmd/compile/internal/gc.compileFunctions
	/usr/local/go/src/cmd/compile/internal/gc/pgen.go:361 +0x100

goroutine 36 [running]:
	goroutine running on other thread; stack unavailable
created by cmd/compile/internal/gc.compileFunctions
	/usr/local/go/src/cmd/compile/internal/gc/pgen.go:361 +0x100

trap    0x6
error   0x0
oldmask 0x0
r0      0x0
r1      0x0
r2      0x19
r3      0x1
r4      0x210842
r5      0x14
r6      0x0
r7      0x9516b8
r8      0x19
r9      0x20
r10     0x2c5e1c0
fp      0x1
ip      0x10ef24df
sp      0xb99394c
lr      0x23e44
pc      0x242f4
cpsr    0x60000010
fault   0x0
@andybons andybons changed the title SIGILL: illegal instruction on linux/arm cmd/compile: SIGILL: illegal instruction on linux/arm Feb 27, 2020
Copy link

@ianlancetaylor ianlancetaylor commented Feb 27, 2020

This does look like a bug running the compiler itself. Is it easy to recreate? If you can recreate it under gdb, please run until the signal is received and use x/i $pc to show us the illegal instruction. Thanks.

Copy link

@randall77 randall77 commented Feb 27, 2020

It's very strange that this is intermittent. That code (heapBitsSetType), and that line in particular, gets executed all the time.
The line it occurs on shouldn't have any wonky instructions - it is just if p != endp.
I suspect a machine flake.

Copy link

@randall77 randall77 commented Feb 27, 2020

I built the compiler on linux/arm GOARM=6 for Go 1.13.7. The instruction looks fine to me:

$ cd go/src
$ GOARM=6 ./make.bash
$ go tool objdump ../pkg/tool/linux_arm/compile | grep mbitmap.go:1276
  mbitmap.go:1276	0x24188			e1a0e003		MOVW R3, R14					
  mbitmap.go:1276	0x2418c			e3a03001		MOVW $1, R3					
  mbitmap.go:1276	0x241b0			e151000e		CMP R14, R1					
  mbitmap.go:1276	0x241b4			0a000018		B.EQ 0x2421c					
  mbitmap.go:1276	0x24214			e1a0300e		MOVW R14, R3					
  mbitmap.go:1276	0x24264			e1a03007		MOVW R7, R3					
  mbitmap.go:1276	0x2428c			e1a0300e		MOVW R14, R3					
  mbitmap.go:1276	0x242b0			e58d107c		MOVW R1, 0x7c(R13)				
  mbitmap.go:1276	0x242b4			e58de070		MOVW R14, 0x70(R13)				
  mbitmap.go:1276	0x242e0			e59d107c		MOVW 0x7c(R13), R1				
  mbitmap.go:1276	0x242ec			e59de070		MOVW 0x70(R13), R14				
  mbitmap.go:1276	0x242f4			e1a0e000		MOVW R0, R14			<- this one
  mbitmap.go:1276	0x24350			e1a0100e		MOVW R14, R1					
  mbitmap.go:1276	0x24354			e3a04000		MOVW $0, R4		

Nothing looks like an invalid instruction.

We should really print the bytes at the failing PC if we get a SIGILL.

I'm going to close, as I don't see any way this could be anything other than a flake. Please reopen if you disagree.

@randall77 randall77 closed this Feb 27, 2020
Copy link

@gopherbot gopherbot commented Feb 28, 2020

Change mentions this issue: runtime: print instruction bytes when reporting a SIGILL

gopherbot pushed a commit that referenced this issue Mar 2, 2020
Print the bytes of the instruction that generated a SIGILL.
This should help us respond to bug reports without having to
go back-and-forth with the reporter to get the instruction involved.
Might also help with SIGILL problems that are difficult to reproduce.

Update #37513

Change-Id: I33059b1dbfc97bce16142a843f32a88a6547e280
Run-TryBot: Keith Randall <>
TryBot-Result: Gobot Gobot <>
Reviewed-by: Ian Lance Taylor <>
Copy link

@petemoore petemoore commented Mar 12, 2020

@randall77 Many thanks for adding the debug! ❤️

Indeed I haven't been able to reproduce the issue, so it most certainly was a flake as you suggested, but this extra debug is great.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
4 participants
You can’t perform that action at this time.