runtime: 1.5.1 compiler crashed with "unswept span"

[omerkirk]

I deployed the same code to two of our servers. These two servers have the same OS (Centos 6.7) and the same kernel (2.6.32-573.8.1.el6.x86_64). However they have different configurations like RAM and CPU.

The code I deployed was working on these servers, however we found a bug and introduced a small fix. One of our servers built the code without issues however on our other server go build panics and produces the following output. I had to reboot the server to be able to build the source.

It looks like the garbage collector didn't shutdown properly with the app. The app crashed and that is why we had to introduce a fix, however I expect go to cleanup after a crash.

We use go version 1.5.1 and here is my go env output

GOARCH="amd64"
GOBIN=""
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/home/kokteyl/go"
GORACE=""
GOROOT="/usr/local/go"
GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"
GO15VENDOREXPERIMENT=""
CC="gcc"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0"
CXX="g++"
CGO_ENABLED="1"

Please let me know if there is any other info you need to understand the issue.

Panic Stacktrace for go build:

sweep 136 8
fatal error: gc: unswept span

runtime stack:
runtime.throw(0x8408c0, 0x10)
        /usr/local/go/src/runtime/panic.go:527 +0x90 fp=0xc82003dce0 sp=0xc82003dcc8
runtime.markroot(0xc82001c000, 0x3)
        /usr/local/go/src/runtime/mgcmark.go:88 +0x375 fp=0xc82003dd80 sp=0xc82003dce0
runtime.parfordo(0xc82001c000)
        /usr/local/go/src/runtime/parfor.go:110 +0x1d4 fp=0xc82003de08 sp=0xc82003dd80
runtime.gchelper()
        /usr/local/go/src/runtime/mgc.go:1701 +0x56 fp=0xc82003de50 sp=0xc82003de08
runtime.stopm()
        /usr/local/go/src/runtime/proc1.go:1131 +0x146 fp=0xc82003de78 sp=0xc82003de50
runtime.gcstopm()
        /usr/local/go/src/runtime/proc1.go:1321 +0xf8 fp=0xc82003dea8 sp=0xc82003de78
runtime.findrunnable(0xc82001e000, 0x0)
        /usr/local/go/src/runtime/proc1.go:1382 +0x35 fp=0xc82003df28 sp=0xc82003dea8
runtime.schedule()
        /usr/local/go/src/runtime/proc1.go:1639 +0x267 fp=0xc82003df60 sp=0xc82003df28
runtime.park_m(0xc820001200)
        /usr/local/go/src/runtime/proc1.go:1698 +0x18b fp=0xc82003df88 sp=0xc82003df60
runtime.mcall(0x0)
        /usr/local/go/src/runtime/asm_amd64.s:204 +0x5b fp=0xc82003df98 sp=0xc82003df88

goroutine 1 [runnable]:
cmd/compile/internal/amd64.copyau(0xc8215e3da8, 0xc8215e3b68, 0xc8215e3b01)
        /usr/local/go/src/cmd/compile/internal/amd64/peep.go:914 fp=0xc8217670a8 sp=0xc8217670a0
cmd/compile/internal/amd64.copyu(0xc8215e3d40, 0xc8215e3b68, 0x0, 0x1)
        /usr/local/go/src/cmd/compile/internal/amd64/peep.go:806 +0x116 fp=0xc8217670d8 sp=0xc8217670a8
cmd/compile/internal/amd64.copy1(0xc8215e3b10, 0xc8215e3b68, 0xc821746028, 0x0, 0x1)
        /usr/local/go/src/cmd/compile/internal/amd64/peep.go:690 +0x54f fp=0xc821767260 sp=0xc8217670d8
cmd/compile/internal/amd64.copyprop(0xc8216a2030, 0xc821745f20, 0xc8216a2001)
        /usr/local/go/src/cmd/compile/internal/amd64/peep.go:661 +0x179 fp=0xc8217672f0 sp=0xc821767260
cmd/compile/internal/amd64.peep(0xc821528d80)
        /usr/local/go/src/cmd/compile/internal/amd64/peep.go:143 +0x717 fp=0xc821767350 sp=0xc8217672f0
cmd/compile/internal/gc.regopt(0xc821528d80)
        /usr/local/go/src/cmd/compile/internal/gc/reg.go:1412 +0x2da1 fp=0xc821767808 sp=0xc821767350
cmd/compile/internal/gc.compile(0xc821254ab0)
        /usr/local/go/src/cmd/compile/internal/gc/pgen.go:508 +0xe82 fp=0xc821767a78 sp=0xc821767808
cmd/compile/internal/gc.funccompile(0xc821254ab0)
        /usr/local/go/src/cmd/compile/internal/gc/dcl.go:1484 +0x1c9 fp=0xc821767af0 sp=0xc821767a78
cmd/compile/internal/gc.Main()
        /usr/local/go/src/cmd/compile/internal/gc/lex.go:473 +0x1f08 fp=0xc821767df0 sp=0xc821767af0
cmd/compile/internal/amd64.Main()
        /usr/local/go/src/cmd/compile/internal/amd64/galign.go:127 +0x58d fp=0xc821767e58 sp=0xc821767df0
main.main()
        /usr/local/go/src/cmd/compile/main.go:26 +0x189 fp=0xc821767f30 sp=0xc821767e58
runtime.main()
        /usr/local/go/src/runtime/proc.go:111 +0x2b0 fp=0xc821767f80 sp=0xc821767f30
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1696 +0x1 fp=0xc821767f88 sp=0xc821767f80

goroutine 2 [force gc (idle)]:
runtime.gopark(0x89b7a8, 0xc7f6b0, 0x81e0d0, 0xf, 0x14, 0x1)
        /usr/local/go/src/runtime/proc.go:185 +0x163 fp=0xc82002a758 sp=0xc82002a730
runtime.goparkunlock(0xc7f6b0, 0x81e0d0, 0xf, 0xc820000114, 0x1)
        /usr/local/go/src/runtime/proc.go:191 +0x54 fp=0xc82002a790 sp=0xc82002a758
runtime.forcegchelper()
        /usr/local/go/src/runtime/proc.go:152 +0xb8 fp=0xc82002a7c0 sp=0xc82002a790
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1696 +0x1 fp=0xc82002a7c8 sp=0xc82002a7c0
created by runtime.init.4
        /usr/local/go/src/runtime/proc.go:141 +0x2b

goroutine 3 [GC sweep wait]:
runtime.gopark(0x89b7a8, 0xc7f880, 0x819ab0, 0xd, 0x44af14, 0x1)
        /usr/local/go/src/runtime/proc.go:185 +0x163 fp=0xc82002af48 sp=0xc82002af20
runtime.goparkunlock(0xc7f880, 0x819ab0, 0xd, 0x14, 0x1)
        /usr/local/go/src/runtime/proc.go:191 +0x54 fp=0xc82002af80 sp=0xc82002af48
runtime.bgsweep(0xc820052000)
        /usr/local/go/src/runtime/mgcsweep.go:67 +0x14d fp=0xc82002afb8 sp=0xc82002af80
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1696 +0x1 fp=0xc82002afc0 sp=0xc82002afb8
created by runtime.gcenable
        /usr/local/go/src/runtime/mgc.go:206 +0x53

goroutine 4 [finalizer wait]:
runtime.gopark(0x89b7a8, 0xda6730, 0x81dcf0, 0xe, 0x14, 0x1)
        /usr/local/go/src/runtime/proc.go:185 +0x163 fp=0xc82002b718 sp=0xc82002b6f0
runtime.goparkunlock(0xda6730, 0x81dcf0, 0xe, 0x14, 0x1)
        /usr/local/go/src/runtime/proc.go:191 +0x54 fp=0xc82002b750 sp=0xc82002b718
runtime.runfinq()
        /usr/local/go/src/runtime/mfinal.go:154 +0xaa fp=0xc82002b7c0 sp=0xc82002b750
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1696 +0x1 fp=0xc82002b7c8 sp=0xc82002b7c0
created by runtime.createfing
        /usr/local/go/src/runtime/mfinal.go:135 +0x60

goroutine 5 [garbage collection]:
runtime.systemstack_switch()
        /usr/local/go/src/runtime/asm_amd64.s:216 fp=0xc82002bc98 sp=0xc82002bc90
runtime.gc(0x0)
        /usr/local/go/src/runtime/mgc.go:1097 +0x3d0 fp=0xc82002bf90 sp=0xc82002bc98
runtime.backgroundgc()
        /usr/local/go/src/runtime/mgc.go:897 +0x3d fp=0xc82002bfc0 sp=0xc82002bf90
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1696 +0x1 fp=0xc82002bfc8 sp=0xc82002bfc0
created by runtime.startGC
        /usr/local/go/src/runtime/mgc.go:870 +0x186

goroutine 6 [mark worker (idle)]:
runtime.gopark(0x89b660, 0xc820030900, 0x841d80, 0x12, 0x14, 0x0)
        /usr/local/go/src/runtime/proc.go:185 +0x163 fp=0xc82002c750 sp=0xc82002c728
runtime.gcBgMarkWorker(0xc82001e000)
        /usr/local/go/src/runtime/mgc.go:1289 +0xf7 fp=0xc82002c7b8 sp=0xc82002c750
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1696 +0x1 fp=0xc82002c7c0 sp=0xc82002c7b8
created by runtime.gcBgMarkStartWorkers
        /usr/local/go/src/runtime/mgc.go:1239 +0x93

goroutine 7 [mark worker (idle)]:
runtime.gopark(0x89b660, 0xc820030480, 0x841d80, 0x12, 0x14, 0x0)
        /usr/local/go/src/runtime/proc.go:185 +0x163 fp=0xc82002cf50 sp=0xc82002cf28
runtime.gcBgMarkWorker(0xc82001f500)
        /usr/local/go/src/runtime/mgc.go:1289 +0xf7 fp=0xc82002cfb8 sp=0xc82002cf50
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1696 +0x1 fp=0xc82002cfc0 sp=0xc82002cfb8
created by runtime.gcBgMarkStartWorkers
        /usr/local/go/src/runtime/mgc.go:1239 +0x93

goroutine 8 [mark worker (idle)]:
runtime.gopark(0x89b660, 0xc8204aa000, 0x841d80, 0x12, 0x14, 0x0)
        /usr/local/go/src/runtime/proc.go:185 +0x163 fp=0xc82002d750 sp=0xc82002d728
runtime.gcBgMarkWorker(0xc820020a00)
        /usr/local/go/src/runtime/mgc.go:1289 +0xf7 fp=0xc82002d7b8 sp=0xc82002d750
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1696 +0x1 fp=0xc82002d7c0 sp=0xc82002d7b8
created by runtime.gcBgMarkStartWorkers
        /usr/local/go/src/runtime/mgc.go:1239 +0x93

goroutine 9 [mark worker (idle)]:
runtime.gopark(0x89b660, 0xc81340, 0x841d80, 0x12, 0x14, 0x0)
        /usr/local/go/src/runtime/proc.go:185 +0x163 fp=0xc82002df50 sp=0xc82002df28
runtime.gcBgMarkWorker(0xc820022000)
        /usr/local/go/src/runtime/mgc.go:1289 +0xf7 fp=0xc82002dfb8 sp=0xc82002df50
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1696 +0x1 fp=0xc82002dfc0 sp=0xc82002dfb8
created by runtime.gcBgMarkStartWorkers
        /usr/local/go/src/runtime/mgc.go:1239 +0x93

goroutine 10 [timer goroutine (idle)]:
runtime.gopark(0x89b7a8, 0xc7f8e0, 0x844920, 0x16, 0x14, 0x1)
        /usr/local/go/src/runtime/proc.go:185 +0x163 fp=0xc820026700 sp=0xc8200266d8
runtime.goparkunlock(0xc7f8e0, 0x844920, 0x16, 0x14, 0x1)
        /usr/local/go/src/runtime/proc.go:191 +0x54 fp=0xc820026738 sp=0xc820026700
runtime.timerproc()
        /usr/local/go/src/runtime/time.go:202 +0x123 fp=0xc8200267c0 sp=0xc820026738
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1696 +0x1 fp=0xc8200267c8 sp=0xc8200267c0
created by runtime.addtimerLocked
        /usr/local/go/src/runtime/time.go:116 +0x11f

[davecheney]

@omerkirk Go 1.5.2 was released recently. Can you please check to see if the crash occurs with Go 1.5.2 ?

[omerkirk]

@davecheney I have been using go for almost 2 years now and this is the first I have encountered an issue like this. And I am not really sure how to reproduce it. We have 5 active high-load web services running on go 1.4.2 without issues. Since we started using 1.5.1 we are experiencing regular crashes related to cgo.

I will try with 1.5.2 and update the issue if there are anymore issues.

[davecheney]

@omerkirk, the issue you reported was a compiler crash.

If you are also having issues with cgo under go 1.5.1, would you please log a separate issue as the compiler does not use cgo, so the issues are unrelated.

On 14 Dec 2015, at 22:56, omerkirk notifications@github.com wrote:

@davecheney I have been using go for almost 2 years now and this is the first I have encountered an issue like this. And I am not really sure how to reproduce it. We have 5 active high-load web services running on go 1.4.2 without issues. Since we started using 1.5.1 we are experiencing regular crashes related to cgo.

I will try with 1.5.2 and update the issue if there are anymore issues.

—
Reply to this email directly or view it on GitHub.

[omerkirk]

@davecheney the compiler issue we encountered was after our app crashed due to cgo. That is why I referred to it. Now that I read it again I didn't make it clear sorry about that.

I am planning to open a seperate issue about the crashes we are seeing related to cgo.

[ianlancetaylor]

The compiler crash reported here is not connected to whatever cgo problems you have been having.

Have you seen this compiler crash more than once?

[ianlancetaylor]

CC @aclements to see if crash might have been fixed in 1.5.2 or tip.

[omerkirk]

@ianlancetaylor This is the first time we had this compiler issue.

The only connection is that it happened after our app crashed because of cgo. As I said that is the only reason I referred to cgo, I don't think they are related also.

We are now using go 1.5.2 and will update the issue if we see it again.

[aclements]

sweep 136 8

This line is saying that the global sweep generation is 8 and the bad span's sweep generation is 136. Since the sweep generation is monotonic, this suggests general memory corruption to me, and not anything specifically related to sweeping. We did fix some memory corruption issues in Go 1.5.2 (specifically relating to non-atomic pointer writes), so this could be fixed.

[ianlancetaylor]

I'm going to close this for now, but please reopen if you have a way to reproduce the problem.