Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

runtime: sporadic crashes with CONFIG_NO_HZ_FULL=y enabled (usually during cgo compilation) #18067

Closed
1-z opened this issue Nov 27, 2016 · 6 comments

Comments

Projects
None yet
4 participants
@1-z
Copy link

commented Nov 27, 2016

Please answer these questions before submitting your issue. Thanks!

What version of Go are you using (go version)?

Problem experienced on 1.7.3 linux/amd64 and tip linux/amd64

What operating system and processor architecture are you using (go env)?

GOARCH="amd64"
GOBIN=""
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH=""
GORACE=""
GOROOT="/usr/local/go"
GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"
CC="gcc"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build252688198=/tmp/go-build -gno-record-gcc-switches"
CXX="g++"
CGO_ENABLED="1"

What did you do?

Using kernel 4.8.11 or mainline, and kernel config option CONFIG_NO_HZ_FULL=y enabled, attempted to compile various projects that use cgo. After much experimentation, determined that disabling kernel config CONFIG_NO_HZ_FULL stops the problem.

Not sure if this is a Go bug or a Linux kernel bug, or where to proceed from here.

What did you expect to see?

go build compiles code successfully.

What did you see instead?

Non-deterministic crashing leading to a variety of stack traces. Approximately 80/100 repeated identical builds are fine, but the remainder are a smattering of the traces seen here (separated by --- SNIP ---)

fatal error: acquireSudog: found s.elem != nil in cache

goroutine 9 [running]:
runtime.throw(0x85a694, 0x2a)
/usr/local/go/src/runtime/panic.go:566 +0x95 fp=0xc420377568 sp=0xc420377548
runtime.acquireSudog(0xc42018bb6c)
/usr/local/go/src/runtime/proc.go:306 +0x2a3 fp=0xc4203775f0 sp=0xc420377568
runtime.semacquire(0xc42018bb6c, 0x4d9401)
/usr/local/go/src/runtime/sema.go:90 +0x5e fp=0xc420377658 sp=0xc4203775f0
sync.runtime_Semacquire(0xc42018bb6c)
/usr/local/go/src/runtime/sema.go:47 +0x30 fp=0xc420377678 sp=0xc420377658
sync.(*Mutex).Lock(0xc42018bb68)
/usr/local/go/src/sync/mutex.go:85 +0xd0 fp=0xc4203776d0 sp=0xc420377678
main.(*builder).do.func2(0xc420233d70, 0xc42018bb20, 0xc420269480)
/usr/local/go/src/cmd/go/build.go:1318 +0x104 fp=0xc4203777a8 sp=0xc4203776d0
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:2086 +0x1 fp=0xc4203777b0 sp=0xc4203777a8
created by main.(*builder).do
/usr/local/go/src/cmd/go/build.go:1327 +0x4b8

goroutine 1 [semacquire]:
sync.runtime_Semacquire(0xc420233d7c)
/usr/local/go/src/runtime/sema.go:47 +0x30
sync.(*WaitGroup).Wait(0xc420233d70)
/usr/local/go/src/sync/waitgroup.go:131 +0x97
main.(*builder).do(0xc42018bb20, 0xc42015b5f0)
/usr/local/go/src/cmd/go/build.go:1330 +0x4e1
main.runBuild(0xa27400, 0xc42000c330, 0x0, 0x0)
/usr/local/go/src/cmd/go/build.go:496 +0x310
main.main()
/usr/local/go/src/cmd/go/main.go:181 +0x624

goroutine 17 [syscall, locked to thread]:
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:2086 +0x1

goroutine 5 [syscall]:
os/signal.signal_recv(0x0)
/usr/local/go/src/runtime/sigqueue.go:116 +0x157
os/signal.loop()
/usr/local/go/src/os/signal/signal_unix.go:22 +0x22
created by os/signal.init.1
/usr/local/go/src/os/signal/signal_unix.go:28 +0x41

goroutine 7 [select]:
main.(*builder).do.func2(0xc420233d70, 0xc42018bb20, 0xc420269480)
/usr/local/go/src/cmd/go/build.go:1311 +0x1bf
created by main.(*builder).do
/usr/local/go/src/cmd/go/build.go:1327 +0x4b8

goroutine 8 [runnable]:
sync.runtime_Semacquire(0xc42018bb6c)
/usr/local/go/src/runtime/sema.go:47 +0x30
sync.(*Mutex).Lock(0xc42018bb68)
/usr/local/go/src/sync/mutex.go:85 +0xd0
main.(*builder).do.func1(0xc42040e8f0)
/usr/local/go/src/cmd/go/build.go:1269 +0xb2
main.(*builder).do.func2(0xc420233d70, 0xc42018bb20, 0xc420269480)
/usr/local/go/src/cmd/go/build.go:1321 +0x147
created by main.(*builder).do
/usr/local/go/src/cmd/go/build.go:1327 +0x4b8

goroutine 10 [select]:
main.(*builder).do.func2(0xc420233d70, 0xc42018bb20, 0xc420269480)
/usr/local/go/src/cmd/go/build.go:1311 +0x1bf
created by main.(*builder).do
/usr/local/go/src/cmd/go/build.go:1327 +0x4b8

--- SNIP ---

fatal error: unexpected signal during runtime execution
[signal SIGSEGV: segmentation violation code=0x2 addr=0x7f0f3cfbe000 pc=0x7f0f3cfbe000]

goroutine 5 [running]:
runtime.throw(0x90b6d5, 0x2a)
/usr/local/go/src/runtime/panic.go:566 +0x95 fp=0xc420029ea8 sp=0xc420029e88
runtime.sigpanic()
/usr/local/go/src/runtime/sigpanic_unix.go:12 +0x2cc fp=0xc420029f00 sp=0xc420029ea8
created by runtime.gcBgMarkStartWorkers
/usr/local/go/src/runtime/mgc.go:1341 +0x98

goroutine 1 [runnable]:
cmd/compile/internal/gc.autoexport(0xc4204741b0, 0xc420474106)
/usr/local/go/src/cmd/compile/internal/gc/export.go:76
cmd/compile/internal/gc.declare(0xc4204741b0, 0xc420476006)
/usr/local/go/src/cmd/compile/internal/gc/dcl.go:215 +0x184
cmd/compile/internal/gc.(*importer).obj(0xc4200da380, 0xfffffffffffffffb)
/usr/local/go/src/cmd/compile/internal/gc/bimport.go:309 +0x491
cmd/compile/internal/gc.Import(0xc42006e300)
/usr/local/go/src/cmd/compile/internal/gc/bimport.go:97 +0x3b2
cmd/compile/internal/gc.importfile(0xc420414800, 0x0, 0x0, 0x0)
/usr/local/go/src/cmd/compile/internal/gc/main.go:843 +0x9b1
cmd/compile/internal/gc.(*parser).importdcl(0xc4204147e0)
/usr/local/go/src/cmd/compile/internal/gc/parser.go:341 +0x178
cmd/compile/internal/gc.(*parser).import_(0xc4204147e0)
/usr/local/go/src/cmd/compile/internal/gc/parser.go:307 +0xd1
cmd/compile/internal/gc.(*parser).file(0xc4204147e0)
/usr/local/go/src/cmd/compile/internal/gc/parser.go:268 +0x5e
cmd/compile/internal/gc.parse_file(0xc42006e240)
/usr/local/go/src/cmd/compile/internal/gc/parser.go:31 +0x54
cmd/compile/internal/gc.Main()
/usr/local/go/src/cmd/compile/internal/gc/main.go:334 +0x14f4
cmd/compile/internal/amd64.Main()
/usr/local/go/src/cmd/compile/internal/amd64/galign.go:93 +0x2fa
main.main()
/usr/local/go/src/cmd/compile/main.go:33 +0x2a3

--- SNIP ---

runtime: nelems=51 nfree=47 nalloc=4 previous allocCount=3 nfreed=65535
fatal error: sweep increased allocation count

runtime stack:
runtime.throw(0x856af1, 0x20)
/usr/local/go/src/runtime/panic.go:566 +0x95
runtime.(*mspan).sweep(0x7f0a0b772c38, 0x300000000, 0xc400000101)
/usr/local/go/src/runtime/mgcsweep.go:287 +0x7ab
runtime.sweepone(0x4a8a06)
/usr/local/go/src/runtime/mgcsweep.go:112 +0xfe
runtime.gosweepone.func1()
/usr/local/go/src/runtime/mgcsweep.go:124 +0x2b
runtime.systemstack(0xc42001ca00)
/usr/local/go/src/runtime/asm_amd64.s:298 +0x79
runtime.mstart()
/usr/local/go/src/runtime/proc.go:1079

goroutine 3 [running]:
runtime.systemstack_switch()
/usr/local/go/src/runtime/asm_amd64.s:252 fp=0xc420026f48 sp=0xc420026f40
runtime.gosweepone(0x1)
/usr/local/go/src/runtime/mgcsweep.go:125 +0x4a fp=0xc420026f78 sp=0xc420026f48
runtime.bgsweep(0xc420056000)
/usr/local/go/src/runtime/mgcsweep.go:66 +0xbb fp=0xc420026fb8 sp=0xc420026f78
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:2086 +0x1 fp=0xc420026fc0 sp=0xc420026fb8
created by runtime.gcenable
/usr/local/go/src/runtime/mgc.go:195 +0x61

goroutine 1 [runnable]:
go/scanner.(*Scanner).next(0xc4201ce320)
/usr/local/go/src/go/scanner/scanner.go:56
go/scanner.(*Scanner).scanComment(0xc4201ce320, 0x100, 0x45b)
/usr/local/go/src/go/scanner/scanner.go:195 +0x1b7
go/scanner.(*Scanner).Scan(0xc4201ce320, 0x891459, 0xc41ffdd6d3, 0x0, 0x0)
/usr/local/go/src/go/scanner/scanner.go:709 +0x101
go/parser.(*parser).next0(0xc4201ce300)
/usr/local/go/src/go/parser/parser.go:257 +0x78
go/parser.(*parser).next(0xc4201ce300)
/usr/local/go/src/go/parser/parser.go:320 +0x69
go/parser.(*parser).expectSemi(0xc4201ce300)
/usr/local/go/src/go/parser/parser.go:419 +0x4e
go/parser.(*parser).parseFile(0xc4201ce300, 0x0)
/usr/local/go/src/go/parser/parser.go:2468 +0xd6
go/parser.ParseFile(0xc4201d1240, 0xc420158720, 0x24, 0x7bd6a0, 0xc420452420, 0x6, 0x0, 0x0, 0x0)
/usr/local/go/src/go/parser/interface.go:121 +0x1e1
go/build.(*Context).Import(0xa2ea40, 0x84cdf9, 0xb, 0xc4201ae3a0, 0x19, 0x4, 0xc4204ccd80, 0xc42040ee88, 0x47fdc4)
/usr/local/go/src/go/build/build.go:739 +0xcac
main.loadImport(0x84cdf9, 0xb, 0xc4201ae3a0, 0x19, 0xc4201b0000, 0xc42040f9e0, 0x0, 0x0, 0x0, 0x1, ...)
/usr/local/go/src/cmd/go/pkg.go:365 +0x3b9
main.(*Package).load(0xc4201b0000, 0xc42040f9e0, 0xc420185500, 0x0, 0x0, 0xc)
/usr/local/go/src/cmd/go/pkg.go:940 +0x12b6
main.loadImport(0x848825, 0x1, 0xc42000c0c4, 0x19, 0x0, 0xc42040f9e0, 0x0, 0x0, 0x0, 0x0, ...)
/usr/local/go/src/cmd/go/pkg.go:374 +0x470
main.loadPackage(0x848825, 0x1, 0xc42040f9e0, 0x0)
/usr/local/go/src/cmd/go/pkg.go:1658 +0x28f
main.packagesAndErrors(0xc4201a2230, 0x1, 0x0, 0xc4201a2210, 0x20, 0x7f3a00)
/usr/local/go/src/cmd/go/pkg.go:1703 +0x337
main.packagesForBuild(0xc42000c330, 0x0, 0x0, 0x893cc8, 0xc420014140, 0x0)
/usr/local/go/src/cmd/go/pkg.go:1719 +0x75
main.runBuild(0xa27400, 0xc42000c330, 0x0, 0x0)
/usr/local/go/src/cmd/go/build.go:440 +0xd4
main.main()
/usr/local/go/src/cmd/go/main.go:181 +0x624

goroutine 17 [syscall, locked to thread]:
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:2086 +0x1

goroutine 5 [syscall]:
os/signal.signal_recv(0x0)
/usr/local/go/src/runtime/sigqueue.go:116 +0x157
os/signal.loop()
/usr/local/go/src/os/signal/signal_unix.go:22 +0x22
created by os/signal.init.1
/usr/local/go/src/os/signal/signal_unix.go:28 +0x41

--- SNIP ---

unexpected fault address 0x0
fatal error: fault
[signal SIGSEGV: segmentation violation code=0x80 addr=0x0 pc=0x44cff6]

goroutine 1 [running]:
runtime.throw(0x84988c, 0x5)
/usr/local/go/src/runtime/panic.go:566 +0x95 fp=0xc42042b6f0 sp=0xc42042b6d0
runtime.sigpanic()
/usr/local/go/src/runtime/sigpanic_unix.go:27 +0x288 fp=0xc42042b748 sp=0xc42042b6f0
main.isStale(0xc42029e480, 0x0, 0x85ae3a, 0x2b)
/usr/local/go/src/cmd/go/pkg.go:1433 +0x136 fp=0xc42042b8b8 sp=0xc42042b748
main.computeStale(0xc42002a430, 0x1, 0x1)
/usr/local/go/src/cmd/go/pkg.go:1101 +0x74 fp=0xc42042b918 sp=0xc42042b8b8
main.packagesAndErrors(0xc4201406c0, 0x1, 0x0, 0xc4201406a0, 0x20, 0x7f3a00)
/usr/local/go/src/cmd/go/pkg.go:1710 +0x4a2 fp=0xc42042bb58 sp=0xc42042b918
main.packagesForBuild(0xc42000c330, 0x0, 0x0, 0x893cc8, 0xc420014140, 0x0)
/usr/local/go/src/cmd/go/pkg.go:1719 +0x75 fp=0xc42042bc58 sp=0xc42042bb58
main.runBuild(0xa27400, 0xc42000c330, 0x0, 0x0)
/usr/local/go/src/cmd/go/build.go:440 +0xd4 fp=0xc42042bdb8 sp=0xc42042bc58
main.main()
/usr/local/go/src/cmd/go/main.go:181 +0x624 fp=0xc42042bf28 sp=0xc42042bdb8
runtime.main()
/usr/local/go/src/runtime/proc.go:183 +0x1f4 fp=0xc42042bf80 sp=0xc42042bf28
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:2086 +0x1 fp=0xc42042bf88 sp=0xc42042bf80

goroutine 17 [syscall, locked to thread]:
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:2086 +0x1

goroutine 5 [syscall]:
os/signal.signal_recv(0x0)
/usr/local/go/src/runtime/sigqueue.go:116 +0x157
os/signal.loop()
/usr/local/go/src/os/signal/signal_unix.go:22 +0x22
created by os/signal.init.1
/usr/local/go/src/os/signal/signal_unix.go:28 +0x41

--- SNIP ---

fatal error: unexpected signal during runtime execution
[signal SIGSEGV: segmentation violation code=0x2 addr=0x7fb4d92c4800 pc=0x7fb4d92c4800]

goroutine 20 [running]:
runtime.throw(0x85a95e, 0x2a)
/usr/local/go/src/runtime/panic.go:566 +0x95 fp=0xc4200236a8 sp=0xc420023688
runtime.sigpanic()
/usr/local/go/src/runtime/sigpanic_unix.go:12 +0x2cc fp=0xc420023700 sp=0xc4200236a8
created by runtime.gcBgMarkStartWorkers
/usr/local/go/src/runtime/mgc.go:1341 +0x98

goroutine 1 [runnable]:
go/build.(*Context).Import(0xa2ea40, 0xc4204257c9, 0x4, 0xc42045a380, 0x14, 0x4, 0xc4201aa480, 0xc420405c48, 0x47fdc4)
/usr/local/go/src/go/build/build.go:810 +0x15e8
main.loadImport(0xc4204257c9, 0x4, 0xc42045a380, 0x14, 0xc420428900, 0xc4204079e0, 0xc4204663c0, 0x1, 0x1, 0x1, ...)
/usr/local/go/src/cmd/go/pkg.go:365 +0x3b9
main.(*Package).load(0xc420428900, 0xc4204079e0, 0xc420339180, 0x0, 0x0, 0x4)
/usr/local/go/src/cmd/go/pkg.go:940 +0x12b6
main.loadImport(0xc420424f19, 0x2, 0xc42042c100, 0x15, 0xc420428480, 0xc4204079e0, 0xc420388b40, 0x2, 0x2, 0x1, ...)
/usr/local/go/src/cmd/go/pkg.go:374 +0x470
main.(*Package).load(0xc420428480, 0xc4204079e0, 0xc420338e00, 0x0, 0x0, 0x4)
/usr/local/go/src/cmd/go/pkg.go:940 +0x12b6
main.loadImport(0xc420194901, 0x3, 0xc4201a6540, 0x19, 0xc4201aa000, 0xc4204079e0, 0xc42017ef00, 0x1, 0x1, 0x1, ...)
/usr/local/go/src/cmd/go/pkg.go:374 +0x470
main.(*Package).load(0xc4201aa000, 0xc4204079e0, 0xc42017d500, 0x0, 0x0, 0xc)
/usr/local/go/src/cmd/go/pkg.go:940 +0x12b6
main.loadImport(0x848825, 0x1, 0xc42000c0c4, 0x19, 0x0, 0xc4204079e0, 0x0, 0x0, 0x0, 0x0, ...)
/usr/local/go/src/cmd/go/pkg.go:374 +0x470
main.loadPackage(0x848825, 0x1, 0xc4204079e0, 0x0)
/usr/local/go/src/cmd/go/pkg.go:1658 +0x28f
main.packagesAndErrors(0xc420194590, 0x1, 0x0, 0xc420194570, 0x20, 0x7f3a00)
/usr/local/go/src/cmd/go/pkg.go:1703 +0x337
main.packagesForBuild(0xc42000c330, 0x0, 0x0, 0x893cc8, 0xc420014140, 0x0)
/usr/local/go/src/cmd/go/pkg.go:1719 +0x75
main.runBuild(0xa27400, 0xc42000c330, 0x0, 0x0)
/usr/local/go/src/cmd/go/build.go:440 +0xd4
main.main()
/usr/local/go/src/cmd/go/main.go:181 +0x624

goroutine 17 [syscall, locked to thread]:
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:2086 +0x1

goroutine 5 [syscall]:
os/signal.signal_recv(0x0)
/usr/local/go/src/runtime/sigqueue.go:116 +0x157
os/signal.loop()
/usr/local/go/src/os/signal/signal_unix.go:22 +0x22
created by os/signal.init.1
/usr/local/go/src/os/signal/signal_unix.go:28 +0x41

--- SNIP ---

fatal error: workbuf is not empty

goroutine 5 [running]:
runtime.throw(0x8ffb1f, 0x14)
/usr/local/go/src/runtime/panic.go:566 +0x95 fp=0xc420029ea8 sp=0xc420029e88
runtime.(*workbuf).checkempty(0xc41ffc7bb5)
/usr/local/go/src/runtime/mgcwork.go:301 +0x4e fp=0xc420029ec8 sp=0xc420029ea8
runtime.putempty(0xc41ffc7bb5)
/usr/local/go/src/runtime/mgcwork.go:326 +0x2b fp=0xc420029ee8 sp=0xc420029ec8
runtime.(*gcWork).tryGet(0xc42001d228, 0xc42001d228)
/usr/local/go/src/runtime/mgcwork.go:157 +0xab fp=0xc420029f10 sp=0xc420029ee8
runtime.gcDrain(0xc42001d228, 0x6)
/usr/local/go/src/runtime/mgcmark.go:986 +0x2da fp=0xc420029f48 sp=0xc420029f10
runtime.gcBgMarkWorker(0xc42001c000)
/usr/local/go/src/runtime/mgc.go:1450 +0x4ad fp=0xc420029fb8 sp=0xc420029f48
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:2086 +0x1 fp=0xc420029fc0 sp=0xc420029fb8
created by runtime.gcBgMarkStartWorkers
/usr/local/go/src/runtime/mgc.go:1341 +0x98

goroutine 1 [runnable]:
cmd/compile/internal/gc.(*Node).SetVal(0xc420718240, 0x8e2d40, 0xc420708ed0)
/usr/local/go/src/cmd/compile/internal/gc/syntax.go:136
cmd/compile/internal/gc.nodlit(0x8e2d40, 0xc420708ed0, 0x0)
/usr/local/go/src/cmd/compile/internal/gc/const.go:1189 +0x63
cmd/compile/internal/gc.(*parser).operand(0xc4204147e0, 0xc4203b7500, 0x0)
/usr/local/go/src/cmd/compile/internal/gc/parser.go:1253 +0x359
cmd/compile/internal/gc.(*parser).pexpr(0xc4204147e0, 0xc400000000, 0x0)
/usr/local/go/src/cmd/compile/internal/gc/parser.go:1352 +0x58
cmd/compile/internal/gc.(*parser).uexpr(0xc4204147e0, 0x0)
/usr/local/go/src/cmd/compile/internal/gc/parser.go:1214 +0x1e5
cmd/compile/internal/gc.(*parser).bexpr(0xc4204147e0, 0x0, 0xc42045f8f0)
/usr/local/go/src/cmd/compile/internal/gc/parser.go:1105 +0x2f
cmd/compile/internal/gc.(*parser).expr(0xc4204147e0, 0x0)
/usr/local/go/src/cmd/compile/internal/gc/parser.go:1119 +0x3d
cmd/compile/internal/gc.(*parser).expr_list(0xc4204147e0, 0x0, 0x0, 0x0)
/usr/local/go/src/cmd/compile/internal/gc/parser.go:2660 +0x56
cmd/compile/internal/gc.(*parser).simple_stmt(0xc4204147e0, 0x8e0001, 0x0)
/usr/local/go/src/cmd/compile/internal/gc/parser.go:617 +0x992
cmd/compile/internal/gc.(*parser).stmt(0xc4204147e0, 0x0)
/usr/local/go/src/cmd/compile/internal/gc/parser.go:2510 +0x354
cmd/compile/internal/gc.(*parser).stmt_list(0xc4204147e0, 0xc4206fcfa0, 0x3, 0x4)
/usr/local/go/src/cmd/compile/internal/gc/parser.go:2593 +0x9e
cmd/compile/internal/gc.(*parser).fnbody(0xc4204147e0, 0x0, 0x0, 0x0)
/usr/local/go/src/cmd/compile/internal/gc/parser.go:2030 +0x78
cmd/compile/internal/gc.(*parser).xfndcl(0xc4204147e0, 0x0)
/usr/local/go/src/cmd/compile/internal/gc/parser.go:1861 +0x66
cmd/compile/internal/gc.(*parser).xdcl_list(0xc4204147e0, 0xc420086500, 0xc, 0x10)
/usr/local/go/src/cmd/compile/internal/gc/parser.go:2055 +0x3bc
cmd/compile/internal/gc.(*parser).file(0xc4204147e0)
/usr/local/go/src/cmd/compile/internal/gc/parser.go:272 +0x8d
cmd/compile/internal/gc.parse_file(0xc42006e240)
/usr/local/go/src/cmd/compile/internal/gc/parser.go:31 +0x54
cmd/compile/internal/gc.Main()
/usr/local/go/src/cmd/compile/internal/gc/main.go:334 +0x14f4
cmd/compile/internal/amd64.Main()
/usr/local/go/src/cmd/compile/internal/amd64/galign.go:93 +0x2fa
main.main()
/usr/local/go/src/cmd/compile/main.go:33 +0x2a3

--- SNIP ---

fatal error: greyobject: obj not pointer-aligned

goroutine 17 [running]:
runtime.throw(0x908394, 0x23)
/usr/local/go/src/runtime/panic.go:566 +0x95 fp=0xc4200245c0 sp=0xc4200245a0
runtime.greyobject(0xc41ffff0af, 0xb31cc0, 0x4210, 0xc400000000, 0xc4314316e0, 0xc42001d228, 0xc42001d228, 0x2)
/usr/local/go/src/runtime/mgcmark.go:1201 +0x392 fp=0xc4200245f0 sp=0xc4200245c0
runtime.scanblock(0xb31cc0, 0x2f010, 0x7fad31435b31, 0xc42001d228)
/usr/local/go/src/runtime/mgcmark.go:1105 +0x173 fp=0xc420024668 sp=0xc4200245f0
runtime.markrootBlock(0xb31cc0, 0x2f010, 0x7fad31435b31, 0xc42001d228, 0x0)
/usr/local/go/src/runtime/mgcmark.go:266 +0x7c fp=0xc420024698 sp=0xc420024668
runtime.markroot(0xc42001d228, 0xc400000004)
/usr/local/go/src/runtime/mgcmark.go:160 +0x145 fp=0xc420024710 sp=0xc420024698
runtime.gcDrain(0xc42001d228, 0x5)
/usr/local/go/src/runtime/mgcmark.go:963 +0xab fp=0xc420024748 sp=0xc420024710
runtime.gcBgMarkWorker(0xc42001c000)
/usr/local/go/src/runtime/mgc.go:1452 +0x1d1 fp=0xc4200247b8 sp=0xc420024748
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:2086 +0x1 fp=0xc4200247c0 sp=0xc4200247b8
created by runtime.gcBgMarkStartWorkers
/usr/local/go/src/runtime/mgc.go:1341 +0x98

goroutine 1 [runnable]:
strings.FieldsFunc(0xc42006e0c0, 0x59, 0x94e728, 0x59, 0x59, 0xc42006e0c0)
/usr/local/go/src/strings/strings.go:325 +0x127
strings.Fields(0xc42006e0c0, 0x59, 0x8fa019, 0xb, 0xffffffffffffff01)
/usr/local/go/src/strings/strings.go:301 +0x41
cmd/compile/internal/gc.(*lexer).getlinepragma(0xc42044a700, 0x2f)
/usr/local/go/src/cmd/compile/internal/gc/lex.go:899 +0x76f
cmd/compile/internal/gc.(*lexer).next(0xc42044a700)
/usr/local/go/src/cmd/compile/internal/gc/lex.go:327 +0x8a6
cmd/compile/internal/gc.(*parser).got(0xc42044a700, 0xc40000003b, 0x1)
/usr/local/go/src/cmd/compile/internal/gc/parser.go:56 +0x36
cmd/compile/internal/gc.(*parser).xdcl_list(0xc42044a700, 0xc4203e0200, 0x13, 0x20)
/usr/local/go/src/cmd/compile/internal/gc/parser.go:2072 +0x1ef
cmd/compile/internal/gc.(*parser).file(0xc42044a700)
/usr/local/go/src/cmd/compile/internal/gc/parser.go:272 +0x8d
cmd/compile/internal/gc.parse_file(0xc4203cc120)
/usr/local/go/src/cmd/compile/internal/gc/parser.go:31 +0x54
cmd/compile/internal/gc.Main()
/usr/local/go/src/cmd/compile/internal/gc/main.go:334 +0x14f4
cmd/compile/internal/amd64.Main()
/usr/local/go/src/cmd/compile/internal/amd64/galign.go:93 +0x2fa
main.main()
/usr/local/go/src/cmd/compile/main.go:33 +0x2a3

--- SNIP ---

panic: runtime error: index out of range
fatal error: panic holding locks

goroutine 5 [running]:
runtime.throw(0x8feeab, 0x13)
/usr/local/go/src/runtime/panic.go:566 +0x95 fp=0xc420029e50 sp=0xc420029e30
panic(0x8b82c0, 0xc4200101a0)
/usr/local/go/src/runtime/panic.go:420 +0x495 fp=0xc420029ee0 sp=0xc420029e50
runtime.panicindex()
/usr/local/go/src/runtime/panic.go:27 +0x6d fp=0xc420029f10 sp=0xc420029ee0
runtime.gcDrain(0xc42001d228, 0x5)
/usr/local/go/src/runtime/mgcmark.go:984 +0x318 fp=0xc420029f48 sp=0xc420029f10
runtime.gcBgMarkWorker(0xc42001c000)
/usr/local/go/src/runtime/mgc.go:1452 +0x1d1 fp=0xc420029fb8 sp=0xc420029f48
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:2086 +0x1 fp=0xc420029fc0 sp=0xc420029fb8
created by runtime.gcBgMarkStartWorkers
/usr/local/go/src/runtime/mgc.go:1341 +0x98

goroutine 1 [runnable]:
cmd/compile/internal/gc.(*Node).SetVal(0xc42045e2d0, 0x8a6320, 0xc420462040)
/usr/local/go/src/cmd/compile/internal/gc/syntax.go:136
cmd/compile/internal/gc.(*importer).field(0xc4200c6380, 0xc42045e1b0)
/usr/local/go/src/cmd/compile/internal/gc/bimport.go:565 +0x14b
cmd/compile/internal/gc.(*importer).fieldList(0xc4200c6380, 0xffffffffffffff16, 0xc420460140, 0x7f9173349080)
/usr/local/go/src/cmd/compile/internal/gc/bimport.go:536 +0x8f
cmd/compile/internal/gc.(*importer).typ(0xc4200c6380, 0xc4204600a0)
/usr/local/go/src/cmd/compile/internal/gc/bimport.go:483 +0xc77
cmd/compile/internal/gc.(*importer).typ(0xc4200c6380, 0xc420464000)
/usr/local/go/src/cmd/compile/internal/gc/bimport.go:407 +0x148
cmd/compile/internal/gc.(*importer).field(0xc4200c6380, 0x2)
/usr/local/go/src/cmd/compile/internal/gc/bimport.go:546 +0x55
cmd/compile/internal/gc.(*importer).fieldList(0xc4200c6380, 0xffffffffffffff16, 0xc420460050, 0x7f9173349080)
/usr/local/go/src/cmd/compile/internal/gc/bimport.go:536 +0x8f
cmd/compile/internal/gc.(*importer).typ(0xc4200c6380, 0xc42043e000)
/usr/local/go/src/cmd/compile/internal/gc/bimport.go:483 +0xc77
cmd/compile/internal/gc.(*importer).typ(0xc4200c6380, 0x0)
/usr/local/go/src/cmd/compile/internal/gc/bimport.go:407 +0x148
cmd/compile/internal/gc.(*importer).obj(0xc4200c6380, 0xfffffffffffffffd)
/usr/local/go/src/cmd/compile/internal/gc/bimport.go:282 +0x967
cmd/compile/internal/gc.Import(0xc42005a300)
/usr/local/go/src/cmd/compile/internal/gc/bimport.go:97 +0x3b2
cmd/compile/internal/gc.importfile(0xc420400800, 0x0, 0x0, 0x0)
/usr/local/go/src/cmd/compile/internal/gc/main.go:843 +0x9b1
cmd/compile/internal/gc.(*parser).importdcl(0xc4204007e0)
/usr/local/go/src/cmd/compile/internal/gc/parser.go:341 +0x178
cmd/compile/internal/gc.(*parser).import_(0xc4204007e0)
/usr/local/go/src/cmd/compile/internal/gc/parser.go:307 +0xd1
cmd/compile/internal/gc.(*parser).file(0xc4204007e0)
/usr/local/go/src/cmd/compile/internal/gc/parser.go:268 +0x5e
cmd/compile/internal/gc.parse_file(0xc42005a240)
/usr/local/go/src/cmd/compile/internal/gc/parser.go:31 +0x54
cmd/compile/internal/gc.Main()
/usr/local/go/src/cmd/compile/internal/gc/main.go:334 +0x14f4
cmd/compile/internal/amd64.Main()
/usr/local/go/src/cmd/compile/internal/amd64/galign.go:93 +0x2fa
main.main()
/usr/local/go/src/cmd/compile/main.go:33 +0x2a3

@quentinmit quentinmit added this to the Go1.8 milestone Nov 27, 2016

@quentinmit

This comment has been minimized.

Copy link
Contributor

commented Nov 27, 2016

/cc @aclements

Any chance this is related to the other random runtime state corruption we've been seeing lately?

@aclements

This comment has been minimized.

Copy link
Member

commented Dec 13, 2016

While not impossible, I find it quite unlikely that this is Go's fault, and more likely that this is a kernel or hardware bug. The primary reason I think that is because it depends on a kernel configuration that should be essentially invisible to user space.

There are subtler reasons I don't think it's the Go runtime, too. Several of the crashes you reported indicate corruption in type-stable memory, which makes it unlikely that the corruption is the result of a dangling pointer on the Go side because the memory at those addresses simply never gets reused. Furthermore, all of the crashes are the results of reads. If something were spraying the address space with bad writes, it would either have to be following a dangling pointer to mapped memory, or I would expect to catch it in the act of writing to some unmapped memory and causing a SIGSEGV of its own.

@aclements

This comment has been minimized.

Copy link
Member

commented Dec 13, 2016

I'm not quite sure how to proceed on this bug. I'm going to move it to unplanned until we get more reports or more actionable details.

@1-z, are you able to reproduce this on different hardware? That would at least rule out the possibility that this is a hardware bug.

@aclements aclements modified the milestones: Unplanned, Go1.8 Dec 13, 2016

@1-z

This comment has been minimized.

Copy link
Author

commented Dec 13, 2016

I have a few ECC-enabled machines I can test this on with similar software configurations. I will attempt to reproduce this issue there.

@1-z

This comment has been minimized.

Copy link
Author

commented Dec 15, 2016

I believe this issue can be closed. After testing on a number of different hardware configurations using Kernel 4.8.13, I can no longer reproduce this on any machine. I don't believe it was hardware, but rather a kernel issue that is now resolved. If I do discover more useful information I will comment here.

@1-z 1-z closed this Dec 15, 2016

@aclements

This comment has been minimized.

Copy link
Member

commented Dec 15, 2016

Thanks for the update!

@golang golang locked and limited conversation to collaborators Dec 15, 2017

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
You can’t perform that action at this time.