runtime: corrupt binary export data seen after signal preemption CL

[mvdan]

$ go version
go version devel +d2c039fb21 Sun Nov 3 01:44:46 2019 +0000 linux/amd64
$ go env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/mvdan/.cache/go-build"
GOENV="/home/mvdan/.config/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GONOPROXY="brank.as/*"
GONOSUMDB="brank.as/*"
GOOS="linux"
GOPATH="/home/mvdan/go"
GOPRIVATE="brank.as/*"
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/home/mvdan/tip"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/home/mvdan/tip/pkg/tool/linux_amd64"
GCCGO="gccgo"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/home/mvdan/src/gio/cmd/go.mod"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build819987201=/tmp/go-build -gno-record-gcc-switches"

After building Go from master, I sometimes see errors like:

$ go test -race
# encoding/json
vet: /home/mvdan/tip/src/encoding/json/decode.go:13:2: could not import fmt (cannot import "fmt" (unknown bexport format version -1 ("\x80\x16\x13\x00\xc0\x00\x00\x00\x80\x17\x13\x00\xc0\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x06format\x01a\x00\x04esc:\x05esc:\x02\x18$GOROOT/src/fmt/print.go\x05Write\x01b\x01n\x03err\x05Width\x03wid\x02ok\tPrecision\x04prec\x04Flag\x01c\x06Format\x01f\x05State\x06String\bGoString\x01w\x06Writer\x02io")), possibly version skew - reinstall package)
PASS
ok      gioui.org/cmd/gogio     4.559s

Here's another crash from earlier today, with a slightly modified (and freshly built) Go master - you can see the error mentions a different std package:

$ go test
# mime
vet: /home/mvdan/tip/src/mime/encodedword.go:12:2: could not import io (cannot import "io" (unknown bexport format version -1 ("DX\xcdq㦔d_\xbf\x97\xa64h\xf7\x8f\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00d\x01p\x01n\x03err\x05Write\x05Close\x04Seek\x06offset\x06whence\x06Reader\x06Writer\x06Closer\x06Seeker\bReadFrom\x01r\aWriteTo\x01w\x06ReadAt\x03off\aWriteAt\bReadByte")), possibly version skew - reinstall package)
PASS
ok      gioui.org/cmd/gogio     7.199s

@heschik correctly points out that this could be a bad version of vet in play, since the bexport format has been replaced with the iexport. However, I already nuked my $GOBIN, and go test -x seems to be running /home/mvdan/tip/pkg/tool/linux_amd64/vet, which is freshly built.

Usually I'd assume this is an issue with my setup, but I can't find anything wrong with it, and I've only started seeing these errors today.

/cc @mdempsky @griesemer @alandonovan

[mvdan]

I get many errors like it when I attempt to test my Go installation, too - via bash run.bash --no-rebuild.

[mvdan]

I built 50 commits behind the current master (cc4b824) and bash run.bash --no-rebuild succeeds without a problem. So perhaps there was a regression in the last 50 commits; will keep digging.

[siebenmann]

I'm seeing this error on all.bash builds at the current git tip, at varying and erratic places through the tests. For me this happens in a completely clean environment (as far as I can arrange it) on Fedora 30 64-bit Linux:

$ mkdir /tmp/gb
$ env -i HOME=/tmp/gb PATH=/usr/bin:/usr/sbin ./all.bash
Building Go cmd/dist using /usr/lib/golang. (go1.12.10 linux/amd64)
Building Go toolchain1 using /usr/lib/golang.
Building Go bootstrap cmd/go (go_bootstrap) using Go toolchain1.
Building Go toolchain2 using go_bootstrap and Go toolchain1.
Building Go toolchain3 using go_bootstrap and Go toolchain2.
Building packages and commands for linux/amd64.

##### Testing packages.
# sync
vet: sync/pool.go:9:2: could not import runtime (cannot import "runtime" (unknown bexport format version -1 ("77;>$history;uma\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00ler.go\x02gc\x1e$GOROOT/src/runtime/cpuprof.go\x02hz\x00\x1eCPUProfile no longer available\x1c$GOROOT/src/runtime/debug.go\x01n\x04ncpu\x06gcount\x1c$GOROOT/src/runtime/error.go\fRuntimeError")), possibly version skew - reinstall package)
[...]

Which parts of the tests and which packages it fails on varies, but it looks like it's always vet failing.

Re-running all.bash sometimes succeeds, but not always. I just had such a rebuild attempt fail with a different error in tests, although some other tests in this run also reported vet failures:

##### Testing packages.
# crypto/subtle.test
fatal error: index out of range

runtime stack:
runtime.throw(0xe30bc2, 0x12)
        /data/code/go-lang/go/src/runtime/panic.go:1045 +0x72
runtime.panicCheck1(0x41fd9f, 0xe30bc2, 0x12)
        /data/code/go-lang/go/src/runtime/panic.go:34 +0xd9
runtime.goPanicIndexU(0x1b9a789ccb, 0x400000)
        /data/code/go-lang/go/src/runtime/panic.go:91 +0x44
runtime.heapBitsForAddr(...)
        /data/code/go-lang/go/src/runtime/mbitmap.go:340
runtime.scanobject(0x6e6962732f727375, 0xc000021270)
        /data/code/go-lang/go/src/runtime/mgcmark.go:1191 +0x39f
runtime.gcDrain(0xc000021270, 0x3)
        /data/code/go-lang/go/src/runtime/mgcmark.go:1028 +0x1fb
runtime.gcBgMarkWorker.func2()
        /data/code/go-lang/go/src/runtime/mgc.go:1926 +0x80
runtime.systemstack(0x0)
        /data/code/go-lang/go/src/runtime/asm_amd64.s:370 +0x66
runtime.mstart()
        /data/code/go-lang/go/src/runtime/proc.go:1069

goroutine 19 [GC worker (idle)]:
runtime.systemstack_switch()
        /data/code/go-lang/go/src/runtime/asm_amd64.s:330 fp=0xc00003c760 sp=0xc00003c758 pc=0x45da10
runtime.gcBgMarkWorker(0xc000020000)
        /data/code/go-lang/go/src/runtime/mgc.go:1913 +0x1be fp=0xc00003c7d8 sp=0xc00003c760 pc=0x41b4ce
runtime.goexit()
        /data/code/go-lang/go/src/runtime/asm_amd64.s:1375 +0x1 fp=0xc00003c7e0 sp=0xc00003c7d8 pc=0x45f9a1
created by runtime.gcBgMarkStartWorkers
        /data/code/go-lang/go/src/runtime/mgc.go:1807 +0x77

goroutine 1 [semacquire]:
sync.runtime_Semacquire(0xc000588488)
        /data/code/go-lang/go/src/runtime/sema.go:56 +0x42
sync.(*WaitGroup).Wait(0xc000588480)
        /data/code/go-lang/go/src/sync/waitgroup.go:130 +0x64
cmd/compile/internal/gc.compileFunctions()
        /data/code/go-lang/go/src/cmd/compile/internal/gc/pgen.go:373 +0x1ce
cmd/compile/internal/gc.Main(0xe4ae98)
        /data/code/go-lang/go/src/cmd/compile/internal/gc/main.go:709 +0x3280
main.main()
        /data/code/go-lang/go/src/cmd/compile/main.go:50 +0xac

goroutine 39 [runnable]:
cmd/compile/internal/ssa.SparseTree.numberBlock(0xc000441200, 0x2, 0x2, 0xc000486c30, 0x1, 0xc00008fd70)
        /data/code/go-lang/go/src/cmd/compile/internal/ssa/sparsetree.go:147 +0x102
cmd/compile/internal/ssa.newSparseTree(0xc0002e0580, 0xc00008fd70, 0x2, 0x2, 0xdee4c0, 0x1, 0xc00042cb10)
        /data/code/go-lang/go/src/cmd/compile/internal/ssa/sparsetree.go:69 +0x18b
cmd/compile/internal/ssa.(*Func).sdom(0xc0002e0580, 0xc00042cb10, 0xd, 0xc00040d100)
        /data/code/go-lang/go/src/cmd/compile/internal/ssa/func.go:652 +0x95
cmd/compile/internal/ssa.cse(0xc0002e0580)
        /data/code/go-lang/go/src/cmd/compile/internal/ssa/cse.go:158 +0xbfa
cmd/compile/internal/ssa.Compile(0xc0002e0580)
        /data/code/go-lang/go/src/cmd/compile/internal/ssa/compile.go:92 +0x9a5
cmd/compile/internal/gc.buildssa(0xc0002e02c0, 0x0, 0x0)
        /data/code/go-lang/go/src/cmd/compile/internal/gc/ssa.go:444 +0xcd8
cmd/compile/internal/gc.compileSSA(0xc0002e02c0, 0x0)
        /data/code/go-lang/go/src/cmd/compile/internal/gc/pgen.go:298 +0x5d
cmd/compile/internal/gc.compileFunctions.func2(0xc0005a39e0, 0xc000588480, 0x0)
        /data/code/go-lang/go/src/cmd/compile/internal/gc/pgen.go:363 +0x49
created by cmd/compile/internal/gc.compileFunctions
        /data/code/go-lang/go/src/cmd/compile/internal/gc/pgen.go:361 +0x128

goroutine 40 [runnable]:
cmd/compile/internal/types.(*Sym).Linksym(0xc0003b24e0, 0xc00001e0c0)
        /data/code/go-lang/go/src/cmd/compile/internal/types/sym.go:79 +0x1d4
cmd/compile/internal/gc.(*state).addr(0xc000010120, 0xc0003b0ff0, 0xc000000800, 0x0)
        /data/code/go-lang/go/src/cmd/compile/internal/gc/ssa.go:4580 +0x275
cmd/compile/internal/gc.(*state).expr(0xc000010120, 0xc0003b0ff0, 0x0)
        /data/code/go-lang/go/src/cmd/compile/internal/gc/ssa.go:2007 +0x143
cmd/compile/internal/gc.(*state).stmt(0xc000010120, 0xc0005c6780)
        /data/code/go-lang/go/src/cmd/compile/internal/gc/ssa.go:1239 +0x234
cmd/compile/internal/gc.(*state).stmtList(0xc000010120, 0xc0005be140)
        /data/code/go-lang/go/src/cmd/compile/internal/gc/ssa.go:1018 +0x58
cmd/compile/internal/gc.(*state).stmt(0xc000010120, 0xc0005c6900)
        /data/code/go-lang/go/src/cmd/compile/internal/gc/ssa.go:1040 +0x15ce
cmd/compile/internal/gc.(*state).stmtList(0xc000010120, 0xc0005be060)
        /data/code/go-lang/go/src/cmd/compile/internal/gc/ssa.go:1018 +0x58
cmd/compile/internal/gc.buildssa(0xc0002e0420, 0x1, 0x0)
        /data/code/go-lang/go/src/cmd/compile/internal/gc/ssa.go:426 +0xc39
cmd/compile/internal/gc.compileSSA(0xc0002e0420, 0x1)
        /data/code/go-lang/go/src/cmd/compile/internal/gc/pgen.go:298 +0x5d
cmd/compile/internal/gc.compileFunctions.func2(0xc0005a39e0, 0xc000588480, 0x1)
        /data/code/go-lang/go/src/cmd/compile/internal/gc/pgen.go:363 +0x49
created by cmd/compile/internal/gc.compileFunctions
        /data/code/go-lang/go/src/cmd/compile/internal/gc/pgen.go:361 +0x128

goroutine 41 [runnable]:
cmd/compile/internal/gc.compileFunctions.func2(0xc0005a39e0, 0xc000588480, 0x2)
        /data/code/go-lang/go/src/cmd/compile/internal/gc/pgen.go:361
created by cmd/compile/internal/gc.compileFunctions
        /data/code/go-lang/go/src/cmd/compile/internal/gc/pgen.go:361 +0x128

goroutine 42 [runnable]:
cmd/compile/internal/gc.compileFunctions.func2(0xc0005a39e0, 0xc000588480, 0x3)
        /data/code/go-lang/go/src/cmd/compile/internal/gc/pgen.go:361
created by cmd/compile/internal/gc.compileFunctions
        /data/code/go-lang/go/src/cmd/compile/internal/gc/pgen.go:361 +0x128
ok      archive/tar     0.031s
ok      archive/zip     0.025s

[mdempsky]

vet: sync/pool.go:9:2: could not import runtime (cannot import "runtime" (unknown bexport format version -1 ("77;>$history;uma\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00ler.go\x02gc\x1e$GOROOT/src/runtime/cpuprof.go\x02hz\x00\x1eCPUProfile no longer available\x1c$GOROOT/src/runtime/debug.go\x01n\x04ncpu\x06gcount\x1c$GOROOT/src/runtime/error.go\fRuntimeError")), possibly version skew - reinstall package)

The trailing part of that looks like valid iexport data, except the first 32 bytes have been corrupted.

This seems like memory corruption, either from miscompilation or a GC issue.

[mdempsky]

In all three of the examples, the first 32 bytes of the buffer were corrupted.

[mdempsky]

BTW, the thing that makes me suspect memory corruption within cmd/compile is that "unknown bexport format" means that the "$$B\n" signature was written out correctly, but then the next 32 byte were garbage.

Also, cmd/compile explicitly flushes its write buffer after writing "$$B\n" and before writing the indexed export data.

Finally, this data isn't specially aligned at all within the file, so it's not like it could be generic file-level corruption.

Not to mention the index out of range error within the runtime during compilation seems bad.

[mdempsky]

@mvdan @siebenmann Can you two describe your test environments in more detail so maybe we can try to identify more commonalities?

E.g., it looks like you're both using linux/amd64. @siebenmann mentioned Fedora 30; @mvdan what Linux distro and version are you using?

Also, how many cores?

[siebenmann]

The machine I'm seeing this on is an i7-8700K (not overclocked) with 32 GB of RAM and HyperThreading still enabled, so a 6/12 cores/threads machine. At the moment it has Fedora's kernel 5.3.7-200.fc30.x86_64. I can't seem to reproduce this on a machine running Ubuntu 18.04 (kernel 4.15.0-52-generic) with a Xeon E5-2680 CPU with HyperThreading off (8 cores) and 64 GB of RAM, nor on another Ubuntu 18.04 machine with the same kernel but a Ryzen 7 1800X CPU (again, not overclocked) with 8 cores (it has HT off) and 32 GB of RAM.

[mvdan]

I am on a Thinkpad with a i5-8350U, HT left on (4 real cores, 8 total), and 16GiB of RAM. I run Arch Linux, with its current 5.3.8 Linux kernel.

Aside from that my Go setup is a pretty standard build from source, using 1.13.4 to bootstrap. I don't have any fancy go env vars other than GOPRIVATE.

[siebenmann]

I've now reproduced this on another Fedora 30 linux/amd64 machine, also with kernel 5.3.7-200.fc30.x86_64 but this time with a Ryzen 1800X (not overclocked, 8/16 cores) with 32 GB.

I've git bisected this and on both of my Fedora 30 machines, the commit where the failures start is 62e53b7, 'runtime: use signals to preempt Gs for suspendG', which is targeted at #10958 and #24543. I think this makes this issue CC @aclements.

[mdempsky]

Linux kernel version-specific signal handling behavior sounds fuuuuun.

[mvdan]

At least it's good that enough people test tip, so that this was caught at the start of the freeze :)

I'm happy to help debug this further, if the runtime maintainers can't reproduce this themselves.

[mdempsky]

@mvdan Oh yeah, definitely. I'm glad you and @siebenmann both reported the issue and have been able to reproduce it reliably enough to track down a culprit CL and identify some likely commonality between test setups. (I'll check the Linux kernel version I was using later, and also give it a shot on my personal laptop running Fedora 31.)

I'm just a little nervous because this is already the second issue discovered in the runtime preemption feature that landed last week.