Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

runtime: corrupt binary export data seen after signal preemption CL #35326

Open
mvdan opened this issue Nov 3, 2019 · 22 comments

Comments

@mvdan
Copy link
Member

@mvdan mvdan commented Nov 3, 2019

$ go version
go version devel +d2c039fb21 Sun Nov 3 01:44:46 2019 +0000 linux/amd64
$ go env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/mvdan/.cache/go-build"
GOENV="/home/mvdan/.config/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GONOPROXY="brank.as/*"
GONOSUMDB="brank.as/*"
GOOS="linux"
GOPATH="/home/mvdan/go"
GOPRIVATE="brank.as/*"
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/home/mvdan/tip"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/home/mvdan/tip/pkg/tool/linux_amd64"
GCCGO="gccgo"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/home/mvdan/src/gio/cmd/go.mod"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build819987201=/tmp/go-build -gno-record-gcc-switches"

After building Go from master, I sometimes see errors like:

$ go test -race
# encoding/json
vet: /home/mvdan/tip/src/encoding/json/decode.go:13:2: could not import fmt (cannot import "fmt" (unknown bexport format version -1 ("\x80\x16\x13\x00\xc0\x00\x00\x00\x80\x17\x13\x00\xc0\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x06format\x01a\x00\x04esc:\x05esc:\x02\x18$GOROOT/src/fmt/print.go\x05Write\x01b\x01n\x03err\x05Width\x03wid\x02ok\tPrecision\x04prec\x04Flag\x01c\x06Format\x01f\x05State\x06String\bGoString\x01w\x06Writer\x02io")), possibly version skew - reinstall package)
PASS
ok      gioui.org/cmd/gogio     4.559s

Here's another crash from earlier today, with a slightly modified (and freshly built) Go master - you can see the error mentions a different std package:

$ go test
# mime
vet: /home/mvdan/tip/src/mime/encodedword.go:12:2: could not import io (cannot import "io" (unknown bexport format version -1 ("DX\xcdq㦔d_\xbf\x97\xa64h\xf7\x8f\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00d\x01p\x01n\x03err\x05Write\x05Close\x04Seek\x06offset\x06whence\x06Reader\x06Writer\x06Closer\x06Seeker\bReadFrom\x01r\aWriteTo\x01w\x06ReadAt\x03off\aWriteAt\bReadByte")), possibly version skew - reinstall package)
PASS
ok      gioui.org/cmd/gogio     7.199s

@heschik correctly points out that this could be a bad version of vet in play, since the bexport format has been replaced with the iexport. However, I already nuked my $GOBIN, and go test -x seems to be running /home/mvdan/tip/pkg/tool/linux_amd64/vet, which is freshly built.

Usually I'd assume this is an issue with my setup, but I can't find anything wrong with it, and I've only started seeing these errors today.

/cc @mdempsky @griesemer @alandonovan

@mvdan mvdan added this to the Go1.14 milestone Nov 3, 2019
@mvdan

This comment has been minimized.

Copy link
Member Author

@mvdan mvdan commented Nov 3, 2019

I get many errors like it when I attempt to test my Go installation, too - via bash run.bash --no-rebuild.

@mvdan

This comment has been minimized.

Copy link
Member Author

@mvdan mvdan commented Nov 3, 2019

I built 50 commits behind the current master (cc4b824) and bash run.bash --no-rebuild succeeds without a problem. So perhaps there was a regression in the last 50 commits; will keep digging.

@siebenmann

This comment has been minimized.

Copy link

@siebenmann siebenmann commented Nov 4, 2019

I'm seeing this error on all.bash builds at the current git tip, at varying and erratic places through the tests. For me this happens in a completely clean environment (as far as I can arrange it) on Fedora 30 64-bit Linux:

$ mkdir /tmp/gb
$ env -i HOME=/tmp/gb PATH=/usr/bin:/usr/sbin ./all.bash
Building Go cmd/dist using /usr/lib/golang. (go1.12.10 linux/amd64)
Building Go toolchain1 using /usr/lib/golang.
Building Go bootstrap cmd/go (go_bootstrap) using Go toolchain1.
Building Go toolchain2 using go_bootstrap and Go toolchain1.
Building Go toolchain3 using go_bootstrap and Go toolchain2.
Building packages and commands for linux/amd64.

##### Testing packages.
# sync
vet: sync/pool.go:9:2: could not import runtime (cannot import "runtime" (unknown bexport format version -1 ("77;>$history;uma\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00ler.go\x02gc\x1e$GOROOT/src/runtime/cpuprof.go\x02hz\x00\x1eCPUProfile no longer available\x1c$GOROOT/src/runtime/debug.go\x01n\x04ncpu\x06gcount\x1c$GOROOT/src/runtime/error.go\fRuntimeError")), possibly version skew - reinstall package)
[...]

Which parts of the tests and which packages it fails on varies, but it looks like it's always vet failing.

Re-running all.bash sometimes succeeds, but not always. I just had such a rebuild attempt fail with a different error in tests, although some other tests in this run also reported vet failures:

##### Testing packages.
# crypto/subtle.test
fatal error: index out of range

runtime stack:
runtime.throw(0xe30bc2, 0x12)
        /data/code/go-lang/go/src/runtime/panic.go:1045 +0x72
runtime.panicCheck1(0x41fd9f, 0xe30bc2, 0x12)
        /data/code/go-lang/go/src/runtime/panic.go:34 +0xd9
runtime.goPanicIndexU(0x1b9a789ccb, 0x400000)
        /data/code/go-lang/go/src/runtime/panic.go:91 +0x44
runtime.heapBitsForAddr(...)
        /data/code/go-lang/go/src/runtime/mbitmap.go:340
runtime.scanobject(0x6e6962732f727375, 0xc000021270)
        /data/code/go-lang/go/src/runtime/mgcmark.go:1191 +0x39f
runtime.gcDrain(0xc000021270, 0x3)
        /data/code/go-lang/go/src/runtime/mgcmark.go:1028 +0x1fb
runtime.gcBgMarkWorker.func2()
        /data/code/go-lang/go/src/runtime/mgc.go:1926 +0x80
runtime.systemstack(0x0)
        /data/code/go-lang/go/src/runtime/asm_amd64.s:370 +0x66
runtime.mstart()
        /data/code/go-lang/go/src/runtime/proc.go:1069

goroutine 19 [GC worker (idle)]:
runtime.systemstack_switch()
        /data/code/go-lang/go/src/runtime/asm_amd64.s:330 fp=0xc00003c760 sp=0xc00003c758 pc=0x45da10
runtime.gcBgMarkWorker(0xc000020000)
        /data/code/go-lang/go/src/runtime/mgc.go:1913 +0x1be fp=0xc00003c7d8 sp=0xc00003c760 pc=0x41b4ce
runtime.goexit()
        /data/code/go-lang/go/src/runtime/asm_amd64.s:1375 +0x1 fp=0xc00003c7e0 sp=0xc00003c7d8 pc=0x45f9a1
created by runtime.gcBgMarkStartWorkers
        /data/code/go-lang/go/src/runtime/mgc.go:1807 +0x77

goroutine 1 [semacquire]:
sync.runtime_Semacquire(0xc000588488)
        /data/code/go-lang/go/src/runtime/sema.go:56 +0x42
sync.(*WaitGroup).Wait(0xc000588480)
        /data/code/go-lang/go/src/sync/waitgroup.go:130 +0x64
cmd/compile/internal/gc.compileFunctions()
        /data/code/go-lang/go/src/cmd/compile/internal/gc/pgen.go:373 +0x1ce
cmd/compile/internal/gc.Main(0xe4ae98)
        /data/code/go-lang/go/src/cmd/compile/internal/gc/main.go:709 +0x3280
main.main()
        /data/code/go-lang/go/src/cmd/compile/main.go:50 +0xac

goroutine 39 [runnable]:
cmd/compile/internal/ssa.SparseTree.numberBlock(0xc000441200, 0x2, 0x2, 0xc000486c30, 0x1, 0xc00008fd70)
        /data/code/go-lang/go/src/cmd/compile/internal/ssa/sparsetree.go:147 +0x102
cmd/compile/internal/ssa.newSparseTree(0xc0002e0580, 0xc00008fd70, 0x2, 0x2, 0xdee4c0, 0x1, 0xc00042cb10)
        /data/code/go-lang/go/src/cmd/compile/internal/ssa/sparsetree.go:69 +0x18b
cmd/compile/internal/ssa.(*Func).sdom(0xc0002e0580, 0xc00042cb10, 0xd, 0xc00040d100)
        /data/code/go-lang/go/src/cmd/compile/internal/ssa/func.go:652 +0x95
cmd/compile/internal/ssa.cse(0xc0002e0580)
        /data/code/go-lang/go/src/cmd/compile/internal/ssa/cse.go:158 +0xbfa
cmd/compile/internal/ssa.Compile(0xc0002e0580)
        /data/code/go-lang/go/src/cmd/compile/internal/ssa/compile.go:92 +0x9a5
cmd/compile/internal/gc.buildssa(0xc0002e02c0, 0x0, 0x0)
        /data/code/go-lang/go/src/cmd/compile/internal/gc/ssa.go:444 +0xcd8
cmd/compile/internal/gc.compileSSA(0xc0002e02c0, 0x0)
        /data/code/go-lang/go/src/cmd/compile/internal/gc/pgen.go:298 +0x5d
cmd/compile/internal/gc.compileFunctions.func2(0xc0005a39e0, 0xc000588480, 0x0)
        /data/code/go-lang/go/src/cmd/compile/internal/gc/pgen.go:363 +0x49
created by cmd/compile/internal/gc.compileFunctions
        /data/code/go-lang/go/src/cmd/compile/internal/gc/pgen.go:361 +0x128

goroutine 40 [runnable]:
cmd/compile/internal/types.(*Sym).Linksym(0xc0003b24e0, 0xc00001e0c0)
        /data/code/go-lang/go/src/cmd/compile/internal/types/sym.go:79 +0x1d4
cmd/compile/internal/gc.(*state).addr(0xc000010120, 0xc0003b0ff0, 0xc000000800, 0x0)
        /data/code/go-lang/go/src/cmd/compile/internal/gc/ssa.go:4580 +0x275
cmd/compile/internal/gc.(*state).expr(0xc000010120, 0xc0003b0ff0, 0x0)
        /data/code/go-lang/go/src/cmd/compile/internal/gc/ssa.go:2007 +0x143
cmd/compile/internal/gc.(*state).stmt(0xc000010120, 0xc0005c6780)
        /data/code/go-lang/go/src/cmd/compile/internal/gc/ssa.go:1239 +0x234
cmd/compile/internal/gc.(*state).stmtList(0xc000010120, 0xc0005be140)
        /data/code/go-lang/go/src/cmd/compile/internal/gc/ssa.go:1018 +0x58
cmd/compile/internal/gc.(*state).stmt(0xc000010120, 0xc0005c6900)
        /data/code/go-lang/go/src/cmd/compile/internal/gc/ssa.go:1040 +0x15ce
cmd/compile/internal/gc.(*state).stmtList(0xc000010120, 0xc0005be060)
        /data/code/go-lang/go/src/cmd/compile/internal/gc/ssa.go:1018 +0x58
cmd/compile/internal/gc.buildssa(0xc0002e0420, 0x1, 0x0)
        /data/code/go-lang/go/src/cmd/compile/internal/gc/ssa.go:426 +0xc39
cmd/compile/internal/gc.compileSSA(0xc0002e0420, 0x1)
        /data/code/go-lang/go/src/cmd/compile/internal/gc/pgen.go:298 +0x5d
cmd/compile/internal/gc.compileFunctions.func2(0xc0005a39e0, 0xc000588480, 0x1)
        /data/code/go-lang/go/src/cmd/compile/internal/gc/pgen.go:363 +0x49
created by cmd/compile/internal/gc.compileFunctions
        /data/code/go-lang/go/src/cmd/compile/internal/gc/pgen.go:361 +0x128

goroutine 41 [runnable]:
cmd/compile/internal/gc.compileFunctions.func2(0xc0005a39e0, 0xc000588480, 0x2)
        /data/code/go-lang/go/src/cmd/compile/internal/gc/pgen.go:361
created by cmd/compile/internal/gc.compileFunctions
        /data/code/go-lang/go/src/cmd/compile/internal/gc/pgen.go:361 +0x128

goroutine 42 [runnable]:
cmd/compile/internal/gc.compileFunctions.func2(0xc0005a39e0, 0xc000588480, 0x3)
        /data/code/go-lang/go/src/cmd/compile/internal/gc/pgen.go:361
created by cmd/compile/internal/gc.compileFunctions
        /data/code/go-lang/go/src/cmd/compile/internal/gc/pgen.go:361 +0x128
ok      archive/tar     0.031s
ok      archive/zip     0.025s
@mdempsky

This comment has been minimized.

Copy link
Member

@mdempsky mdempsky commented Nov 4, 2019

vet: sync/pool.go:9:2: could not import runtime (cannot import "runtime" (unknown bexport format version -1 ("77;>$history;uma\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00ler.go\x02gc\x1e$GOROOT/src/runtime/cpuprof.go\x02hz\x00\x1eCPUProfile no longer available\x1c$GOROOT/src/runtime/debug.go\x01n\x04ncpu\x06gcount\x1c$GOROOT/src/runtime/error.go\fRuntimeError")), possibly version skew - reinstall package)

The trailing part of that looks like valid iexport data, except the first 32 bytes have been corrupted.

This seems like memory corruption, either from miscompilation or a GC issue.

@mdempsky

This comment has been minimized.

Copy link
Member

@mdempsky mdempsky commented Nov 4, 2019

In all three of the examples, the first 32 bytes of the buffer were corrupted.

@mdempsky

This comment has been minimized.

Copy link
Member

@mdempsky mdempsky commented Nov 4, 2019

BTW, the thing that makes me suspect memory corruption within cmd/compile is that "unknown bexport format" means that the "$$B\n" signature was written out correctly, but then the next 32 byte were garbage.

Also, cmd/compile explicitly flushes its write buffer after writing "$$B\n" and before writing the indexed export data.

Finally, this data isn't specially aligned at all within the file, so it's not like it could be generic file-level corruption.

Not to mention the index out of range error within the runtime during compilation seems bad.

@mdempsky

This comment has been minimized.

Copy link
Member

@mdempsky mdempsky commented Nov 4, 2019

@mvdan @siebenmann Can you two describe your test environments in more detail so maybe we can try to identify more commonalities?

E.g., it looks like you're both using linux/amd64. @siebenmann mentioned Fedora 30; @mvdan what Linux distro and version are you using?

Also, how many cores?

@siebenmann

This comment has been minimized.

Copy link

@siebenmann siebenmann commented Nov 4, 2019

The machine I'm seeing this on is an i7-8700K (not overclocked) with 32 GB of RAM and HyperThreading still enabled, so a 6/12 cores/threads machine. At the moment it has Fedora's kernel 5.3.7-200.fc30.x86_64. I can't seem to reproduce this on a machine running Ubuntu 18.04 (kernel 4.15.0-52-generic) with a Xeon E5-2680 CPU with HyperThreading off (8 cores) and 64 GB of RAM, nor on another Ubuntu 18.04 machine with the same kernel but a Ryzen 7 1800X CPU (again, not overclocked) with 8 cores (it has HT off) and 32 GB of RAM.

@mvdan

This comment has been minimized.

Copy link
Member Author

@mvdan mvdan commented Nov 4, 2019

I am on a Thinkpad with a i5-8350U, HT left on (4 real cores, 8 total), and 16GiB of RAM. I run Arch Linux, with its current 5.3.8 Linux kernel.

Aside from that my Go setup is a pretty standard build from source, using 1.13.4 to bootstrap. I don't have any fancy go env vars other than GOPRIVATE.

@siebenmann

This comment has been minimized.

Copy link

@siebenmann siebenmann commented Nov 4, 2019

I've now reproduced this on another Fedora 30 linux/amd64 machine, also with kernel 5.3.7-200.fc30.x86_64 but this time with a Ryzen 1800X (not overclocked, 8/16 cores) with 32 GB.

I've git bisected this and on both of my Fedora 30 machines, the commit where the failures start is 62e53b7, 'runtime: use signals to preempt Gs for suspendG', which is targeted at #10958 and #24543. I think this makes this issue CC @aclements.

@mdempsky

This comment has been minimized.

Copy link
Member

@mdempsky mdempsky commented Nov 4, 2019

Linux kernel version-specific signal handling behavior sounds fuuuuun.

@mvdan

This comment has been minimized.

Copy link
Member Author

@mvdan mvdan commented Nov 4, 2019

At least it's good that enough people test tip, so that this was caught at the start of the freeze :)

I'm happy to help debug this further, if the runtime maintainers can't reproduce this themselves.

@mdempsky

This comment has been minimized.

Copy link
Member

@mdempsky mdempsky commented Nov 4, 2019

@mvdan Oh yeah, definitely. I'm glad you and @siebenmann both reported the issue and have been able to reproduce it reliably enough to track down a culprit CL and identify some likely commonality between test setups. (I'll check the Linux kernel version I was using later, and also give it a shot on my personal laptop running Fedora 31.)

I'm just a little nervous because this is already the second issue discovered in the runtime preemption feature that landed last week.

@ianlancetaylor ianlancetaylor changed the title cmd/vet: "possiby version skew" errors after building Go from master runtime: corrupt binary export data seen after signal preemption CL Nov 4, 2019
@bpowers

This comment has been minimized.

Copy link
Contributor

@bpowers bpowers commented Nov 12, 2019

I see similar spurious problems, e.g. tonights tip c2edcf4:

##### internal linking of -buildmode=pie
# runtime/pprof [reflect.test]
fatal error: index out of range

runtime stack:
runtime.throw(0xe512d5, 0x12)
	/home/bpowers/go/src/runtime/panic.go:1106 +0x72
runtime.panicCheck1(0x41eeff, 0xe512d5, 0x12)
	/home/bpowers/go/src/runtime/panic.go:34 +0xd9
runtime.goPanicIndexU(0x2e4a9badd8, 0x400000)
	/home/bpowers/go/src/runtime/panic.go:91 +0x44
runtime.heapBitsForAddr(...)
	/home/bpowers/go/src/runtime/mbitmap.go:340
runtime.scanobject(0xb929eeb762d7c042, 0xc000032e90)
	/home/bpowers/go/src/runtime/mgcmark.go:1197 +0x39f
runtime.gcDrain(0xc000032e90, 0x3)
	/home/bpowers/go/src/runtime/mgcmark.go:1034 +0x1fb
runtime.gcBgMarkWorker.func2()
	/home/bpowers/go/src/runtime/mgc.go:1941 +0x80
runtime.systemstack(0x0)
	/home/bpowers/go/src/runtime/asm_amd64.s:370 +0x66
runtime.mstart()
	/home/bpowers/go/src/runtime/proc.go:1069

goroutine 28 [GC worker (idle)]:
runtime.systemstack_switch()
	/home/bpowers/go/src/runtime/asm_amd64.s:330 fp=0xc000484760 sp=0xc000484758 pc=0x460910
runtime.gcBgMarkWorker(0xc000031800)
	/home/bpowers/go/src/runtime/mgc.go:1928 +0x1be fp=0xc0004847d8 sp=0xc000484760 pc=0x41b53e
runtime.goexit()
	/home/bpowers/go/src/runtime/asm_amd64.s:1375 +0x1 fp=0xc0004847e0 sp=0xc0004847d8 pc=0x4628a1
created by runtime.gcBgMarkStartWorkers
	/home/bpowers/go/src/runtime/mgc.go:1822 +0x77

goroutine 1 [syscall]:
syscall.Syscall6(0x9, 0x0, 0x2192, 0x1, 0x1, 0x3, 0x0, 0x7f6981fd3000, 0x1, 0x0)
	/home/bpowers/go/src/syscall/asm_linux_amd64.s:41 +0x5
syscall.mmap(0x0, 0x2192, 0x1, 0x1, 0x3, 0x0, 0xc000358e38, 0x4aeee3, 0xc0004a32c0)
	/home/bpowers/go/src/syscall/zsyscall_linux_amd64.go:1627 +0x73
syscall.(*mmapper).Mmap(0x154ae80, 0x3, 0x0, 0x2192, 0x1, 0x1, 0x0, 0x0, 0x0, 0x0, ...)
	/home/bpowers/go/src/syscall/syscall_unix.go:58 +0xbe
syscall.Mmap(...)
	/home/bpowers/go/src/syscall/syscall_linux.go:985
cmd/compile/internal/gc.mapFile(0xc000494290, 0xdb, 0x20b7, 0x80, 0x200, 0x1b, 0x1c)
	/home/bpowers/go/src/cmd/compile/internal/gc/mapfile_mmap.go:28 +0xaf
cmd/compile/internal/gc.iimport(0xc0004ba050, 0xc00049c170)
	/home/bpowers/go/src/cmd/compile/internal/gc/iimport.go:113 +0x106
cmd/compile/internal/gc.importfile(0xc000359528, 0x0)
	/home/bpowers/go/src/cmd/compile/internal/gc/main.go:1270 +0x8a9
cmd/compile/internal/gc.(*noder).importDecl(0xc000350240, 0xc000064090)
	/home/bpowers/go/src/cmd/compile/internal/gc/noder.go:317 +0x86
cmd/compile/internal/gc.(*noder).decls(0xc000350240, 0xc000012800, 0x42, 0x80, 0x0, 0x0, 0xc000359678)
	/home/bpowers/go/src/cmd/compile/internal/gc/noder.go:289 +0x2d0
cmd/compile/internal/gc.(*noder).node(0xc000350240)
	/home/bpowers/go/src/cmd/compile/internal/gc/noder.go:244 +0xcc
cmd/compile/internal/gc.parseFiles(0xc0000a4600, 0x8, 0x8, 0x2)
	/home/bpowers/go/src/cmd/compile/internal/gc/noder.go:62 +0x35f
cmd/compile/internal/gc.Main(0xe6c650)
	/home/bpowers/go/src/cmd/compile/internal/gc/main.go:543 +0x2702
main.main()
	/home/bpowers/go/src/cmd/compile/main.go:50 +0xac
FAIL	reflect [build failed]

I'm similarly running Fedora with a 5.3.10 kernel and an intel i7-8550U (4 core with hyperthreading disabled). I haven't tried bisecting this yet, but can

@aclements

This comment has been minimized.

Copy link
Member

@aclements aclements commented Nov 12, 2019

Thanks for all of the reports!

Could someone who can reproduce this try setting debugScanConservative to true in runtime/mgc.go and capturing a failure? In a second run, could you both set debugScanConservative and run with GODEBUG=gccheckmark=1?

The two "index out of range" failures are interesting. In both cases they're in scanobject -> heapBitsForAddr. We can tell the argument for heapBitsForAddr from the argument to scanobject. In the first one, it's 0x6e6962732f727375 and in the second it's 0xb929eeb762d7c042. Both are obviously bad addresses that must have come from a gcWork buffer. Since all pointers that go in to a gcWork buffer have to pass findObject, which does the exact same lookup as heapBitsForAddr, they must have been valid when they were inserted and were corrupted while in the gcWork buffer.

@siebenmann

This comment has been minimized.

Copy link

@siebenmann siebenmann commented Nov 12, 2019

Is it useful to do this if I can only reliably reproduce the vet-based failures, not the crashes? And is there something smaller to reproduce problems than running all.bash, which produces several MB of output with these options that I don't know how to usefully reduce? (I can make the full output available if it would be useful.)

@aclements

This comment has been minimized.

Copy link
Member

@aclements aclements commented Nov 12, 2019

Yes, it's useful for the vet-based failures, too. I only need the output from the failed process (basically, after the last "# ..."), which should be substantially less.

@myitcv

This comment has been minimized.

Copy link
Member

@myitcv myitcv commented Nov 12, 2019

I just ran into this via go test -short -count=1 ./... but cannot reproduce. The output was:

vet: cmd/govim/internal/golang_org_x_tools/telemetry/id.go:8:8: could not import crypto/rand (cannot import "crypto/rand" (unknown bexport format version -1 ("ashdefault -o de\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00and.go\x06Reader\x02io\x00\x01b\x01n\x03err\x1f$GOROOT/src/crypto/rand/util.go\x04rand\x04bits\x01p\x03Int\bmath/big\x03max\x05esc:\x02\x1b$GOROOT/src/math/big/int.go\x01x\x03nat\x04esc:\x04v·3$$GOROOT/src/math/bits/bits_tables.go\x1d$GOROOT/src/math/bits/bits.go\alen8tab\tmath/bits\x05Len64\x14$GOROOT/src/io/io.go\x04Read\x03neg\x03abs\x04Sign\bSetInt64\x01z\tSetUint64\x03Set\x04Bits\x04Word\aSetBits\x03Abs\x03Neg\x03Add\x01y\x03Sub\x03Mul\bMulRange\x01a\bBinomial\x01k\x03Quo\x03Rem\x06QuoRem\x01r\x03Div\x03Mod\x06DivMod\x01m\x03Cmp\x06CmpAbs\x05Int64\x06Uint64\aIsInt64\bIsUint64\tSetString\x01s\x04base\x0esetFromScanner\vByteScanner\bSetBytes\x03buf\x05Bytes\x06BitLen\x10TrailingZeroBits\x03Exp\x03GCD\tlehmerGCD\x04Rand\x03rnd\tmath/rand")), possibly version skew - reinstall package)
@myitcv

This comment has been minimized.

Copy link
Member

@myitcv myitcv commented Nov 12, 2019

Sorry, sent too early there. System details as follows:

$ go env
GO111MODULE="on"
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/myitcv/.cache/go-build"
GOENV="/home/myitcv/.config/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOINSECURE=""
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/home/myitcv/gostuff"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/home/myitcv/gos"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/home/myitcv/gos/pkg/tool/linux_amd64"
GCCGO="gccgo"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/home/myitcv/gostuff/src/github.com/myitcv/govim/go.mod"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build072574281=/tmp/go-build -gno-record-gcc-switches"
$ go version
go version devel +99957b6930 Tue Nov 12 05:35:33 2019 +0000 linux/amd64
$ uname -r
5.3.0-19-generic
$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 19.10
Release:        19.10
Codename:       eoan
@mvdan

This comment has been minimized.

Copy link
Member Author

@mvdan mvdan commented Nov 12, 2019

@aclements I'm attaching the two logs you wanted. It's hard to reproduce it when running go vet on a single package, so I had an easier time with a for loop and many Go packages at once. I used the following bits of shell:

# first file
while true; do go clean -testcache -cache; go vet encoding/... |& tee out; grep bexport out && break; done
# second file
while true; do go clean -testcache -cache; GODEBUG=gccheckmark=1 go vet encoding/... |& tee out; grep bexport out && break; done

conservative.txt
conservative-gccheckmark.txt

If these files don't contain enough information, let me know and I can try again.

@mpx

This comment has been minimized.

Copy link
Contributor

@mpx mpx commented Nov 13, 2019

go version devel +43ec1b12f5 Wed Nov 13 01:15:54 2019 +0000 linux/amd64

I have been seeing very repeatable failures building all.bash under Fedora 30 (currently kernel 5.3.8-200.fc30.x86_64). The ... unknown bexport format version -1 ... always occurs in one of the tests.

Builds/tests are successful under GODEBUG=asyncpreemptoff=1.

Here is my debug output in case it's useful:

$ ./all.bash >& all-debugScanConservative.txt
$ GODEBUG=gccheckmark=1 ./all.bash >& all-debugScanConservative-gccheckmark.txt

all-debugScanConservative.txt
all-debugScanConservative-gccheckmark.txt

@zikaeroh

This comment has been minimized.

Copy link

@zikaeroh zikaeroh commented Nov 13, 2019

I just saw this too, while running some benchmarks:

# crypto/ecdsa
vet: ../../../sdk/gotip/src/crypto/ecdsa/ecdsa.go:36:2: could not import encoding/asn1 (cannot import "encoding/asn1" (unknown bexport format version -1 ("e/.antigen/bundl\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00/asn1.go\x00\x03Msg\x05Error\x01e\x0fStructuralError\x04esc:\x17asn1: structure error: \vSyntaxError\x14asn1: syntax error: \x05Bytes\tBitLength\x02At\x01b\tBitString\x01i")), possibly version skew - reinstall package)
# github.com/pmylund/go-cache
vet: ../../../nobackup/gotip_home/gopath/pkg/mod/github.com/pmylund/go-cache@v2.1.0+incompatible/cache.go:5:2: could not import fmt (cannot import "fmt" (unknown bexport format version -1 ("e/.antigen/bundl\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x06format\x01a\x00\x04esc:\x05esc:\x02\x18$GOROOT/src/fmt/print.go\x05Write\x01b\x01n\x03err\x05Width\x03wid\x02ok\tPrecision\x04prec\x04Flag\x01c\x06Format\x01f\x05State\x06String\bGoString\x01w\x06Writer\x02io")), possibly version skew - reinstall package)
?   	github.com/hortbot/hortbot	[no test files]

e/.antigen/bundl is something from my environment variables which has somehow made its way here...

go version devel +49e05d4f Wed Nov 13 20:53:39 2019 +0000 linux/amd64
Linux Jake-X1 5.3.11-arch1-1 #1 SMP PREEMPT Tue, 12 Nov 2019 22:19:48 +0000 x86_64 GNU/Linux
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
9 participants
You can’t perform that action at this time.