syscall: memory corruption on OpenBSD/amd64 when forking

[jrick]

What version of Go are you using (go version)?

$ go version
go version go1.13.2 openbsd/amd64

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

go env Output

$ go env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/jrick/.cache/go-build"
GOENV="/home/jrick/.config/go/env"
GOEXE=""
GOFLAGS="-tags=netgo -ldflags=-extldflags=-static"
GOHOSTARCH="amd64"
GOHOSTOS="openbsd"
GONOPROXY=""
GONOSUMDB=""
GOOS="openbsd"
GOPATH="/home/jrick/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/home/jrick/src/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/home/jrick/src/go/pkg/tool/openbsd_amd64"
GCCGO="gccgo"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0"

What did you do?

I observed these issues in one of my applications, and assumed it was a race or invalid unsafe.Pointer usage or some other fault of the application code. When the 1.13.2 release dropped yesterday I built it from source and observed a similar issue running the regression tests. The failed regression test does not look related to the memory corruption, but I can reproduce the problem by repeatedly running the test in a loop:

$ cd test # from go repo root
$ while :; do go run run.go -- fixedbugs/issue27829.go || break; done >go.panic 2>&1

It can take several minutes to observe the issue but here are some of the captured panics and fatal runtime errors:

https://gist.githubusercontent.com/jrick/f8b21ecbfbe516e1282b757d1bfe4165/raw/6cf0efb9ba47ba869f98817ce945971f2dff47d6/gistfile1.txt

https://gist.githubusercontent.com/jrick/9a54c085b918aa32910f4ece84e5aa21/raw/91ec29275c2eb1be49f62ad8a01a5317ad168c94/gistfile1.txt

https://gist.githubusercontent.com/jrick/8faf088593331c104cc0da0adb3f24da/raw/7c92e7e7d60d426b2156fd1bdff42e0717b708f1/gistfile1.txt

https://gist.githubusercontent.com/jrick/4645316444c12cd815fb71874f6bdfc4/raw/bffac2a448b07242a538b77a2823c9db34b6ef6f/gistfile1.txt

https://gist.githubusercontent.com/jrick/3843b180670811069319e4122d32507a/raw/0d1f897aa25d91307b04ae951f1b260f33246b61/gistfile1.txt

https://gist.githubusercontent.com/jrick/99b7171c5a49b4b069edf06884ad8e17/raw/740c7b9e8fa64d9ad149fd2669df94e89c466927/gistfile1.txt

Additionally, I observed go run hanging (no runtime failure due to deadlock) and it had to be killed with SIGABRT to get a trace: https://gist.githubusercontent.com/jrick/d4ae1e4355a7ac42f1910b7bb10a1297/raw/54e408c51a01444abda76dc32ac55c2dd217822b/gistfile1.txt

It may not matter which regression test is run as the errors also occur in run.go.

[jrick]

I missed that 1.13.3 was also released yesterday. Currently updating to that and will report whether this is still an issue.

[randall77]

This looks like cmd/go crashing while building the test, not the test itself.
The errors look heap realated. @mknyszek

[mknyszek]

@jrick maybe you meant this in your original post, but I just want to be clear. Does this reproduce with Go 1.12.X or older versions of Go?

Since we have a reasonable reproducer, the next step to me would be to just bisect what went into Go 1.13, if we know it isn't reproducing in Go 1.12. I genuinely have no idea what this could be. I thought at first that it could be scavenging related but that's highly unlikely for a number of reasons. I won't rule it out yet, though.

[jrick]

I haven't tested 1.12.x but will follow up testing that next. Currently hammering this test with 1.13.3 and so far it has not failed, but my application built with 1.13.3 still fails with SIGBUS (could be unrelated).

[jrick]

@mknyszek it still hasn't failed on 1.13.3 (running close to an hour now) but quickly failed on 1.12.12.

https://gist.githubusercontent.com/jrick/bb5a493e6ebd88e1e846f1c5c09c9e9a/raw/e82b0136b0826581f6e591915d3a634112f323a1/gistfile1.txt

[jrick]

1.13.3 finally errored after an hour.

https://gist.githubusercontent.com/jrick/bc61caaf7c9ec42a4dbdfe35bc466f81/raw/d8d12e83b3739aab06916bdd3829f5f7d6465906/gistfile1.txt

more errors from 1.13.3:

https://gist.githubusercontent.com/jrick/4c3f95703d723a00ccfdd5bc1e66a2d7/raw/2e96bb7bde68df5ba4ec2616694daa5dc7735307/gistfile1.txt

https://gist.githubusercontent.com/jrick/91ef3e4563e888799eab108911fee783/raw/9879d012c8f05f87b1ebc634584f63ca43ce2f9a/gistfile1.txt

https://gist.githubusercontent.com/jrick/526547300e41d3862020c49cc635a627/raw/019e65c005cf85be192d105ca9bf2b3aa0f2fca9/gistfile1.txt

https://gist.githubusercontent.com/jrick/b8c602332d1ecffc50b532d13f526bb3/raw/6d81e6035eeca726b7e105803507b7f7660eacef/gistfile1.txt

https://gist.githubusercontent.com/jrick/790cb535cc29666c93f7e2ffa1e06b7f/raw/67457c764b19514073de3c4b6b55e9aa8bea52db/gistfile1.txt

https://gist.githubusercontent.com/jrick/b393efebc2f6fe034fbde5070b413d47/raw/87ba312c229a9210bd0220aa957f7bb730970b23/gistfile1.txt

[jrick]

This remains a problem in 1.13.5, so it's not addressed by the recent fixes to the go tool.

https://gist.githubusercontent.com/jrick/a2499b2ae10b4c63359174e26c0fd936/raw/b233f14a518ca828c4416d803f81b1e8ca34d073/gistfile1.txt

[jrick]

This may be fork/exec related. This program exhibits similar crashes on OpenBSD 6.7 and Go 1.14.3.

package main

import (
        "os/exec"
)

func main() {
        sem := make(chan struct{}, 100)
        for {
                sem <- struct{}{}
                go func() {
                        err := exec.Command("/usr/bin/true").Run()
                        if err != nil {
                                panic(err)
                        }
                        <-sem
                }()
        }
}

crash trace: https://gist.github.com/jrick/8d6ef72796a772668b891310a18dd805

Synchronizing the os/exec call with an additional mutex appears to remove the crash.

[ianlancetaylor]

Thanks for the stack trace. That looks very much like a forked child process is changing the memory seen by the parent process. Which should of course be impossible. Specifically it seems that sched.lock.key is being set to zero while the lock is held during goschedImpl.

[jrick]

I'm seeing another strange thing in addition to that crash. Sometimes the program will run forever, spinning cpu, but appears to be deadlocked because none of the pids of those true processes are ever changing. Here's the trace after sending sigquit: https://gist.github.com/jrick/74aaa63624961145b7bc7b9518da75e1

[jrick]

I am currently testing with this OpenBSD kernel patch to the virtual memory system:

https://marc.info/?l=openbsd-tech&m=160008279223088&w=2

however these crashes still persist.

https://gist.githubusercontent.com/jrick/ce975453b230b583597b5cd99e989bf2/raw/088219ee1b4a28d8617ba3fc5c04904b55116046/gistfile1.txt

https://gist.githubusercontent.com/jrick/957e7cb28bef3cf5a60d9bda0cbd1257/raw/b9b9b4d18557d8c23bc0d9fb9f839d85d20a900a/gistfile1.txt

https://gist.githubusercontent.com/jrick/295159593904a7d8404dbb9216ff6566/raw/ff3fe4752815bab7857ef22baa1eaa2acd72d525/gistfile1.txt

Another interesting data point: so far it appears that this only reproduces on amd ryzen cpus, and not any intel ones.

[bcmills]

https://build.golang.org/log/3f45171bc52a0a86435abb9f795c0e8a45c4a0b0 looks similar:

haserrors/haserrors.go:3:18: undeclared name: undefined
fatal error: unexpected signal during runtime execution
[signal SIGSEGV: segmentation violation code=0x1 addr=0x4d578d58 pc=0x804a257]

runtime stack:
runtime/internal/atomic.Xadd64(0x83de928, 0x20360)
	/tmp/workdir/go/src/runtime/internal/atomic/atomic_386.s:145 +0x27

…

goroutine 4371 [runnable]:
syscall.syscall(0x80b4b40, 0x12, 0x0, 0x0)
	/tmp/workdir/go/src/runtime/sys_openbsd3.go:22 +0x20
syscall.Close(0x12)
	/tmp/workdir/go/src/syscall/zsyscall_openbsd_386.go:513 +0x39
syscall.forkExec({0xa630a408, 0x16}, {0xa6158a80, 0xe, 0xe}, 0x6bc0cbc0)
	/tmp/workdir/go/src/syscall/exec_unix.go:227 +0x4cc
syscall.StartProcess(...)
	/tmp/workdir/go/src/syscall/exec_unix.go:264
os.startProcess({0xa630a408, 0x16}, {0xa6158a80, 0xe, 0xe}, 0x6bc0cc84)
	/tmp/workdir/go/src/os/exec_posix.go:55 +0x256
os.StartProcess({0xa630a408, 0x16}, {0xa6158a80, 0xe, 0xe}, 0x6bc0cc84)
	/tmp/workdir/go/src/os/exec.go:106 +0x57
os/exec.(*Cmd).Start(0xa3537b80)
	/tmp/workdir/go/src/os/exec/exec.go:422 +0x588
os/exec.(*Cmd).Run(0xa3537b80)
	/tmp/workdir/go/src/os/exec/exec.go:338 +0x1b
golang.org/x/tools/go/internal/cgo.Run(0xa26a8b40, {0xa2764090, 0x17}, {0xa60778e0, 0x20}, 0x0)
	/tmp/workdir/gopath/src/golang.org/x/tools/go/internal/cgo/cgo.go:172 +0xc74
golang.org/x/tools/go/internal/cgo.ProcessFiles(0xa26a8b40, 0x837eecc0, 0x0, 0x0)
	/tmp/workdir/gopath/src/golang.org/x/tools/go/internal/cgo/cgo.go:85 +0x1a1
golang.org/x/tools/go/loader.(*Config).parsePackageFiles(0x9d428420, 0xa26a8b40, 0x67)
	/tmp/workdir/gopath/src/golang.org/x/tools/go/loader/loader.go:758 +0x232
golang.org/x/tools/go/loader.(*importer).load(0x837ed770, 0xa26a8b40)
	/tmp/workdir/gopath/src/golang.org/x/tools/go/loader/loader.go:976 +0x68
golang.org/x/tools/go/loader.(*importer).startLoad.func1(0x837ed770, 0xa26a8b40, 0xa2afe0e0)
	/tmp/workdir/gopath/src/golang.org/x/tools/go/loader/loader.go:962 +0x23
created by golang.org/x/tools/go/loader.(*importer).startLoad
	/tmp/workdir/gopath/src/golang.org/x/tools/go/loader/loader.go:961 +0x174