Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

runtime: runtime crashes, reproducer same as #20427 #36371

Closed
t57root opened this issue Jan 3, 2020 · 8 comments
Closed

runtime: runtime crashes, reproducer same as #20427 #36371

t57root opened this issue Jan 3, 2020 · 8 comments
Milestone

Comments

@t57root
Copy link

@t57root t57root commented Jan 3, 2020

What version of Go are you using (go version)?

$ go version
go version go1.13.5 linux/amd64

Does this issue reproduce with the latest release?

yes

What operating system and processor architecture are you using (go env)?

uname -a
Linux hostname 3.18.6-2.el7.centos.x86_64 #1 SMP Mon Oct 24 13:01:33 CST 2016 x86_64 x86_64 x86_64 GNU/Linux

go env Output
$ go env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/root/.cache/go-build"
GOENV="/root/.config/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/root/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/local/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"
GCCGO="gccgo"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build454867560=/tmp/go-build -gno-record-gcc-switches"

What did you do?

The program crash randomly on some specific servers, I've met this problem when using go1.12.4 about half year ago(issue #31538 ), but can't find a reproducer back then.
Now I'm using go1.13.5 on the same server, the problem remains. After some googling I've find a reproducer through this blog which is extactly the same as the reproducer in issue #20437

package main

/* stdlib includes */
import (
        "fmt"
        "os/exec"
)

func run(done chan struct{}) {
        cmd := exec.Command("true")
        if err := cmd.Start(); err != nil {
                goto finished
        }

        cmd.Wait()

finished:
        done <- struct{}{}
        return
}

func main() {
        fmt.Println("Starting a bunch of goroutines...")

        // 8 & 16 are arbitrary
        done := make(chan struct{}, 16)

        for i := 0; i < 8; i++ {
                go run(done)
        }

        for {
                select {
                case <-done:
                        go run(done)
                }
        }
}

What did you expect to see?

Run without crash

What did you see instead?

Starting a bunch of goroutines...
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x4051d8]

goroutine 386 [running]:
panic(0x4c3220, 0x58c380)
	/usr/lib/go-1.13/src/runtime/panic.go:722 +0x2c2 fp=0xc0000c5e70 sp=0xc0000c5de0 pc=0x429f82
runtime.panicmem(...)
	/usr/lib/go-1.13/src/runtime/panic.go:199
runtime.sigpanic()
	/usr/lib/go-1.13/src/runtime/signal_unix.go:394 +0x3ec fp=0xc0000c5ea0 sp=0xc0000c5e70 pc=0x43e65c
runtime.send(0xc0000ba0c0, 0xc00009a000, 0xc0000c5fbf, 0xc0000c5f30, 0x3)
	/usr/lib/go-1.13/src/runtime/chan.go:295 +0x48 fp=0xc0000c5ed0 sp=0xc0000c5ea0 pc=0x4051d8
runtime.chansend(0xc0000ba0c0, 0xc0000c5fbf, 0x1, 0x4a60ae, 0x0)
	/usr/lib/go-1.13/src/runtime/chan.go:193 +0x51a fp=0xc0000c5f50 sp=0xc0000c5ed0 pc=0x40507a
runtime.chansend1(0xc0000ba0c0, 0xc0000c5fbf)
	/usr/lib/go-1.13/src/runtime/chan.go:127 +0x35 fp=0xc0000c5f88 sp=0xc0000c5f50 pc=0x404b55
main.run(0xc0000ba0c0)
	/root/go/src/testapi/coretest.go:18 +0x7e fp=0xc0000c5fd8 sp=0xc0000c5f88 pc=0x4a60ae
runtime.goexit()
	/usr/lib/go-1.13/src/runtime/asm_amd64.s:1357 +0x1 fp=0xc0000c5fe0 sp=0xc0000c5fd8 pc=0x455b21
created by main.main
	/root/go/src/testapi/coretest.go:35 +0x10c

goroutine 1 [chan receive]:
runtime.gopark(0x4ea2c8, 0xc0000ba118, 0x170e, 0x3)
	/usr/lib/go-1.13/src/runtime/proc.go:304 +0xe0 fp=0xc0000c4e30 sp=0xc0000c4e10 pc=0x42c090
runtime.goparkunlock(...)
	/usr/lib/go-1.13/src/runtime/proc.go:310
runtime.chanrecv(0xc0000ba0c0, 0x0, 0xc000000101, 0x4a61ec)
	/usr/lib/go-1.13/src/runtime/chan.go:524 +0x2e8 fp=0xc0000c4ec0 sp=0xc0000c4e30 pc=0x405978
runtime.chanrecv1(0xc0000ba0c0, 0x0)
	/usr/lib/go-1.13/src/runtime/chan.go:406 +0x2b fp=0xc0000c4ef0 sp=0xc0000c4ec0 pc=0x40567b
main.main()
	/root/go/src/testapi/coretest.go:34 +0xea fp=0xc0000c4f60 sp=0xc0000c4ef0 pc=0x4a61ca
runtime.main()
	/usr/lib/go-1.13/src/runtime/proc.go:203 +0x21e fp=0xc0000c4fe0 sp=0xc0000c4f60 pc=0x42bcce
runtime.goexit()
	/usr/lib/go-1.13/src/runtime/asm_amd64.s:1357 +0x1 fp=0xc0000c4fe8 sp=0xc0000c4fe0 pc=0x455b21

goroutine 2 [force gc (idle)]:
runtime.gopark(0x4ea2c8, 0x593590, 0x1411, 0x1)
	/usr/lib/go-1.13/src/runtime/proc.go:304 +0xe0 fp=0xc00007cfb0 sp=0xc00007cf90 pc=0x42c090
runtime.goparkunlock(...)
	/usr/lib/go-1.13/src/runtime/proc.go:310
runtime.forcegchelper()
	/usr/lib/go-1.13/src/runtime/proc.go:253 +0xb7 fp=0xc00007cfe0 sp=0xc00007cfb0 pc=0x42bf47
runtime.goexit()
	/usr/lib/go-1.13/src/runtime/asm_amd64.s:1357 +0x1 fp=0xc00007cfe8 sp=0xc00007cfe0 pc=0x455b21
created by runtime.init.5
	/usr/lib/go-1.13/src/runtime/proc.go:242 +0x35

goroutine 3 [GC sweep wait]:
runtime.gopark(0x4ea2c8, 0x5936c0, 0x140c, 0x1)
	/usr/lib/go-1.13/src/runtime/proc.go:304 +0xe0 fp=0xc00007d7a8 sp=0xc00007d788 pc=0x42c090
runtime.goparkunlock(...)
	/usr/lib/go-1.13/src/runtime/proc.go:310
runtime.bgsweep(0xc00001e070)
	/usr/lib/go-1.13/src/runtime/mgcsweep.go:70 +0x9c fp=0xc00007d7d8 sp=0xc00007d7a8 pc=0x41fa6c
runtime.goexit()
	/usr/lib/go-1.13/src/runtime/asm_amd64.s:1357 +0x1 fp=0xc00007d7e0 sp=0xc00007d7d8 pc=0x455b21
created by runtime.gcenable
	/usr/lib/go-1.13/src/runtime/mgc.go:210 +0x5c

goroutine 4 [GC scavenge wait]:
runtime.gopark(0x4ea2c8, 0x5937a0, 0x140d, 0x1)
	/usr/lib/go-1.13/src/runtime/proc.go:304 +0xe0 fp=0xc00007df40 sp=0xc00007df20 pc=0x42c090
runtime.goparkunlock(...)
	/usr/lib/go-1.13/src/runtime/proc.go:310
runtime.bgscavenge(0xc00001e070)
	/usr/lib/go-1.13/src/runtime/mgcscavenge.go:299 +0xe1 fp=0xc00007dfd8 sp=0xc00007df40 pc=0x41f0f1
runtime.goexit()
	/usr/lib/go-1.13/src/runtime/asm_amd64.s:1357 +0x1 fp=0xc00007dfe0 sp=0xc00007dfd8 pc=0x455b21
created by runtime.gcenable
	/usr/lib/go-1.13/src/runtime/mgc.go:211 +0x7e

goroutine 5 [finalizer wait]:
runtime.gopark(0x4ea2c8, 0x5aeb78, 0x1410, 0x1)
	/usr/lib/go-1.13/src/runtime/proc.go:304 +0xe0 fp=0xc00007c758 sp=0xc00007c738 pc=0x42c090
runtime.goparkunlock(...)
	/usr/lib/go-1.13/src/runtime/proc.go:310
runtime.runfinq()
	/usr/lib/go-1.13/src/runtime/mfinal.go:175 +0xa3 fp=0xc00007c7e0 sp=0xc00007c758 pc=0x415ad3
runtime.goexit()
	/usr/lib/go-1.13/src/runtime/asm_amd64.s:1357 +0x1 fp=0xc00007c7e8 sp=0xc00007c7e0 pc=0x455b21
created by runtime.createfing
	/usr/lib/go-1.13/src/runtime/mfinal.go:156 +0x61

goroutine 333 [GC worker (idle)]:
runtime.gopark(0x4ea188, 0xc0003eac10, 0xc000221418, 0x0)
	/usr/lib/go-1.13/src/runtime/proc.go:304 +0xe0 fp=0xc000220760 sp=0xc000220740 pc=0x42c090
runtime.gcBgMarkWorker(0xc000054f00)
	/usr/lib/go-1.13/src/runtime/mgc.go:1846 +0xff fp=0xc0002207d8 sp=0xc000220760 pc=0x41935f
runtime.goexit()
	/usr/lib/go-1.13/src/runtime/asm_amd64.s:1357 +0x1 fp=0xc0002207e0 sp=0xc0002207d8 pc=0x455b21
created by runtime.gcBgMarkStartWorkers
	/usr/lib/go-1.13/src/runtime/mgc.go:1794 +0x77

goroutine 388 [GC worker (idle)]:
runtime.gopark(0x4ea188, 0xc0003eac30, 0xc0000d1418, 0x0)
	/usr/lib/go-1.13/src/runtime/proc.go:304 +0xe0 fp=0xc00007f760 sp=0xc00007f740 pc=0x42c090
runtime.gcBgMarkWorker(0xc000059900)
	/usr/lib/go-1.13/src/runtime/mgc.go:1846 +0xff fp=0xc00007f7d8 sp=0xc00007f760 pc=0x41935f
runtime.goexit()
	/usr/lib/go-1.13/src/runtime/asm_amd64.s:1357 +0x1 fp=0xc00007f7e0 sp=0xc00007f7d8 pc=0x455b21
created by runtime.gcBgMarkStartWorkers
	/usr/lib/go-1.13/src/runtime/mgc.go:1794 +0x77

goroutine 348 [GC worker (idle)]:
runtime.gopark(0x4ea188, 0xc0003eabe0, 0xc0001b1418, 0x0)
	/usr/lib/go-1.13/src/runtime/proc.go:304 +0xe0 fp=0xc00007ff60 sp=0xc00007ff40 pc=0x42c090
runtime.gcBgMarkWorker(0xc000042500)
	/usr/lib/go-1.13/src/runtime/mgc.go:1846 +0xff fp=0xc00007ffd8 sp=0xc00007ff60 pc=0x41935f
runtime.goexit()
	/usr/lib/go-1.13/src/runtime/asm_amd64.s:1357 +0x1 fp=0xc00007ffe0 sp=0xc00007ffd8 pc=0x455b21
created by runtime.gcBgMarkStartWorkers
	/usr/lib/go-1.13/src/runtime/mgc.go:1794 +0x77

goroutine 320 [runnable]:
syscall.Wait4(0x1f847, 0xc0000feed4, 0x0, 0xc0004c6000, 0xd, 0x0, 0x3)
	/usr/lib/go-1.13/src/syscall/syscall_linux.go:339 +0x91 fp=0xc0000fee98 sp=0xc0000fee90 pc=0x486a61
os.(*Process).wait(0xc0004a8000, 0x4e9fa8, 0x4e9fb0, 0x4e9fa0)
	/usr/lib/go-1.13/src/os/exec_unix.go:38 +0xa8 fp=0xc0000fef10 sp=0xc0000fee98 pc=0x493068
os.(*Process).Wait(...)
	/usr/lib/go-1.13/src/os/exec.go:125
os/exec.(*Cmd).Wait(0xc0003cbce0, 0x0, 0x0)
	/usr/lib/go-1.13/src/os/exec/exec.go:501 +0x60 fp=0xc0000fef88 sp=0xc0000fef10 pc=0x4a4cf0
main.run(0xc0000ba0c0)
	/root/go/src/testapi/coretest.go:15 +0x96 fp=0xc0000fefd8 sp=0xc0000fef88 pc=0x4a60c6
runtime.goexit()
	/usr/lib/go-1.13/src/runtime/asm_amd64.s:1357 +0x1 fp=0xc0000fefe0 sp=0xc0000fefd8 pc=0x455b21
created by main.main
	/root/go/src/testapi/coretest.go:35 +0x10c

goroutine 404 [GC worker (idle)]:
runtime.gopark(0x4ea188, 0xc000438580, 0x1418, 0x0)
	/usr/lib/go-1.13/src/runtime/proc.go:304 +0xe0 fp=0xc00007af60 sp=0xc00007af40 pc=0x42c090
runtime.gcBgMarkWorker(0xc000044a00)
	/usr/lib/go-1.13/src/runtime/mgc.go:1846 +0xff fp=0xc00007afd8 sp=0xc00007af60 pc=0x41935f
runtime.goexit()
	/usr/lib/go-1.13/src/runtime/asm_amd64.s:1357 +0x1 fp=0xc00007afe0 sp=0xc00007afd8 pc=0x455b21
created by runtime.gcBgMarkStartWorkers
	/usr/lib/go-1.13/src/runtime/mgc.go:1794 +0x77

goroutine 347 [runnable]:
runtime.gopark(0x4ea2c8, 0x59d380, 0xc0001c1912, 0x4)
	/usr/lib/go-1.13/src/runtime/proc.go:304 +0xe0 fp=0xc000148cf0 sp=0xc000148cd0 pc=0x42c090
runtime.goparkunlock(...)
	/usr/lib/go-1.13/src/runtime/proc.go:310
runtime.semacquire1(0x593e10, 0xc000148d00, 0x0, 0x0)
	/usr/lib/go-1.13/src/runtime/sema.go:144 +0x1c0 fp=0xc000148d58 sp=0xc000148cf0 pc=0x43bb10
runtime.semacquire(...)
	/usr/lib/go-1.13/src/runtime/sema.go:95
runtime.gcStart(0x0, 0x0, 0xc000000000)
	/usr/lib/go-1.13/src/runtime/mgc.go:1231 +0xe7 fp=0xc000148dc8 sp=0xc000148d58 pc=0x417dd7
runtime.mallocgc(0x90, 0x4dae80, 0x148d01, 0x0)
	/usr/lib/go-1.13/src/runtime/malloc.go:1115 +0x416 fp=0xc000148e68 sp=0xc000148dc8 pc=0x40b646
runtime.newobject(0x4dae80, 0x1)
	/usr/lib/go-1.13/src/runtime/malloc.go:1151 +0x38 fp=0xc000148e98 sp=0xc000148e68 pc=0x40bdb8
os.(*Process).wait(0xc0004a8030, 0x4e9fa8, 0x4e9fb0, 0x4e9fa0)
	/usr/lib/go-1.13/src/os/exec_unix.go:37 +0x75 fp=0xc000148f10 sp=0xc000148e98 pc=0x493035
os.(*Process).Wait(...)
	/usr/lib/go-1.13/src/os/exec.go:125
os/exec.(*Cmd).Wait(0xc00041d1e0, 0x0, 0x0)
	/usr/lib/go-1.13/src/os/exec/exec.go:501 +0x60 fp=0xc000148f88 sp=0xc000148f10 pc=0x4a4cf0
main.run(0xc0000ba0c0)
	/root/go/src/testapi/coretest.go:15 +0x96 fp=0xc000148fd8 sp=0xc000148f88 pc=0x4a60c6
runtime.goexit()
	/usr/lib/go-1.13/src/runtime/asm_amd64.s:1357 +0x1 fp=0xc000148fe0 sp=0xc000148fd8 pc=0x455b21
created by main.main
	/root/go/src/testapi/coretest.go:35 +0x10c

goroutine 402 [GC worker (idle)]:
runtime.systemstack_switch()
	/usr/lib/go-1.13/src/runtime/asm_amd64.s:330 fp=0xc000079f60 sp=0xc000079f58 pc=0x453bd0
runtime.gcBgMarkWorker(0xc000024000)
	/usr/lib/go-1.13/src/runtime/mgc.go:1900 +0x1bd fp=0xc000079fd8 sp=0xc000079f60 pc=0x41941d
runtime.goexit()
	/usr/lib/go-1.13/src/runtime/asm_amd64.s:1357 +0x1 fp=0xc000079fe0 sp=0xc000079fd8 pc=0x455b21
created by runtime.gcBgMarkStartWorkers
	/usr/lib/go-1.13/src/runtime/mgc.go:1794 +0x77

goroutine 366 [GC worker (idle)]:
runtime.gopark(0x4ea188, 0xc0001bb5f0, 0xc000131418, 0x0)
	/usr/lib/go-1.13/src/runtime/proc.go:304 +0xe0 fp=0xc00007e760 sp=0xc00007e740 pc=0x42c090
runtime.gcBgMarkWorker(0xc000026500)
	/usr/lib/go-1.13/src/runtime/mgc.go:1846 +0xff fp=0xc00007e7d8 sp=0xc00007e760 pc=0x41935f
runtime.goexit()
	/usr/lib/go-1.13/src/runtime/asm_amd64.s:1357 +0x1 fp=0xc00007e7e0 sp=0xc00007e7d8 pc=0x455b21
created by runtime.gcBgMarkStartWorkers
	/usr/lib/go-1.13/src/runtime/mgc.go:1794 +0x77

goroutine 403 [GC worker (idle)]:
runtime.gopark(0x4ea188, 0xc0001bb600, 0x1418, 0x0)
	/usr/lib/go-1.13/src/runtime/proc.go:304 +0xe0 fp=0xc00007a760 sp=0xc00007a740 pc=0x42c090
runtime.gcBgMarkWorker(0xc000028a00)
	/usr/lib/go-1.13/src/runtime/mgc.go:1846 +0xff fp=0xc00007a7d8 sp=0xc00007a760 pc=0x41935f
runtime.goexit()
	/usr/lib/go-1.13/src/runtime/asm_amd64.s:1357 +0x1 fp=0xc00007a7e0 sp=0xc00007a7d8 pc=0x455b21
created by runtime.gcBgMarkStartWorkers
	/usr/lib/go-1.13/src/runtime/mgc.go:1794 +0x77

goroutine 367 [GC worker (idle)]:
runtime.gopark(0x4ea188, 0xc0001bb610, 0x1418, 0x0)
	/usr/lib/go-1.13/src/runtime/proc.go:304 +0xe0 fp=0xc00021c760 sp=0xc00021c740 pc=0x42c090
runtime.gcBgMarkWorker(0xc00002af00)
	/usr/lib/go-1.13/src/runtime/mgc.go:1846 +0xff fp=0xc00021c7d8 sp=0xc00021c760 pc=0x41935f
runtime.goexit()
	/usr/lib/go-1.13/src/runtime/asm_amd64.s:1357 +0x1 fp=0xc00021c7e0 sp=0xc00021c7d8 pc=0x455b21
created by runtime.gcBgMarkStartWorkers
	/usr/lib/go-1.13/src/runtime/mgc.go:1794 +0x77

goroutine 371 [GC worker (idle)]:
runtime.gopark(0x4ea188, 0xc0001360a0, 0xc000181418, 0x0)
	/usr/lib/go-1.13/src/runtime/proc.go:304 +0xe0 fp=0xc000079760 sp=0xc000079740 pc=0x42c090
runtime.gcBgMarkWorker(0xc00002d400)
	/usr/lib/go-1.13/src/runtime/mgc.go:1846 +0xff fp=0xc0000797d8 sp=0xc000079760 pc=0x41935f
runtime.goexit()
	/usr/lib/go-1.13/src/runtime/asm_amd64.s:1357 +0x1 fp=0xc0000797e0 sp=0xc0000797d8 pc=0x455b21
created by runtime.gcBgMarkStartWorkers
	/usr/lib/go-1.13/src/runtime/mgc.go:1794 +0x77

goroutine 417 [GC worker (idle)]:
runtime.gopark(0x4ea188, 0xc0003b10c0, 0xc0000f1418, 0x0)
	/usr/lib/go-1.13/src/runtime/proc.go:304 +0xe0 fp=0xc000078f60 sp=0xc000078f40 pc=0x42c090
runtime.gcBgMarkWorker(0xc00002f900)
	/usr/lib/go-1.13/src/runtime/mgc.go:1846 +0xff fp=0xc000078fd8 sp=0xc000078f60 pc=0x41935f
runtime.goexit()
	/usr/lib/go-1.13/src/runtime/asm_amd64.s:1357 +0x1 fp=0xc000078fe0 sp=0xc000078fd8 pc=0x455b21
created by runtime.gcBgMarkStartWorkers
	/usr/lib/go-1.13/src/runtime/mgc.go:1794 +0x77

goroutine 368 [GC worker (idle)]:
runtime.gopark(0x4ea188, 0xc0001bb620, 0x1418, 0x0)
	/usr/lib/go-1.13/src/runtime/proc.go:304 +0xe0 fp=0xc00021cf60 sp=0xc00021cf40 pc=0x42c090
runtime.gcBgMarkWorker(0xc000032000)
	/usr/lib/go-1.13/src/runtime/mgc.go:1846 +0xff fp=0xc00021cfd8 sp=0xc00021cf60 pc=0x41935f
runtime.goexit()
	/usr/lib/go-1.13/src/runtime/asm_amd64.s:1357 +0x1 fp=0xc00021cfe0 sp=0xc00021cfd8 pc=0x455b21
created by runtime.gcBgMarkStartWorkers
	/usr/lib/go-1.13/src/runtime/mgc.go:1794 +0x77

goroutine 418 [GC worker (idle)]:
runtime.gopark(0x4ea188, 0xc000492080, 0x1418, 0x0)
	/usr/lib/go-1.13/src/runtime/proc.go:304 +0xe0 fp=0xc0004a2760 sp=0xc0004a2740 pc=0x42c090
runtime.gcBgMarkWorker(0xc000034500)
	/usr/lib/go-1.13/src/runtime/mgc.go:1846 +0xff fp=0xc0004a27d8 sp=0xc0004a2760 pc=0x41935f
runtime.goexit()
	/usr/lib/go-1.13/src/runtime/asm_amd64.s:1357 +0x1 fp=0xc0004a27e0 sp=0xc0004a27d8 pc=0x455b21
created by runtime.gcBgMarkStartWorkers
	/usr/lib/go-1.13/src/runtime/mgc.go:1794 +0x77

goroutine 387 [GC worker (idle)]:
runtime.gopark(0x4ea188, 0xc000492090, 0xc0000d1418, 0x0)
	/usr/lib/go-1.13/src/runtime/proc.go:304 +0xe0 fp=0xc00007ef60 sp=0xc00007ef40 pc=0x42c090
runtime.gcBgMarkWorker(0xc000036a00)
	/usr/lib/go-1.13/src/runtime/mgc.go:1846 +0xff fp=0xc00007efd8 sp=0xc00007ef60 pc=0x41935f
runtime.goexit()
	/usr/lib/go-1.13/src/runtime/asm_amd64.s:1357 +0x1 fp=0xc00007efe0 sp=0xc00007efd8 pc=0x455b21
created by runtime.gcBgMarkStartWorkers
	/usr/lib/go-1.13/src/runtime/mgc.go:1794 +0x77

goroutine 419 [GC worker (idle)]:
runtime.gopark(0x4ea188, 0xc0001360b0, 0x1418, 0x0)
	/usr/lib/go-1.13/src/runtime/proc.go:304 +0xe0 fp=0xc0004a2f60 sp=0xc0004a2f40 pc=0x42c090
runtime.gcBgMarkWorker(0xc000038f00)
	/usr/lib/go-1.13/src/runtime/mgc.go:1846 +0xff fp=0xc0004a2fd8 sp=0xc0004a2f60 pc=0x41935f
runtime.goexit()
	/usr/lib/go-1.13/src/runtime/asm_amd64.s:1357 +0x1 fp=0xc0004a2fe0 sp=0xc0004a2fd8 pc=0x455b21
created by runtime.gcBgMarkStartWorkers
	/usr/lib/go-1.13/src/runtime/mgc.go:1794 +0x77

goroutine 420 [GC worker (idle)]:
runtime.gopark(0x4ea188, 0xc0001360c0, 0x1418, 0x0)
	/usr/lib/go-1.13/src/runtime/proc.go:304 +0xe0 fp=0xc0004a3760 sp=0xc0004a3740 pc=0x42c090
runtime.gcBgMarkWorker(0xc00003b400)
	/usr/lib/go-1.13/src/runtime/mgc.go:1846 +0xff fp=0xc0004a37d8 sp=0xc0004a3760 pc=0x41935f
runtime.goexit()
	/usr/lib/go-1.13/src/runtime/asm_amd64.s:1357 +0x1 fp=0xc0004a37e0 sp=0xc0004a37d8 pc=0x455b21
created by runtime.gcBgMarkStartWorkers
	/usr/lib/go-1.13/src/runtime/mgc.go:1794 +0x77

goroutine 421 [GC worker (idle)]:
runtime.gopark(0x4ea188, 0xc0001360d0, 0x1418, 0x0)
	/usr/lib/go-1.13/src/runtime/proc.go:304 +0xe0 fp=0xc0004a3f60 sp=0xc0004a3f40 pc=0x42c090
runtime.gcBgMarkWorker(0xc00003d900)
	/usr/lib/go-1.13/src/runtime/mgc.go:1846 +0xff fp=0xc0004a3fd8 sp=0xc0004a3f60 pc=0x41935f
runtime.goexit()
	/usr/lib/go-1.13/src/runtime/asm_amd64.s:1357 +0x1 fp=0xc0004a3fe0 sp=0xc0004a3fd8 pc=0x455b21
created by runtime.gcBgMarkStartWorkers
	/usr/lib/go-1.13/src/runtime/mgc.go:1794 +0x77

goroutine 422 [GC worker (idle)]:
runtime.gopark(0x4ea188, 0xc0001bb630, 0x1418, 0x0)
	/usr/lib/go-1.13/src/runtime/proc.go:304 +0xe0 fp=0xc0004a4760 sp=0xc0004a4740 pc=0x42c090
runtime.gcBgMarkWorker(0xc000040000)
	/usr/lib/go-1.13/src/runtime/mgc.go:1846 +0xff fp=0xc0004a47d8 sp=0xc0004a4760 pc=0x41935f
runtime.goexit()
	/usr/lib/go-1.13/src/runtime/asm_amd64.s:1357 +0x1 fp=0xc0004a47e0 sp=0xc0004a47d8 pc=0x455b21
created by runtime.gcBgMarkStartWorkers
	/usr/lib/go-1.13/src/runtime/mgc.go:1794 +0x77

goroutine 349 [GC worker (idle)]:
runtime.gopark(0x4ea188, 0xc0001bb640, 0x1418, 0x0)
	/usr/lib/go-1.13/src/runtime/proc.go:304 +0xe0 fp=0xc00049e760 sp=0xc00049e740 pc=0x42c090
runtime.gcBgMarkWorker(0xc000046f00)
	/usr/lib/go-1.13/src/runtime/mgc.go:1846 +0xff fp=0xc00049e7d8 sp=0xc00049e760 pc=0x41935f
runtime.goexit()
	/usr/lib/go-1.13/src/runtime/asm_amd64.s:1357 +0x1 fp=0xc00049e7e0 sp=0xc00049e7d8 pc=0x455b21
created by runtime.gcBgMarkStartWorkers
	/usr/lib/go-1.13/src/runtime/mgc.go:1794 +0x77

goroutine 405 [GC worker (idle)]:
runtime.gopark(0x4ea188, 0xc0001bb650, 0x1418, 0x0)
	/usr/lib/go-1.13/src/runtime/proc.go:304 +0xe0 fp=0xc00007b760 sp=0xc00007b740 pc=0x42c090
runtime.gcBgMarkWorker(0xc000049400)
	/usr/lib/go-1.13/src/runtime/mgc.go:1846 +0xff fp=0xc00007b7d8 sp=0xc00007b760 pc=0x41935f
runtime.goexit()
	/usr/lib/go-1.13/src/runtime/asm_amd64.s:1357 +0x1 fp=0xc00007b7e0 sp=0xc00007b7d8 pc=0x455b21
created by runtime.gcBgMarkStartWorkers
	/usr/lib/go-1.13/src/runtime/mgc.go:1794 +0x77

goroutine 350 [GC worker (idle)]:
runtime.gopark(0x4ea188, 0xc0001bb660, 0x1418, 0x0)
	/usr/lib/go-1.13/src/runtime/proc.go:304 +0xe0 fp=0xc00049ef60 sp=0xc00049ef40 pc=0x42c090
runtime.gcBgMarkWorker(0xc00004b900)
	/usr/lib/go-1.13/src/runtime/mgc.go:1846 +0xff fp=0xc00049efd8 sp=0xc00049ef60 pc=0x41935f
runtime.goexit()
	/usr/lib/go-1.13/src/runtime/asm_amd64.s:1357 +0x1 fp=0xc00049efe0 sp=0xc00049efd8 pc=0x455b21
created by runtime.gcBgMarkStartWorkers
	/usr/lib/go-1.13/src/runtime/mgc.go:1794 +0x77

goroutine 406 [GC worker (idle)]:
runtime.gopark(0x4ea188, 0xc0003eabf0, 0x1418, 0x0)
	/usr/lib/go-1.13/src/runtime/proc.go:304 +0xe0 fp=0xc00007bf60 sp=0xc00007bf40 pc=0x42c090
runtime.gcBgMarkWorker(0xc00004e000)
	/usr/lib/go-1.13/src/runtime/mgc.go:1846 +0xff fp=0xc00007bfd8 sp=0xc00007bf60 pc=0x41935f
runtime.goexit()
	/usr/lib/go-1.13/src/runtime/asm_amd64.s:1357 +0x1 fp=0xc00007bfe0 sp=0xc00007bfd8 pc=0x455b21
created by runtime.gcBgMarkStartWorkers
	/usr/lib/go-1.13/src/runtime/mgc.go:1794 +0x77

goroutine 351 [GC worker (idle)]:
runtime.gopark(0x4ea188, 0xc0003eac00, 0x1418, 0x0)
	/usr/lib/go-1.13/src/runtime/proc.go:304 +0xe0 fp=0xc00049f760 sp=0xc00049f740 pc=0x42c090
runtime.gcBgMarkWorker(0xc000050500)
	/usr/lib/go-1.13/src/runtime/mgc.go:1846 +0xff fp=0xc00049f7d8 sp=0xc00049f760 pc=0x41935f
runtime.goexit()
	/usr/lib/go-1.13/src/runtime/asm_amd64.s:1357 +0x1 fp=0xc00049f7e0 sp=0xc00049f7d8 pc=0x455b21
created by runtime.gcBgMarkStartWorkers
	/usr/lib/go-1.13/src/runtime/mgc.go:1794 +0x77

goroutine 407 [GC worker (idle)]:
runtime.gopark(0x4ea188, 0xc00044a580, 0x1418, 0x0)
	/usr/lib/go-1.13/src/runtime/proc.go:304 +0xe0 fp=0xc0004ae760 sp=0xc0004ae740 pc=0x42c090
runtime.gcBgMarkWorker(0xc000052a00)
	/usr/lib/go-1.13/src/runtime/mgc.go:1846 +0xff fp=0xc0004ae7d8 sp=0xc0004ae760 pc=0x41935f
runtime.goexit()
	/usr/lib/go-1.13/src/runtime/asm_amd64.s:1357 +0x1 fp=0xc0004ae7e0 sp=0xc0004ae7d8 pc=0x455b21
created by runtime.gcBgMarkStartWorkers
	/usr/lib/go-1.13/src/runtime/mgc.go:1794 +0x77

goroutine 372 [GC worker (idle)]:
runtime.gopark(0x4ea188, 0xc0003eac20, 0x1418, 0x0)
	/usr/lib/go-1.13/src/runtime/proc.go:304 +0xe0 fp=0xc0004aa760 sp=0xc0004aa740 pc=0x42c090
runtime.gcBgMarkWorker(0xc000057400)
	/usr/lib/go-1.13/src/runtime/mgc.go:1846 +0xff fp=0xc0004aa7d8 sp=0xc0004aa760 pc=0x41935f
runtime.goexit()
	/usr/lib/go-1.13/src/runtime/asm_amd64.s:1357 +0x1 fp=0xc0004aa7e0 sp=0xc0004aa7d8 pc=0x455b21
created by runtime.gcBgMarkStartWorkers
	/usr/lib/go-1.13/src/runtime/mgc.go:1794 +0x77

goroutine 373 [GC worker (idle)]:
runtime.gopark(0x4ea188, 0xc0003eac40, 0x1418, 0x0)
	/usr/lib/go-1.13/src/runtime/proc.go:304 +0xe0 fp=0xc0004aaf60 sp=0xc0004aaf40 pc=0x42c090
runtime.gcBgMarkWorker(0xc00005c000)
	/usr/lib/go-1.13/src/runtime/mgc.go:1846 +0xff fp=0xc0004aafd8 sp=0xc0004aaf60 pc=0x41935f
runtime.goexit()
	/usr/lib/go-1.13/src/runtime/asm_amd64.s:1357 +0x1 fp=0xc0004aafe0 sp=0xc0004aafd8 pc=0x455b21
created by runtime.gcBgMarkStartWorkers
	/usr/lib/go-1.13/src/runtime/mgc.go:1794 +0x77

goroutine 389 [GC worker (idle)]:
runtime.gopark(0x4ea188, 0xc0003eac50, 0x1418, 0x0)
	/usr/lib/go-1.13/src/runtime/proc.go:304 +0xe0 fp=0xc0004b8760 sp=0xc0004b8740 pc=0x42c090
runtime.gcBgMarkWorker(0xc00005e500)
	/usr/lib/go-1.13/src/runtime/mgc.go:1846 +0xff fp=0xc0004b87d8 sp=0xc0004b8760 pc=0x41935f
runtime.goexit()
	/usr/lib/go-1.13/src/runtime/asm_amd64.s:1357 +0x1 fp=0xc0004b87e0 sp=0xc0004b87d8 pc=0x455b21
created by runtime.gcBgMarkStartWorkers
	/usr/lib/go-1.13/src/runtime/mgc.go:1794 +0x77

goroutine 374 [GC worker (idle)]:
runtime.gopark(0x4ea188, 0xc0003eac60, 0x1418, 0x0)
	/usr/lib/go-1.13/src/runtime/proc.go:304 +0xe0 fp=0xc0004ab760 sp=0xc0004ab740 pc=0x42c090
runtime.gcBgMarkWorker(0xc000060a00)
	/usr/lib/go-1.13/src/runtime/mgc.go:1846 +0xff fp=0xc0004ab7d8 sp=0xc0004ab760 pc=0x41935f
runtime.goexit()
	/usr/lib/go-1.13/src/runtime/asm_amd64.s:1357 +0x1 fp=0xc0004ab7e0 sp=0xc0004ab7d8 pc=0x455b21
created by runtime.gcBgMarkStartWorkers
	/usr/lib/go-1.13/src/runtime/mgc.go:1794 +0x77

goroutine 390 [GC worker (idle)]:
runtime.gopark(0x4ea188, 0xc0003eac70, 0x1418, 0x0)
	/usr/lib/go-1.13/src/runtime/proc.go:304 +0xe0 fp=0xc0004b8f60 sp=0xc0004b8f40 pc=0x42c090
runtime.gcBgMarkWorker(0xc000062f00)
	/usr/lib/go-1.13/src/runtime/mgc.go:1846 +0xff fp=0xc0004b8fd8 sp=0xc0004b8f60 pc=0x41935f
runtime.goexit()
	/usr/lib/go-1.13/src/runtime/asm_amd64.s:1357 +0x1 fp=0xc0004b8fe0 sp=0xc0004b8fd8 pc=0x455b21
created by runtime.gcBgMarkStartWorkers
	/usr/lib/go-1.13/src/runtime/mgc.go:1794 +0x77

goroutine 375 [GC worker (idle)]:
runtime.gopark(0x4ea188, 0xc0003eac80, 0x1418, 0x0)
	/usr/lib/go-1.13/src/runtime/proc.go:304 +0xe0 fp=0xc0004abf60 sp=0xc0004abf40 pc=0x42c090
runtime.gcBgMarkWorker(0xc000065400)
	/usr/lib/go-1.13/src/runtime/mgc.go:1846 +0xff fp=0xc0004abfd8 sp=0xc0004abf60 pc=0x41935f
runtime.goexit()
	/usr/lib/go-1.13/src/runtime/asm_amd64.s:1357 +0x1 fp=0xc0004abfe0 sp=0xc0004abfd8 pc=0x455b21
created by runtime.gcBgMarkStartWorkers
	/usr/lib/go-1.13/src/runtime/mgc.go:1794 +0x77

goroutine 391 [GC worker (idle)]:
runtime.gopark(0x4ea188, 0xc0004920a0, 0x1418, 0x0)
	/usr/lib/go-1.13/src/runtime/proc.go:304 +0xe0 fp=0xc0004b9760 sp=0xc0004b9740 pc=0x42c090
runtime.gcBgMarkWorker(0xc000067900)
	/usr/lib/go-1.13/src/runtime/mgc.go:1846 +0xff fp=0xc0004b97d8 sp=0xc0004b9760 pc=0x41935f
runtime.goexit()
	/usr/lib/go-1.13/src/runtime/asm_amd64.s:1357 +0x1 fp=0xc0004b97e0 sp=0xc0004b97d8 pc=0x455b21
created by runtime.gcBgMarkStartWorkers
	/usr/lib/go-1.13/src/runtime/mgc.go:1794 +0x77

goroutine 392 [GC worker (idle)]:
runtime.gopark(0x4ea188, 0xc0003eac90, 0x1418, 0x0)
	/usr/lib/go-1.13/src/runtime/proc.go:304 +0xe0 fp=0xc0004b9f60 sp=0xc0004b9f40 pc=0x42c090
runtime.gcBgMarkWorker(0xc00006a000)
	/usr/lib/go-1.13/src/runtime/mgc.go:1846 +0xff fp=0xc0004b9fd8 sp=0xc0004b9f60 pc=0x41935f
runtime.goexit()
	/usr/lib/go-1.13/src/runtime/asm_amd64.s:1357 +0x1 fp=0xc0004b9fe0 sp=0xc0004b9fd8 pc=0x455b21
created by runtime.gcBgMarkStartWorkers
	/usr/lib/go-1.13/src/runtime/mgc.go:1794 +0x77

goroutine 376 [GC worker (idle)]:
runtime.gopark(0x4ea188, 0xc0003eaca0, 0x1418, 0x0)
	/usr/lib/go-1.13/src/runtime/proc.go:304 +0xe0 fp=0xc0004ac760 sp=0xc0004ac740 pc=0x42c090
runtime.gcBgMarkWorker(0xc00006c500)
	/usr/lib/go-1.13/src/runtime/mgc.go:1846 +0xff fp=0xc0004ac7d8 sp=0xc0004ac760 pc=0x41935f
runtime.goexit()
	/usr/lib/go-1.13/src/runtime/asm_amd64.s:1357 +0x1 fp=0xc0004ac7e0 sp=0xc0004ac7d8 pc=0x455b21
created by runtime.gcBgMarkStartWorkers
	/usr/lib/go-1.13/src/runtime/mgc.go:1794 +0x77

goroutine 393 [GC worker (idle)]:
runtime.gopark(0x4ea188, 0xc0003eacb0, 0x1418, 0x0)
	/usr/lib/go-1.13/src/runtime/proc.go:304 +0xe0 fp=0xc0004ba760 sp=0xc0004ba740 pc=0x42c090
runtime.gcBgMarkWorker(0xc00006ea00)
	/usr/lib/go-1.13/src/runtime/mgc.go:1846 +0xff fp=0xc0004ba7d8 sp=0xc0004ba760 pc=0x41935f
runtime.goexit()
	/usr/lib/go-1.13/src/runtime/asm_amd64.s:1357 +0x1 fp=0xc0004ba7e0 sp=0xc0004ba7d8 pc=0x455b21
created by runtime.gcBgMarkStartWorkers
	/usr/lib/go-1.13/src/runtime/mgc.go:1794 +0x77

goroutine 423 [GC worker (idle)]:
runtime.gopark(0x4ea188, 0xc0003eacc0, 0x1418, 0x0)
	/usr/lib/go-1.13/src/runtime/proc.go:304 +0xe0 fp=0xc0004a4f60 sp=0xc0004a4f40 pc=0x42c090
runtime.gcBgMarkWorker(0xc000070f00)
	/usr/lib/go-1.13/src/runtime/mgc.go:1846 +0xff fp=0xc0004a4fd8 sp=0xc0004a4f60 pc=0x41935f
runtime.goexit()
	/usr/lib/go-1.13/src/runtime/asm_amd64.s:1357 +0x1 fp=0xc0004a4fe0 sp=0xc0004a4fd8 pc=0x455b21
created by runtime.gcBgMarkStartWorkers
	/usr/lib/go-1.13/src/runtime/mgc.go:1794 +0x77

goroutine 394 [GC worker (idle)]:
runtime.gopark(0x4ea188, 0xc0003eacd0, 0x1418, 0x0)
	/usr/lib/go-1.13/src/runtime/proc.go:304 +0xe0 fp=0xc0004baf60 sp=0xc0004baf40 pc=0x42c090
runtime.gcBgMarkWorker(0xc000073400)
	/usr/lib/go-1.13/src/runtime/mgc.go:1846 +0xff fp=0xc0004bafd8 sp=0xc0004baf60 pc=0x41935f
runtime.goexit()
	/usr/lib/go-1.13/src/runtime/asm_amd64.s:1357 +0x1 fp=0xc0004bafe0 sp=0xc0004bafd8 pc=0x455b21
created by runtime.gcBgMarkStartWorkers
	/usr/lib/go-1.13/src/runtime/mgc.go:1794 +0x77

goroutine 424 [GC worker (idle)]:
runtime.gopark(0x4ea188, 0xc0003eace0, 0x1418, 0x0)
	/usr/lib/go-1.13/src/runtime/proc.go:304 +0xe0 fp=0xc0004a5760 sp=0xc0004a5740 pc=0x42c090
runtime.gcBgMarkWorker(0xc000075900)
	/usr/lib/go-1.13/src/runtime/mgc.go:1846 +0xff fp=0xc0004a57d8 sp=0xc0004a5760 pc=0x41935f
runtime.goexit()
	/usr/lib/go-1.13/src/runtime/asm_amd64.s:1357 +0x1 fp=0xc0004a57e0 sp=0xc0004a57d8 pc=0x455b21
created by runtime.gcBgMarkStartWorkers
	/usr/lib/go-1.13/src/runtime/mgc.go:1794 +0x77
Aborted (core dumped)
@mvdan

This comment has been minimized.

Copy link
Member

@mvdan mvdan commented Jan 3, 2020

/cc @aclements @ianlancetaylor @rhysh, since all three were involved in the original issue and CL

@ianlancetaylor ianlancetaylor changed the title Random crash in golang, reproducer same as #20427 runtime: runtime crashes, reproducer same as #20427 Jan 3, 2020
@ianlancetaylor ianlancetaylor added this to the Go1.15 milestone Jan 3, 2020
@ianlancetaylor

This comment has been minimized.

Copy link
Contributor

@ianlancetaylor ianlancetaylor commented Jan 3, 2020

Tell us more about the machines on which the program crashes. Is there any thing that distinguishes systems where the program crashes from systems where it does not crash?

Can you attach the output of strace -f for a system where the program reliably crashes?

@t57root

This comment has been minimized.

Copy link
Author

@t57root t57root commented Jan 6, 2020

@ianlancetaylor
when run the program with strace attached, it seems it takes more time to crash.
the log is fairly large and I uploaded the latter part of the log to pastebin.

Also, go build on that machine sometimes crashes too.

@ianlancetaylor

This comment has been minimized.

Copy link
Contributor

@ianlancetaylor ianlancetaylor commented Jan 6, 2020

Tell us more about the machines on which the program crashes. Is there any thing that distinguishes systems where the program crashes from systems where it does not crash?

@ulikunitz

This comment has been minimized.

Copy link
Contributor

@ulikunitz ulikunitz commented Jan 6, 2020

3.18 is not the default kernel for CentOS 7, 3.10 is. I guess the kernel is provided by http://elrepo.org. Since you also state that go build crashes, there might be a problem with the specific kernel build used on your servers.

@ianlancetaylor

This comment has been minimized.

Copy link
Contributor

@ianlancetaylor ianlancetaylor commented Jan 6, 2020

Thanks for the strace -f output. Is the failing PC consistently 0x4051d8?

That address suggests that the crash is happening because gp is nil on line 295 of runtime/chan.go in the function runtime.send. The relevant part of the function is

	if sg.elem != nil {
		sendDirect(c.elemtype, sg, ep)
		sg.elem = nil
	}
	gp := sg.g
	unlockf()
	gp.param = unsafe.Pointer(sg)
	if sg.releasetime != 0 {
		sg.releasetime = cputicks()
	}
	goready(gp, skip+1)

The crash is happening on the line gp.param = unsafe.Pointer(sg). The value of gp is nil. Needless to say, this should be impossible.

Can you see if it makes any difference if you change syscall/exec_linux.go to change the line

	hasRawVforkSyscall := runtime.GOARCH == "amd64" || runtime.GOARCH == "ppc64" || runtime.GOARCH == "s390x"

to

	hasRawVforkSyscall := false

? Thanks.

@t57root

This comment has been minimized.

Copy link
Author

@t57root t57root commented Jan 7, 2020

@ulikunitz
@ianlancetaylor

the server is initialized and configured by our operation team. After asking our kernel team, they said it’s a bug in kernel 3.18 regarding ‘xsaves’. And they have confirmed that golang programs frequently encounter this bug. If this is the reason, then I think there should be no business with golang itself at all.

For the integrality of this discussion: failing PC is not consistently 0x4051d8, but it‘s 0x4051d8 in most cases (e.g. 0x492791 and 0x4a4090 also appeared). After setting hasRawVforkSyscall to false, the program still crashes

@ianlancetaylor

This comment has been minimized.

Copy link
Contributor

@ianlancetaylor ianlancetaylor commented Jan 7, 2020

OK, thanks. Sounds like we can close this issue then.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
4 participants
You can’t perform that action at this time.