Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

runtime: crash on linux/arm64 with "invalid p state" on 1.13 #39252

Open
choury opened this issue May 26, 2020 · 6 comments
Open

runtime: crash on linux/arm64 with "invalid p state" on 1.13 #39252

choury opened this issue May 26, 2020 · 6 comments

Comments

@choury
Copy link

@choury choury commented May 26, 2020

What version of Go are you using (go version)?

$ go version
go version go1.13.10 linux/arm64

Does this issue reproduce with the latest release?

Not happened with 1.14, but also in 1.13.11

What operating system and processor architecture are you using (go env)?

go env Output
$ go env
GO111MODULE=""
GOARCH="arm64"
GOBIN=""
GOCACHE="/data/choury/.cache/go-build"
GOENV="/data/choury/.config/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="arm64"
GOHOSTOS="linux"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/data/choury/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/data/choury/.local/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/data/choury/.local/go/pkg/tool/linux_arm64"
GCCGO="gccgo"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build335762780=/tmp/go-build -gno-record-gcc-switches"

What did you do?

Run this code

package main

import (
        "io/ioutil"
        "math/rand"
        "os/exec"
        "time"
)

func main() {
        for i := 0; i < 100; i++ {
                go func() {
                        for {
                                d := time.Duration(rand.Intn(100)+1) * time.Millisecond
                                time.Sleep(d)
                                cmd := exec.Command("ls")
                                cmd.CombinedOutput()
                        }
                }()
                go func() {
                        for {
                                d := time.Duration(rand.Intn(100)+1) * time.Millisecond
                                time.Sleep(d)
                                ioutil.ReadFile("/proc/cpuinfo")
                        }
                }()
        }
        select {}
}

What did you expect to see?

No error, run forever...

What did you see instead?

crash with:

releasep: m=0x4000f82000 m->p=0x400003a500 p->m=0x0 m->mcache=0xfd4da32dac00 p->mcache=0xfd4da32dac00 p->status=0
fatal error: releasep: invalid p state
runtime: newstack at os/exec.(*Cmd).Run+0x78 sp=0x4000baef60 stack=[0x4000bae000, 0x4000baf000]
        morebuf={pc:0x88f2c sp:0x4000baef60 lr:0x0}
        sched={pc:0x883a8 sp:0x4000baef60 lr:0x88f2c ctxt:0x0}
os/exec.(*Cmd).CombinedOutput(0x4000934b00, 0x2, 0x0, 0x0, 0x0, 0x4000934b00)
        /usr/local/go/src/os/exec/exec.go:566 +0x8c fp=0x4000baef90 sp=0x4000baef60 pc=0x88f2c
main.main.func1()
        /crash/main.go:17 +0x80 fp=0x4000baefd0 sp=0x4000baef90 pc=0x8a0a0
runtime.goexit()
        /usr/local/go/src/runtime/asm_arm64.s:1128 +0x4 fp=0x4000baefd0 sp=0x4000baefd0 pc=0x60494
created by main.main
        /crash/main.go:12 +0x3c
fatal error: runtime: stack split at bad time
signal: segmentation fault (core dumped)
@andybons
Copy link
Member

@andybons andybons commented May 26, 2020

If I’m understanding correctly, this reproduces reliably on 1.13, but not 1.14? Is that right?

@andybons andybons added this to the Unplanned milestone May 26, 2020
@andybons
Copy link
Member

@andybons andybons commented May 26, 2020

@andybons andybons changed the title go 1.13 crash on linux/arm64 with "invalid p state" runtime: crash on linux/arm64 with "invalid p state" on 1.13 May 26, 2020
@choury
Copy link
Author

@choury choury commented May 27, 2020

If I’m understanding correctly, this reproduces reliably on 1.13, but not 1.14? Is that right?

Yes

@shawn-xdji
Copy link
Contributor

@shawn-xdji shawn-xdji commented Jun 1, 2020

@choury I tried the case on various arm64 machines with 'go1.13.11' but it didn't crash after 10min or so, how long should I expect the error to happen? Once ran into the same error with 'go tool dist test' but never reproduced it again, so want to try with your case. Thanks.

@choury
Copy link
Author

@choury choury commented Jun 1, 2020

@choury I tried the case on various arm64 machines with 'go1.13.11' but it didn't crash after 10min or so, how long should I expect the error to happen? Once ran into the same error with 'go tool dist test' but never reproduced it again, so want to try with your case. Thanks.

go tool dist test:

##### Testing packages.
ok  	archive/tar	0.189s
ok  	archive/zip	0.508s
ok  	bufio	0.632s
ok  	bytes	0.697s
ok  	compress/bzip2	0.087s
ok  	compress/flate	0.677s
ok  	compress/gzip	1.007s
ok  	compress/lzw	0.517s
ok  	compress/zlib	0.560s
ok  	container/heap	0.026s
ok  	container/list	0.507s
ok  	container/ring	0.570s
ok  	context	0.988s
ok  	crypto	0.554s
ok  	crypto/aes	0.586s
ok  	crypto/cipher	0.552s
ok  	crypto/des	0.574s
ok  	crypto/dsa	0.027s
ok  	crypto/ecdsa	0.160s
ok  	crypto/ed25519	0.067s
ok  	crypto/elliptic	0.593s
ok  	crypto/hmac	0.588s
ok  	crypto/internal/subtle	0.583s
ok  	crypto/md5	0.523s
ok  	crypto/rand	0.063s
ok  	crypto/rc4	1.188s
ok  	crypto/rsa	0.591s
ok  	crypto/sha1	0.054s
ok  	crypto/sha256	0.550s
ok  	crypto/sha512	0.470s
ok  	crypto/subtle	0.571s
ok  	crypto/tls	0.776s
ok  	crypto/x509	0.423s
ok  	database/sql	0.588s
ok  	database/sql/driver	0.577s
ok  	debug/dwarf	1.128s
ok  	debug/elf	0.149s
ok  	debug/gosym	0.507s
ok  	debug/macho	0.077s
ok  	debug/pe	0.564s
ok  	debug/plan9obj	0.016s
ok  	encoding/ascii85	0.565s
ok  	encoding/asn1	0.575s
ok  	encoding/base32	0.542s
ok  	encoding/base64	0.549s
ok  	encoding/binary	0.547s
ok  	encoding/csv	0.543s
ok  	encoding/gob	0.062s
ok  	encoding/hex	1.123s
ok  	encoding/json	0.085s
ok  	encoding/pem	0.046s
ok  	encoding/xml	0.543s
ok  	errors	0.545s
ok  	expvar	0.016s
ok  	flag	0.026s
ok  	fmt	0.570s
ok  	go/ast	0.263s
ok  	go/build	0.143s
ok  	go/constant	0.554s
ok  	go/doc	0.600s
ok  	go/format	0.020s
ok  	go/importer	0.371s
ok  	go/internal/gccgoimporter	0.038s
ok  	go/internal/gcimporter	0.606s
ok  	go/internal/srcimporter	1.178s
ok  	go/parser	0.615s
ok  	go/printer	0.856s
ok  	go/scanner	0.543s
ok  	go/token	0.233s
ok  	go/types	1.016s
ok  	hash	0.520s
ok  	hash/adler32	0.563s
ok  	hash/crc32	0.534s
ok  	hash/crc64	0.072s
ok  	hash/fnv	0.019s
ok  	html	0.051s
ok  	html/template	0.527s
ok  	image	0.112s
ok  	image/color	0.581s
ok  	image/draw	0.551s
ok  	image/gif	0.865s
ok  	image/jpeg	0.733s
ok  	image/png	0.550s
ok  	index/suffixarray	0.702s
ok  	internal/cpu	0.146s
ok  	internal/fmtsort	0.558s
ok  	internal/poll	0.077s
ok  	internal/reflectlite	0.554s
ok  	internal/singleflight	0.581s
ok  	internal/trace	0.621s
ok  	internal/xcoff	0.560s
ok  	io	0.635s
ok  	io/ioutil	0.593s
ok  	log	0.509s
ok  	log/syslog	1.251s
ok  	math	0.521s
ok  	math/big	0.840s
ok  	math/bits	0.208s
ok  	math/cmplx	0.536s
ok  	math/rand	0.568s
ok  	mime	0.515s
ok  	mime/multipart	0.194s
ok  	mime/quotedprintable	0.298s
ok  	net	2.931s
ok  	net/http	2.843s
ok  	net/http/cgi	0.253s
ok  	net/http/cookiejar	0.023s
ok  	net/http/fcgi	0.036s
ok  	net/http/httptest	0.040s
ok  	net/http/httptrace	0.016s
ok  	net/http/httputil	0.061s
ok  	net/http/internal	0.575s
ok  	net/http/pprof	2.021s
ok  	net/internal/socktest	0.591s
ok  	net/mail	0.025s
ok  	net/rpc	0.036s
ok  	net/rpc/jsonrpc	0.023s
ok  	net/smtp	0.050s
ok  	net/textproto	0.158s
ok  	net/url	0.107s
ok  	os	2.131s
ok  	os/exec	0.593s
ok  	os/signal	4.676s
ok  	os/user	0.145s
ok  	path	0.023s
ok  	path/filepath	0.526s
ok  	reflect	0.745s
ok  	regexp	0.646s
ok  	regexp/syntax	0.866s
ok  	runtime	14.929s
ok  	runtime/debug	0.071s
ok  	runtime/internal/atomic	0.530s
ok  	runtime/internal/math	0.040s
ok  	runtime/internal/sys	0.043s
ok  	runtime/pprof	5.884s
ok  	runtime/pprof/internal/profile	0.548s
ok  	runtime/trace	0.715s
ok  	sort	0.534s
ok  	strconv	1.010s
ok  	strings	0.578s
ok  	sync	0.853s
ok  	sync/atomic	0.687s
ok  	syscall	0.073s
ok  	testing	0.658s
ok  	testing/quick	0.573s
ok  	text/scanner	0.521s
ok  	text/tabwriter	0.514s
ok  	text/template	0.502s
ok  	text/template/parse	0.470s
ok  	time	3.062s
ok  	unicode	0.465s
ok  	unicode/utf16	0.524s
ok  	unicode/utf8	0.475s
ok  	cmd/addr2line	1.679s
ok  	cmd/api	0.543s
ok  	cmd/asm/internal/asm	0.690s
ok  	cmd/asm/internal/lex	0.032s
ok  	cmd/compile	0.016s
ok  	cmd/compile/internal/gc	6.612s
ok  	cmd/compile/internal/ssa	0.763s
ok  	cmd/compile/internal/syntax	0.530s
ok  	cmd/compile/internal/test	0.536s [no tests to run]
ok  	cmd/compile/internal/types	0.260s
ok  	cmd/cover	1.974s
ok  	cmd/doc	0.332s
ok  	cmd/fix	0.620s
ok  	cmd/go	33.515s
ok  	cmd/go/internal/auth	0.014s
ok  	cmd/go/internal/cache	0.623s
ok  	cmd/go/internal/dirhash	0.463s
ok  	cmd/go/internal/generate	0.045s
ok  	cmd/go/internal/get	0.035s
ok  	cmd/go/internal/imports	0.229s
ok  	cmd/go/internal/load	0.119s
ok  	cmd/go/internal/lockedfile	0.217s
ok  	cmd/go/internal/lockedfile/internal/filelock	0.311s
ok  	cmd/go/internal/modconv	0.027s
ok  	cmd/go/internal/modfetch	0.022s
ok  	cmd/go/internal/modfetch/codehost	0.243s
ok  	cmd/go/internal/modfile	0.335s
ok  	cmd/go/internal/modload	0.022s
ok  	cmd/go/internal/module	0.322s
ok  	cmd/go/internal/mvs	0.536s
ok  	cmd/go/internal/note	0.238s
ok  	cmd/go/internal/par	0.331s
ok  	cmd/go/internal/renameio	0.311s
ok  	cmd/go/internal/search	0.221s
ok  	cmd/go/internal/semver	0.428s
ok  	cmd/go/internal/sumweb	0.053s
ok  	cmd/go/internal/tlog	0.331s
ok  	cmd/go/internal/txtar	0.274s
ok  	cmd/go/internal/web	0.031s
ok  	cmd/go/internal/work	0.113s
ok  	cmd/gofmt	0.251s
ok  	cmd/internal/buildid	0.370s
ok  	cmd/internal/dwarf	0.061s
ok  	cmd/internal/edit	0.169s
ok  	cmd/internal/goobj	0.891s
ok  	cmd/internal/obj	0.028s
ok  	cmd/internal/obj/arm64	0.221s
ok  	cmd/internal/obj/x86	1.875s
ok  	cmd/internal/objabi	0.039s
ok  	cmd/internal/src	0.140s
ok  	cmd/internal/test2json	0.234s
ok  	cmd/link	6.006s
ok  	cmd/link/internal/ld	0.758s
ok  	cmd/link/internal/sym	0.091s
ok  	cmd/nm	2.166s
ok  	cmd/objdump	1.833s
ok  	cmd/pack	1.278s
ok  	cmd/trace	0.043s
ok  	cmd/vet	3.957s

##### os/user with tag osusergo
ok  	os/user	0.003s

##### GOMAXPROCS=2 runtime -cpu=1,2,4 -quick
ok  	runtime	4.667s

##### cmd/go terminal test
skipping terminal test; stdout/stderr not terminals

##### Testing without libgcc.
ok  	crypto/x509	0.325s
ok  	net	0.011s
ok  	os/user	0.005s

##### internal linking of -buildmode=pie
ok  	reflect	1.488s
ok  	os/user	0.337s

##### sync -cpu=10
ok  	sync	0.277s

##### Testing race detector
ok  	runtime/race	10.073s
ok  	flag	1.016s
ok  	net	1.082s
ok  	os	1.093s
ok  	os/exec	3.042s
ok  	encoding/gob	1.018s
ok  	flag	1.016s
ok  	os/exec	3.045s

##### ../misc/cgo/stdio
PASS

##### ../misc/cgo/life
PASS

##### ../misc/cgo/test
PASS
ok  	misc/cgo/test	3.219s
PASS
ok  	misc/cgo/test	3.107s
PASS
ok  	misc/cgo/test	3.305s
PASS
ok  	misc/cgo/testtls	0.005s
PASS
ok  	misc/cgo/testtls	0.005s
PASS
ok  	misc/cgo/testtls	0.005s
PASS
ok  	misc/cgo/nocgo	0.004s
PASS
ok  	misc/cgo/nocgo	0.005s
PASS
ok  	misc/cgo/nocgo	0.005s
PASS
ok  	misc/cgo/test	3.081s
PASS
ok  	misc/cgo/test	3.257s
PASS
ok  	misc/cgo/test	3.063s
PASS
ok  	misc/cgo/testtls	0.005s
PASS
ok  	misc/cgo/nocgo	0.006s

##### ../misc/cgo/testgodefs

##### ../misc/cgo/testso
ok  	misc/cgo/testso	0.820s

##### ../misc/cgo/testsovar
ok  	misc/cgo/testsovar	0.904s

##### ../misc/cgo/testcshared
SKIP - short mode and $GO_BUILDER_NAME not set

##### ../misc/cgo/testshared
ok  	misc/cgo/testshared	31.782s

##### ../misc/cgo/testsanitizers
PASS

##### ../misc/cgo/errors
PASS

##### ../misc/cgo/testsigfwd

##### ../test/bench/go1

##### ../test

##### API check
Go version is "go1.13.11", ignoring -next /usr/local/go/api/next.txt

ALL TESTS PASSED

It will crash within 10 min in docker golang:1.13.11 image on my machine.

@shawn-xdji
Copy link
Contributor

@shawn-xdji shawn-xdji commented Jun 3, 2020

@choury Thanks, I verified the case in golang:1.13.11 containers on two different models of machine, it didn't crash after dozens of minutes. Will update later if I'm able to reproduce the issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
3 participants
You can’t perform that action at this time.