New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

runtime: "runtime: unexpected return pc for runtime.gopark called from 0x0" #23561

Open
xiaoyulei opened this Issue Jan 26, 2018 · 15 comments

Comments

Projects
None yet
5 participants
@xiaoyulei

xiaoyulei commented Jan 26, 2018

What version of Go are you using (go version)?

go version go1.9.2 linux/amd64

Does this issue reproduce with the latest release?

I am not sure, it is not happen everytime.

What operating system and processor architecture are you using (go env)?

[xiaoyulei@dev package]$ go env
GOARCH="amd64"
GOBIN=""
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/mnt/hdd1/work/R"
GORACE=""
GOROOT="/mnt/hdd1/tools/go"
GOTOOLDIR="/mnt/hdd1/tools/go/pkg/tool/linux_amd64"
GCCGO="gccgo"
CC="gcc"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build058191915=/tmp/go-build -gno-record-gcc-switches"
CXX="g++"
CGO_ENABLED="1"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"

What did you do?

this happen when process in quit progress. I notify signal 15 to program.

What did you expect to see?

not crash

What did you see instead?

runtime: unexpected return pc for runtime.gopark called from 0x0
fatal error: unknown caller pc

runtime stack:
runtime.throw(0x7f41582ace04, 0x11)
	/home/ci/jks/go/src/runtime/panic.go:605 +0x97
runtime.gentraceback(0xffffffffffffffff, 0xffffffffffffffff, 0x0, 0xc420001680, 0x0, 0x0, 0x7fffffff, 0x7f3b87ffea28, 0x0, 0x0, ...)
	/home/ci/jks/go/src/runtime/traceback.go:285 +0x19b6
runtime.scanstack(0xc420001680, 0xc420023860)
	/home/ci/jks/go/src/runtime/mgcmark.go:779 +0x158
runtime.scang(0xc420001680, 0xc420023860)
	/home/ci/jks/go/src/runtime/proc.go:869 +0x1f8
runtime.markroot.func1()
	/home/ci/jks/go/src/runtime/mgcmark.go:264 +0x71
runtime.systemstack(0x7f3b87ffeb48)
	/home/ci/jks/go/src/runtime/asm_amd64.s:360 +0xa1
runtime.markroot(0xc420023860, 0x7f4100000009)
	/home/ci/jks/go/src/runtime/mgcmark.go:245 +0x30a
runtime.gcDrain(0xc420023860, 0xd)
	/home/ci/jks/go/src/runtime/mgcmark.go:912 +0xd8
runtime.gcBgMarkWorker.func2()
	/home/ci/jks/go/src/runtime/mgc.go:1799 +0x191
runtime.systemstack(0xc420023900)
	/home/ci/jks/go/src/runtime/asm_amd64.s:344 +0x72
runtime.mstart()
	/home/ci/jks/go/src/runtime/proc.go:1135

goroutine 173 [GC worker (idle)]:
runtime.systemstack_switch()
	/home/ci/jks/go/src/runtime/asm_amd64.s:298 fp=0xc420381f48 sp=0xc420381f40 pc=0x7f4157ef4400
runtime.gcBgMarkWorker(0xc420022600)
	/home/ci/jks/go/src/runtime/mgc.go:1763 +0x20a fp=0xc420381fd8 sp=0xc420381f48 pc=0x7f4157eb81da
runtime.goexit()
	/home/ci/jks/go/src/runtime/asm_amd64.s:2337 +0x1 fp=0xc420381fe0 sp=0xc420381fd8 pc=0x7f4157ef7041
created by runtime.gcBgMarkStartWorkers
	/home/ci/jks/go/src/runtime/mgc.go:1658 +0x80

goroutine 5 [select]:
github.com/coreos/etcd/clientv3.(*simpleBalancer).updateNotifyLoop(0xc4200c9220)
	/home/ci/jks/workspace/CDB/RR/src/github.com/coreos/etcd/clientv3/balancer.go:211 +0x38f
created by github.com/coreos/etcd/clientv3.newSimpleBalancer
	/home/ci/jks/workspace/CDB/RR/src/github.com/coreos/etcd/clientv3/balancer.go:92 +0x363

goroutine 9 [chan receive]:
google.golang.org/grpc.(*ClientConn).lbWatcher(0xc420001c80)
	/home/ci/jks/workspace/CDB/RR/src/google.golang.org/grpc/clientconn.go:481 +0x742
created by google.golang.org/grpc.DialContext
	/home/ci/jks/workspace/CDB/RR/src/google.golang.org/grpc/clientconn.go:424 +0x577

goroutine 8 [select]:
google.golang.org/grpc.(*addrConn).transportMonitor(0xc4200bcd00)
	/home/ci/jks/workspace/CDB/RR/src/google.golang.org/grpc/clientconn.go:869 +0x1e0
google.golang.org/grpc.(*ClientConn).resetAddrConn.func1(0xc4200bcd00)
	/home/ci/jks/workspace/CDB/RR/src/google.golang.org/grpc/clientconn.go:614 +0x1ca
created by google.golang.org/grpc.(*ClientConn).resetAddrConn
	/home/ci/jks/workspace/CDB/RR/src/google.golang.org/grpc/clientconn.go:605 +0x372

goroutine 20 [IO wait]:
internal/poll.runtime_pollWait(0x7f4158c5ef70, 0x72, 0x0)
	/home/ci/jks/go/src/runtime/netpoll.go:173 +0x59
internal/poll.(*pollDesc).wait(0xc4201d2118, 0x72, 0xffffffffffffff00, 0x7f4158888400, 0x7f4158884178)
	/home/ci/jks/go/src/internal/poll/fd_poll_runtime.go:85 +0xb0
internal/poll.(*pollDesc).waitRead(0xc4201d2118, 0xc4201e2000, 0x8000, 0x8000)
	/home/ci/jks/go/src/internal/poll/fd_poll_runtime.go:90 +0x3f
internal/poll.(*FD).Read(0xc4201d2100, 0xc4201e2000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
	/home/ci/jks/go/src/internal/poll/fd_unix.go:126 +0x18c
net.(*netFD).Read(0xc4201d2100, 0xc4201e2000, 0x8000, 0x8000, 0xc42007dca0, 0x7f4157ec80da, 0x0)
	/home/ci/jks/go/src/net/fd_unix.go:202 +0x54
net.(*conn).Read(0xc4201dc000, 0xc4201e2000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
	/home/ci/jks/go/src/net/net.go:176 +0x6f
bufio.(*Reader).Read(0xc4201c6240, 0xc4201f2038, 0x9, 0x9, 0xc4201da000, 0x7f4158687880, 0xc4201ac040)
	/home/ci/jks/go/src/bufio/bufio.go:213 +0x30d
io.ReadAtLeast(0x7f4158885800, 0xc4201c6240, 0xc4201f2038, 0x9, 0x9, 0x9, 0x7f41000002b9, 0xc400000000, 0x0)
	/home/ci/jks/go/src/io/io.go:309 +0x88
io.ReadFull(0x7f4158885800, 0xc4201c6240, 0xc4201f2038, 0x9, 0x9, 0xe790e7e8dcaebe01, 0xe7000000000002b9, 0x0)
	/home/ci/jks/go/src/io/io.go:327 +0x5a
golang.org/x/net/http2.readFrameHeader(0xc4201f2038, 0x9, 0x9, 0x7f4158885800, 0xc4201c6240, 0x0, 0x2b900000000, 0xc400000027, 0x0)
	/home/ci/jks/workspace/CDB/RR/src/golang.org/x/net/http2/frame.go:237 +0x7d
golang.org/x/net/http2.(*Framer).ReadFrame(0xc4201f2000, 0x7f4158889140, 0xc4203e1350, 0x0, 0x0)
	/home/ci/jks/workspace/CDB/RR/src/golang.org/x/net/http2/frame.go:492 +0xa6
google.golang.org/grpc/transport.(*framer).readFrame(0xc4201ca120, 0xc4203e1350, 0xc4203e1350, 0x0, 0x0)
	/home/ci/jks/workspace/CDB/RR/src/google.golang.org/grpc/transport/http_util.go:506 +0x31
google.golang.org/grpc/transport.(*http2Client).reader(0xc4201da180)
	/home/ci/jks/workspace/CDB/RR/src/google.golang.org/grpc/transport/http2_client.go:1039 +0xc2
created by google.golang.org/grpc/transport.newHTTP2Client
	/home/ci/jks/workspace/CDB/RR/src/google.golang.org/grpc/transport/http2_client.go:247 +0xb8f

goroutine 21 [select]:
google.golang.org/grpc/transport.(*http2Client).controller(0xc4201da180)
	/home/ci/jks/workspace/CDB/RR/src/google.golang.org/grpc/transport/http2_client.go:1112 +0x144
created by google.golang.org/grpc/transport.newHTTP2Client
	/home/ci/jks/workspace/CDB/RR/src/google.golang.org/grpc/transport/http2_client.go:277 +0xf4e

goroutine 36 [IO wait]:
internal/poll.runtime_pollWait(0x7f4158c5eeb0, 0x72, 0x0)
	/home/ci/jks/go/src/runtime/netpoll.go:173 +0x59
internal/poll.(*pollDesc).wait(0xc42023c118, 0x72, 0xffffffffffffff00, 0x7f4158888400, 0x7f4158884178)
	/home/ci/jks/go/src/internal/poll/fd_poll_runtime.go:85 +0xb0
internal/poll.(*pollDesc).waitRead(0xc42023c118, 0xc42024a000, 0x8000, 0x8000)
	/home/ci/jks/go/src/internal/poll/fd_poll_runtime.go:90 +0x3f
internal/poll.(*FD).Read(0xc42023c100, 0xc42024a000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
	/home/ci/jks/go/src/internal/poll/fd_unix.go:126 +0x18c
net.(*netFD).Read(0xc42023c100, 0xc42024a000, 0x8000, 0x8000, 0x9, 0x0, 0x0)
	/home/ci/jks/go/src/net/fd_unix.go:202 +0x54
net.(*conn).Read(0xc420244000, 0xc42024a000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
	/home/ci/jks/go/src/net/net.go:176 +0x6f
bufio.(*Reader).Read(0xc420228240, 0xc42025a038, 0x9, 0x9, 0x28, 0x7f415863d5e0, 0x0)
	/home/ci/jks/go/src/bufio/bufio.go:213 +0x30d
io.ReadAtLeast(0x7f4158885800, 0xc420228240, 0xc42025a038, 0x9, 0x9, 0x9, 0xc4201bedb0, 0x7f4157e9ec50, 0xc4201bee5f)
	/home/ci/jks/go/src/io/io.go:309 +0x88
io.ReadFull(0x7f4158885800, 0xc420228240, 0xc42025a038, 0x9, 0x9, 0x7f41581bad16, 0xc4200bbbc0, 0xc4200b0004)
	/home/ci/jks/go/src/io/io.go:327 +0x5a
golang.org/x/net/http2.readFrameHeader(0xc42025a038, 0x9, 0x9, 0x7f4158885800, 0xc420228240, 0x0, 0x7f4100000000, 0xc4200bbbc0, 0x0)
	/home/ci/jks/workspace/CDB/RR/src/golang.org/x/net/http2/frame.go:237 +0x7d
golang.org/x/net/http2.(*Framer).ReadFrame(0xc42025a000, 0x7f4158889380, 0xc4200bbbc0, 0x0, 0x0)
	/home/ci/jks/workspace/CDB/RR/src/golang.org/x/net/http2/frame.go:492 +0xa6
google.golang.org/grpc/transport.(*framer).readFrame(0xc42022c120, 0xc4200bbbc0, 0xc4200bbbc0, 0x0, 0x0)
	/home/ci/jks/workspace/CDB/RR/src/google.golang.org/grpc/transport/http_util.go:506 +0x31
google.golang.org/grpc/transport.(*http2Client).reader(0xc4200ac300)
	/home/ci/jks/workspace/CDB/RR/src/google.golang.org/grpc/transport/http2_client.go:1039 +0xc2
created by google.golang.org/grpc/transport.newHTTP2Client
	/home/ci/jks/workspace/CDB/RR/src/google.golang.org/grpc/transport/http2_client.go:247 +0xb8f

goroutine 23 [select]:
github.com/coreos/etcd/clientv3.(*simpleBalancer).updateNotifyLoop(0xc42020a000)
	/home/ci/jks/workspace/CDB/RR/src/github.com/coreos/etcd/clientv3/balancer.go:211 +0x38f
created by github.com/coreos/etcd/clientv3.newSimpleBalancer
	/home/ci/jks/workspace/CDB/RR/src/github.com/coreos/etcd/clientv3/balancer.go:92 +0x363

goroutine 11 [chan receive]:
google.golang.org/grpc.(*ClientConn).lbWatcher(0xc4201da780)
	/home/ci/jks/workspace/CDB/RR/src/google.golang.org/grpc/clientconn.go:481 +0x742
created by google.golang.org/grpc.DialContext
	/home/ci/jks/workspace/CDB/RR/src/google.golang.org/grpc/clientconn.go:424 +0x577

goroutine 10 [select]:
google.golang.org/grpc.(*addrConn).transportMonitor(0xc4200bcea0)
	/home/ci/jks/workspace/CDB/RR/src/google.golang.org/grpc/clientconn.go:869 +0x1e0
google.golang.org/grpc.(*ClientConn).resetAddrConn.func1(0xc4200bcea0)
	/home/ci/jks/workspace/CDB/RR/src/google.golang.org/grpc/clientconn.go:614 +0x1ca
created by google.golang.org/grpc.(*ClientConn).resetAddrConn
	/home/ci/jks/workspace/CDB/RR/src/google.golang.org/grpc/clientconn.go:605 +0x372

goroutine 37 [select]:
google.golang.org/grpc/transport.(*http2Client).controller(0xc4200ac300)
	/home/ci/jks/workspace/CDB/RR/src/google.golang.org/grpc/transport/http2_client.go:1112 +0x144
created by google.golang.org/grpc/transport.newHTTP2Client
	/home/ci/jks/workspace/CDB/RR/src/google.golang.org/grpc/transport/http2_client.go:277 +0xf4e

goroutine 26 [IO wait]:
internal/poll.runtime_pollWait(0x7f4158c5edf0, 0x72, 0x0)
	/home/ci/jks/go/src/runtime/netpoll.go:173 +0x59
internal/poll.(*pollDesc).wait(0xc4201d2618, 0x72, 0xffffffffffffff00, 0x7f4158888400, 0x7f4158884178)
	/home/ci/jks/go/src/internal/poll/fd_poll_runtime.go:85 +0xb0
internal/poll.(*pollDesc).waitRead(0xc4201d2618, 0xc42028a000, 0x8000, 0x8000)
	/home/ci/jks/go/src/internal/poll/fd_poll_runtime.go:90 +0x3f
internal/poll.(*FD).Read(0xc4201d2600, 0xc42028a000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
	/home/ci/jks/go/src/internal/poll/fd_unix.go:126 +0x18c
net.(*netFD).Read(0xc4201d2600, 0xc42028a000, 0x8000, 0x8000, 0x7f4157ef1740, 0xc420210000, 0x0)
	/home/ci/jks/go/src/net/fd_unix.go:202 +0x54
net.(*conn).Read(0xc4201dc050, 0xc42028a000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
	/home/ci/jks/go/src/net/net.go:176 +0x6f
bufio.(*Reader).Read(0xc4201c6720, 0xc4201f21f8, 0x9, 0x9, 0xc4204de000, 0xc420268d80, 0xc4202d7148)
	/home/ci/jks/go/src/bufio/bufio.go:213 +0x30d
io.ReadAtLeast(0x7f4158885800, 0xc4201c6720, 0xc4201f21f8, 0x9, 0x9, 0x9, 0xc4204cf600, 0x7f4157eaffba, 0xc420268da0)
	/home/ci/jks/go/src/io/io.go:309 +0x88
io.ReadFull(0x7f4158885800, 0xc4201c6720, 0xc4201f21f8, 0x9, 0x9, 0xc4201f21e8, 0x0, 0x0)
	/home/ci/jks/go/src/io/io.go:327 +0x5a
golang.org/x/net/http2.readFrameHeader(0xc4201f21f8, 0x9, 0x9, 0x7f4158885800, 0xc4201c6720, 0x0, 0x2b800000000, 0x100000019, 0x19000002b8)
	/home/ci/jks/workspace/CDB/RR/src/golang.org/x/net/http2/frame.go:237 +0x7d
golang.org/x/net/http2.(*Framer).ReadFrame(0xc4201f21c0, 0x7f4158889140, 0xc420479fb0, 0x0, 0x0)
	/home/ci/jks/workspace/CDB/RR/src/golang.org/x/net/http2/frame.go:492 +0xa6
google.golang.org/grpc/transport.(*framer).readFrame(0xc4201ca5d0, 0xc420479fb0, 0xc420479fb0, 0x0, 0x0)
	/home/ci/jks/workspace/CDB/RR/src/google.golang.org/grpc/transport/http_util.go:506 +0x31
google.golang.org/grpc/transport.(*http2Client).reader(0xc4201dad80)
	/home/ci/jks/workspace/CDB/RR/src/google.golang.org/grpc/transport/http2_client.go:1039 +0xc2
created by google.golang.org/grpc/transport.newHTTP2Client
	/home/ci/jks/workspace/CDB/RR/src/google.golang.org/grpc/transport/http2_client.go:247 +0xb8f

goroutine 39 [select]:
github.com/coreos/etcd/clientv3.(*simpleBalancer).updateNotifyLoop(0xc420272000)
	/home/ci/jks/workspace/CDB/RR/src/github.com/coreos/etcd/clientv3/balancer.go:211 +0x38f
created by github.com/coreos/etcd/clientv3.newSimpleBalancer
	/home/ci/jks/workspace/CDB/RR/src/github.com/coreos/etcd/clientv3/balancer.go:92 +0x363

goroutine 42 [chan receive]:
google.golang.org/grpc.(*ClientConn).lbWatcher(0xc4200ac900)
	/home/ci/jks/workspace/CDB/RR/src/google.golang.org/grpc/clientconn.go:481 +0x742
created by google.golang.org/grpc.DialContext
	/home/ci/jks/workspace/CDB/RR/src/google.golang.org/grpc/clientconn.go:424 +0x577

goroutine 41 [select]:
google.golang.org/grpc.(*addrConn).transportMonitor(0xc42027a000)
	/home/ci/jks/workspace/CDB/RR/src/google.golang.org/grpc/clientconn.go:869 +0x1e0
google.golang.org/grpc.(*ClientConn).resetAddrConn.func1(0xc42027a000)
	/home/ci/jks/workspace/CDB/RR/src/google.golang.org/grpc/clientconn.go:614 +0x1ca
created by google.golang.org/grpc.(*ClientConn).resetAddrConn
	/home/ci/jks/workspace/CDB/RR/src/google.golang.org/grpc/clientconn.go:605 +0x372

goroutine 27 [select]:
google.golang.org/grpc/transport.(*http2Client).controller(0xc4201dad80)
	/home/ci/jks/workspace/CDB/RR/src/google.golang.org/grpc/transport/http2_client.go:1112 +0x144
created by google.golang.org/grpc/transport.newHTTP2Client
	/home/ci/jks/workspace/CDB/RR/src/google.golang.org/grpc/transport/http2_client.go:277 +0xf4e

goroutine 102 [select]:
context.propagateCancel.func1(0x7f412e411158, 0xc420085380, 0x7f4158888b80, 0xc42009ec40)
	/home/ci/jks/go/src/context/context.go:260 +0x115
created by context.propagateCancel
	/home/ci/jks/go/src/context/context.go:259 +0x1dc

goroutine 93 [select]:
github.com/coreos/etcd/clientv3.(*watchGrpcStream).run(0xc420280160)
	/home/ci/jks/workspace/CDB/RR/src/github.com/coreos/etcd/clientv3/watch.go:430 +0x3ef
created by github.com/coreos/etcd/clientv3.(*watcher).newWatcherGrpcStream
	/home/ci/jks/workspace/CDB/RR/src/github.com/coreos/etcd/clientv3/watch.go:228 +0x393

goroutine 74 [select]:
context.propagateCancel.func1(0x7f412e411158, 0xc4202de0f0, 0x7f4158888b80, 0xc4201ad380)
	/home/ci/jks/go/src/context/context.go:260 +0x115
created by context.propagateCancel
	/home/ci/jks/go/src/context/context.go:259 +0x1dc

goroutine 75 [select]:
github.com/coreos/etcd/clientv3.(*watchGrpcStream).run(0xc420376000)
	/home/ci/jks/workspace/CDB/RR/src/github.com/coreos/etcd/clientv3/watch.go:430 +0x3ef
created by github.com/coreos/etcd/clientv3.(*watcher).newWatcherGrpcStream
	/home/ci/jks/workspace/CDB/RR/src/github.com/coreos/etcd/clientv3/watch.go:228 +0x393

goroutine 92 [select]:
context.propagateCancel.func1(0x7f412e411158, 0xc4202331f0, 0x7f4158888b80, 0xc420468980)
	/home/ci/jks/go/src/context/context.go:260 +0x115
created by context.propagateCancel
	/home/ci/jks/go/src/context/context.go:259 +0x1dc

goroutine 77 [select]:
google.golang.org/grpc.newClientStream.func3(0x7f4158890680, 0xc4200ac300, 0xc420310120, 0xc4203082c0)
	/home/ci/jks/workspace/CDB/RR/src/google.golang.org/grpc/stream.go:236 +0x1a3
created by google.golang.org/grpc.newClientStream
	/home/ci/jks/workspace/CDB/RR/src/google.golang.org/grpc/stream.go:235 +0xbc9

goroutine 78 [select]:
google.golang.org/grpc/transport.(*recvBufferReader).Read(0xc4201ad400, 0xc42021a590, 0x5, 0x5, 0x0, 0x0, 0x0)
	/home/ci/jks/workspace/CDB/RR/src/google.golang.org/grpc/transport/transport.go:143 +0x2ee
google.golang.org/grpc/transport.(*Stream).Read(0xc420310120, 0xc42021a590, 0x5, 0x5, 0x0, 0x0, 0x0)
	/home/ci/jks/workspace/CDB/RR/src/google.golang.org/grpc/transport/transport.go:333 +0x5e
io.ReadAtLeast(0x7f4158885e80, 0xc420310120, 0xc42021a590, 0x5, 0x5, 0x5, 0x0, 0x0, 0x0)
	/home/ci/jks/go/src/io/io.go:309 +0x88
io.ReadFull(0x7f4158885e80, 0xc420310120, 0xc42021a590, 0x5, 0x5, 0x0, 0x0, 0x0)
	/home/ci/jks/go/src/io/io.go:327 +0x5a
google.golang.org/grpc.(*parser).recvMsg(0xc42021a580, 0x7fffffff, 0x0, 0x0, 0x0, 0x0, 0x3, 0xc4201f6900)
	/home/ci/jks/workspace/CDB/RR/src/google.golang.org/grpc/rpc_util.go:243 +0x6b
google.golang.org/grpc.recv(0xc42021a580, 0x7f415888d700, 0x7f41588e3c08, 0xc420310120, 0x0, 0x0, 0x7f415864f800, 0xc420361130, 0x7fffffff, 0x0, ...)
	/home/ci/jks/workspace/CDB/RR/src/google.golang.org/grpc/rpc_util.go:339 +0x4f
google.golang.org/grpc.(*clientStream).RecvMsg(0xc4203082c0, 0x7f415864f800, 0xc420361130, 0x0, 0x0)
	/home/ci/jks/workspace/CDB/RR/src/google.golang.org/grpc/stream.go:386 +0x116
github.com/coreos/etcd/etcdserver/etcdserverpb.(*watchWatchClient).Recv(0xc4202de150, 0x0, 0x0, 0x0)
	/home/ci/jks/workspace/CDB/RR/src/github.com/coreos/etcd/etcdserver/etcdserverpb/rpc.pb.go:2473 +0x64
github.com/coreos/etcd/clientv3.(*watchGrpcStream).serveWatchClient(0xc420376000, 0x7f4158890920, 0xc4202de150)
	/home/ci/jks/workspace/CDB/RR/src/github.com/coreos/etcd/clientv3/watch.go:557 +0x50
created by github.com/coreos/etcd/clientv3.(*watchGrpcStream).newWatchClient
	/home/ci/jks/workspace/CDB/RR/src/github.com/coreos/etcd/clientv3/watch.go:712 +0x656

goroutine 65 [select]:
github.com/coreos/etcd/clientv3.(*watchGrpcStream).serveSubstream(0xc4200d9290, 0xc42029a2c0, 0xc4200b9260)
	/home/ci/jks/workspace/CDB/RR/src/github.com/coreos/etcd/clientv3/watch.go:603 +0x2ed
created by github.com/coreos/etcd/clientv3.(*watchGrpcStream).run
	/home/ci/jks/workspace/CDB/RR/src/github.com/coreos/etcd/clientv3/watch.go:444 +0x1121

goroutine 103 [select]:
github.com/coreos/etcd/clientv3.(*watchGrpcStream).run(0xc4200d9290)
	/home/ci/jks/workspace/CDB/RR/src/github.com/coreos/etcd/clientv3/watch.go:430 +0x3ef
created by github.com/coreos/etcd/clientv3.(*watcher).newWatcherGrpcStream
	/home/ci/jks/workspace/CDB/RR/src/github.com/coreos/etcd/clientv3/watch.go:228 +0x393

goroutine 105 [select]:
google.golang.org/grpc.newClientStream.func3(0x7f4158890680, 0xc4201dad80, 0xc4201a85a0, 0xc4200df4a0)
	/home/ci/jks/workspace/CDB/RR/src/google.golang.org/grpc/stream.go:236 +0x1a3
created by google.golang.org/grpc.newClientStream
	/home/ci/jks/workspace/CDB/RR/src/google.golang.org/grpc/stream.go:235 +0xbc9

goroutine 106 [runnable]:
google.golang.org/grpc/transport.(*recvBufferReader).Read(0xc42009ec80, 0xc420193750, 0x5, 0x5, 0x0, 0x0, 0x0)
	/home/ci/jks/workspace/CDB/RR/src/google.golang.org/grpc/transport/transport.go:143 +0x2ee
google.golang.org/grpc/transport.(*Stream).Read(0xc4201a85a0, 0xc420193750, 0x5, 0x5, 0x17, 0xc4201bfb60, 0x7f4157f21466)
	/home/ci/jks/workspace/CDB/RR/src/google.golang.org/grpc/transport/transport.go:333 +0x5e
io.ReadAtLeast(0x7f4158885e80, 0xc4201a85a0, 0xc420193750, 0x5, 0x5, 0x5, 0xc4201bfed0, 0x7f4157ecaead, 0xc4201bfb90)
	/home/ci/jks/go/src/io/io.go:309 +0x88
io.ReadFull(0x7f4158885e80, 0xc4201a85a0, 0xc420193750, 0x5, 0x5, 0xc4201bfea0, 0xc4201aef00, 0xc4200a1160)
	/home/ci/jks/go/src/io/io.go:327 +0x5a
google.golang.org/grpc.(*parser).recvMsg(0xc420193740, 0x7fffffff, 0x0, 0x0, 0x0, 0x2, 0x3, 0xc4201af380)
	/home/ci/jks/workspace/CDB/RR/src/google.golang.org/grpc/rpc_util.go:243 +0x6b
google.golang.org/grpc.recv(0xc420193740, 0x7f415888d700, 0x7f41588e3c08, 0xc4201a85a0, 0x0, 0x0, 0x7f415864f800, 0xc4204b2c80, 0x7fffffff, 0x0, ...)
	/home/ci/jks/workspace/CDB/RR/src/google.golang.org/grpc/rpc_util.go:339 +0x4f
google.golang.org/grpc.(*clientStream).RecvMsg(0xc4200df4a0, 0x7f415864f800, 0xc4204b2c80, 0x0, 0x0)
	/home/ci/jks/workspace/CDB/RR/src/google.golang.org/grpc/stream.go:386 +0x116
github.com/coreos/etcd/etcdserver/etcdserverpb.(*watchWatchClient).Recv(0xc4200853e0, 0x0, 0x0, 0x0)
	/home/ci/jks/workspace/CDB/RR/src/github.com/coreos/etcd/etcdserver/etcdserverpb/rpc.pb.go:2473 +0x64
github.com/coreos/etcd/clientv3.(*watchGrpcStream).serveWatchClient(0xc4200d9290, 0x7f4158890920, 0xc4200853e0)
	/home/ci/jks/workspace/CDB/RR/src/github.com/coreos/etcd/clientv3/watch.go:557 +0x50
created by github.com/coreos/etcd/clientv3.(*watchGrpcStream).newWatchClient
	/home/ci/jks/workspace/CDB/RR/src/github.com/coreos/etcd/clientv3/watch.go:712 +0x656

goroutine 96 [select]:
google.golang.org/grpc/transport.(*recvBufferReader).Read(0xc4204689c0, 0xc42046a6b0, 0x5, 0x5, 0x0, 0x0, 0x0)
	/home/ci/jks/workspace/CDB/RR/src/google.golang.org/grpc/transport/transport.go:143 +0x2ee
google.golang.org/grpc/transport.(*Stream).Read(0xc420271560, 0xc42046a6b0, 0x5, 0x5, 0x0, 0x90, 0x3)
	/home/ci/jks/workspace/CDB/RR/src/google.golang.org/grpc/transport/transport.go:333 +0x5e
io.ReadAtLeast(0x7f4158885e80, 0xc420271560, 0xc42046a6b0, 0x5, 0x5, 0x5, 0xc42006abe0, 0x7f4157ff9d14, 0xc42006abd0)
	/home/ci/jks/go/src/io/io.go:309 +0x88
io.ReadFull(0x7f4158885e80, 0xc420271560, 0xc42046a6b0, 0x5, 0x5, 0x7f41586716e0, 0x7f41582935dc, 0x7f4158290c98)
	/home/ci/jks/go/src/io/io.go:327 +0x5a
google.golang.org/grpc.(*parser).recvMsg(0xc42046a6a0, 0x7fffffff, 0xc420244000, 0xc42024a000, 0x6, 0xc4201db080, 0x3, 0xc4200acf00)
	/home/ci/jks/workspace/CDB/RR/src/google.golang.org/grpc/rpc_util.go:243 +0x6b
google.golang.org/grpc.recv(0xc42046a6a0, 0x7f415888d700, 0x7f41588e3c08, 0xc420271560, 0x0, 0x0, 0x7f415864f800, 0xc4203f1a90, 0x7fffffff, 0x0, ...)
	/home/ci/jks/workspace/CDB/RR/src/google.golang.org/grpc/rpc_util.go:339 +0x4f
google.golang.org/grpc.(*clientStream).RecvMsg(0xc42025d080, 0x7f415864f800, 0xc4203f1a90, 0x0, 0x0)
	/home/ci/jks/workspace/CDB/RR/src/google.golang.org/grpc/stream.go:386 +0x116
github.com/coreos/etcd/etcdserver/etcdserverpb.(*watchWatchClient).Recv(0xc420233250, 0x0, 0x0, 0x0)
	/home/ci/jks/workspace/CDB/RR/src/github.com/coreos/etcd/etcdserver/etcdserverpb/rpc.pb.go:2473 +0x64
github.com/coreos/etcd/clientv3.(*watchGrpcStream).serveWatchClient(0xc420280160, 0x7f4158890920, 0xc420233250)
	/home/ci/jks/workspace/CDB/RR/src/github.com/coreos/etcd/clientv3/watch.go:557 +0x50
created by github.com/coreos/etcd/clientv3.(*watchGrpcStream).newWatchClient
	/home/ci/jks/workspace/CDB/RR/src/github.com/coreos/etcd/clientv3/watch.go:712 +0x656

goroutine 95 [select]:
google.golang.org/grpc.newClientStream.func3(0x7f4158890680, 0xc4201da180, 0xc420271560, 0xc42025d080)
	/home/ci/jks/workspace/CDB/RR/src/google.golang.org/grpc/stream.go:236 +0x1a3
created by google.golang.org/grpc.newClientStream
	/home/ci/jks/workspace/CDB/RR/src/google.golang.org/grpc/stream.go:235 +0xbc9

goroutine 98 [chan receive]:
main.wrapperWatch.func1(0xc4201d2780, 0x7f415888dd40, 0xc420468a00, 0xc4201ce440, 0xc4202332a0, 0x2, 0x2, 0xc420466ba0)
	/home/ci/jks/workspace/CDB/RR/src/github.com/sdb/etcd-client-v3/src/wrapper.go:584 +0x1b5
created by main.wrapperWatch
	/home/ci/jks/workspace/CDB/RR/src/github.com/sdb/etcd-client-v3/src/wrapper.go:580 +0x19b

goroutine 179 [chan receive]:
main.wrapperWatch.func1(0xc4201d2780, 0x7f415888dd40, 0xc420468a80, 0xc4202303c0, 0xc42046a7e0, 0x3, 0x4, 0xc420466c00)
	/home/ci/jks/workspace/CDB/RR/src/github.com/sdb/etcd-client-v3/src/wrapper.go:584 +0x1b5
created by main.wrapperWatch
	/home/ci/jks/workspace/CDB/RR/src/github.com/sdb/etcd-client-v3/src/wrapper.go:580 +0x19b

goroutine 180 [select]:
github.com/coreos/etcd/clientv3.(*watchGrpcStream).serveSubstream(0xc4200d9290, 0xc4202802c0, 0xc4200b9260)
	/home/ci/jks/workspace/CDB/RR/src/github.com/coreos/etcd/clientv3/watch.go:603 +0x2ed
created by github.com/coreos/etcd/clientv3.(*watchGrpcStream).run
	/home/ci/jks/workspace/CDB/RR/src/github.com/coreos/etcd/clientv3/watch.go:444 +0x1121

@ianlancetaylor ianlancetaylor changed the title from get "runtime: unexpected return pc for runtime.gopark called from 0x0" to runtime: "runtime: unexpected return pc for runtime.gopark called from 0x0" Jan 26, 2018

@ianlancetaylor

This comment has been minimized.

Contributor

ianlancetaylor commented Jan 26, 2018

Is it possible for you to try this with today's Go1.10rc1 release candidate? I think that will print more information.

CC @aclements

@xiaoyulei

This comment has been minimized.

xiaoyulei commented Jan 26, 2018

@ianlancetaylor @aclements I can try, but I am not sure if it can happen again.

@aclements

This comment has been minimized.

Member

aclements commented Jan 26, 2018

It's interesting that after the panic, gopark doesn't appear in any of the printed stacks. That suggests that either the original traceback was completely confused, or the stack it was tracebacking wasn't actually stopped.

@xiaoyulei

This comment has been minimized.

xiaoyulei commented Feb 1, 2018

@ianlancetaylor @aclements go1.10rc not support monotonic time?
I get this panic: Go version must be 1.9.* to support monotonic time

@ianlancetaylor

This comment has been minimized.

Contributor

ianlancetaylor commented Feb 1, 2018

@YuleiXiao I don't know where that error message is coming from. It's not coming from anything in the Go distribution. I would guess that some code that you are using is not using build tags in the expected way. Go 1.10 does support monotonic time just as 1.9 does.

@xiaoyulei

This comment has been minimized.

xiaoyulei commented Feb 2, 2018

@ianlancetaylor I know the panic where from. I have remove it and test it for a night, but gc crash not happen, I will test it for more time.

@maxxant

This comment has been minimized.

maxxant commented Feb 8, 2018

i have similar issue while app running with go linux/amd64 1.9.2, 1.9.3 or 1.10rc1 . The go env is the same.
next trace for go 1.9.3

runtime: unexpected return pc for runtime.gopark called from 0x0
fatal error: unknown caller pc

runtime stack:
runtime.throw(0x462a278, 0x11)
        /usr/local/go/src/runtime/panic.go:605 +0x97
runtime.gentraceback(0xffffffffffffffff, 0xffffffffffffffff, 0x0, 0xc420001380, 0x0, 0x0, 0x7fffffff, 0x7f1c2cad4a50, 0x0, 0x0, ...)
        /usr/local/go/src/runtime/traceback.go:285 +0x19b6
runtime.scanstack(0xc420001380, 0xc420027260)
        /usr/local/go/src/runtime/mgcmark.go:779 +0x158
runtime.scang(0xc420001380, 0xc420027260)
        /usr/local/go/src/runtime/proc.go:869 +0x1f8
runtime.markroot.func1()
        /usr/local/go/src/runtime/mgcmark.go:264 +0x71
runtime.systemstack(0x7f1c2cad4b70)
        /usr/local/go/src/runtime/asm_amd64.s:360 +0xa1
runtime.markroot(0xc420027260, 0x9)
        /usr/local/go/src/runtime/mgcmark.go:245 +0x30a
runtime.gcDrainN(0xc420027260, 0x10000, 0x7f1c478127b8)
        /usr/local/go/src/runtime/mgcmark.go:1028 +0xfd
runtime.gcAssistAlloc1(0xc420001500, 0x10000)
        /usr/local/go/src/runtime/mgcmark.go:557 +0x125
runtime.gcAssistAlloc.func1()
        /usr/local/go/src/runtime/mgcmark.go:468 +0x35
runtime.systemstack(0xc420028600)
        /usr/local/go/src/runtime/asm_amd64.s:344 +0x72
runtime.mstart()
        /usr/local/go/src/runtime/proc.go:1135

goroutine 7 [GC assist marking]:
runtime.systemstack_switch()
        /usr/local/go/src/runtime/asm_amd64.s:298 fp=0xc420051690 sp=0xc420051688 pc=0x1b449e0
runtime.gcAssistAlloc(0xc420001500)
        /usr/local/go/src/runtime/mgcmark.go:467 +0x157 fp=0xc4200516e0 sp=0xc420051690 pc=0x1b0a447
runtime.mallocgc(0x48, 0x531b2a0, 0xc420023501, 0x65)
        /usr/local/go/src/runtime/malloc.go:615 +0x817 fp=0xc420051788 sp=0xc4200516e0 pc=0x1aff067
runtime.newobject(0x531b2a0, 0x65)
        /usr/local/go/src/runtime/malloc.go:840 +0x3a fp=0xc4200517b8 sp=0xc420051788 pc=0x1aff29a
libs/ads_gobridge/gosrc/vendor/github.com/maxxant/go-fstab.ParseLine(0xc4200235e0, 0x65, 0x65, 0x64b, 0xc4200235e0)
        /builds/ads/ads-rep/src/libs/ads_gobridge/gosrc/vendor/github.com/maxxant/go-fstab/mount.go:187 +0xd6 fp=0xc420051850 sp=0xc4200517b8 pc=0x1f658a6
libs/ads_gobridge/gosrc/vendor/github.com/maxxant/go-fstab.Parse(0x5655ea0, 0xc42000e710, 0xc42000e710, 0x0, 0x0, 0x0, 0x0)
        /builds/ads/ads-rep/src/libs/ads_gobridge/gosrc/vendor/github.com/maxxant/go-fstab/fstab.go:66 +0x14f fp=0xc420051998 sp=0xc420051850 pc=0x1f6416f
libs/ads_gobridge/gosrc/vendor/github.com/maxxant/go-fstab.ParseFile(0x4627683, 0xc, 0x0, 0x0, 0x0, 0x0, 0x0)
        /builds/ads/ads-rep/src/libs/ads_gobridge/gosrc/vendor/github.com/maxxant/go-fstab/fstab.go:54 +0xd0 fp=0xc4200519e8 sp=0xc420051998 pc=0x1f63f90
libs/ads_gobridge/gosrc/vendor/github.com/maxxant/mxdisk.mapMntFile(0x4627683, 0xc, 0xc4204a4570, 0xc4203b79e0)
        /builds/ads/ads-rep/src/libs/ads_gobridge/gosrc/vendor/github.com/maxxant/mxdisk/mnt.go:38 +0x79 fp=0xc420051a70 sp=0xc4200519e8 pc=0x1f6fc99
libs/ads_gobridge/gosrc/vendor/github.com/maxxant/mxdisk.Watch.func3.1()
        /builds/ads/ads-rep/src/libs/ads_gobridge/gosrc/vendor/github.com/maxxant/mxdisk/mxdisk.go:67 +0x106 fp=0xc420051ad8 sp=0xc420051a70 pc=0x1f724e6
libs/ads_gobridge/gosrc/vendor/github.com/maxxant/mxdisk.Watch.func3(0xc42012d4a0, 0xc42000e520, 0xc42000e4f8, 0xc42000e510, 0xc420168060, 0xc42014b3e0, 0xc42007a180, 0xc4200164c0, 0xc420168000, 0xc42000e588, ...)
        /builds/ads/ads-rep/src/libs/ads_gobridge/gosrc/vendor/github.com/maxxant/mxdisk/mxdisk.go:74 +0x626 fp=0xc420051f78 sp=0xc420051ad8 pc=0x1f72bc6
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:2337 +0x1 fp=0xc420051f80 sp=0xc420051f78 pc=0x1b47621
created by libs/ads_gobridge/gosrc/vendor/github.com/maxxant/mxdisk.Watch
        /builds/ads/ads-rep/src/libs/ads_gobridge/gosrc/vendor/github.com/maxxant/mxdisk/mxdisk.go:47 +0x46d

goroutine 17 [select, locked to thread]:
libs/ads_gobridge/gosrc/hal.GetLastJSONData(0xc420055e80, 0xc420055e88)
        /builds/ads/ads-rep/src/libs/ads_gobridge/gosrc/hal/hal_disk.go:48 +0x109
main.goHalStorageGetJSON(0xc400000008)
        /builds/ads/ads-rep/src/libs/ads_gobridge/gosrc/reg_hal.go:43 +0x24
main._cgoexpwrap_17a8fac53b30_goHalStorageGetJSON(0x0)
        libs/ads_gobridge/gosrc/_obj/_cgo_gotypes.go:184 +0x53

In my case the go programm was compiled as a static library and started from c++

@maxxant

This comment has been minimized.

maxxant commented Feb 8, 2018

one trace for 1.10rc1:

runtime: unexpected return pc for runtime.gopark called from 0x0
stack: frame={sp:0xc4200476f8, fp:0xc420047718} stack=[0xc420047000,0xc420047800)
000000c4200475f8:  0000000000000000  0000000000000000 
000000c420047608:  0000000000000000  0000000000000000 
000000c420047618:  0000000000000000  0000000000000000 
000000c420047628:  00007fca86c4af14  0000000000000000 
000000c420047638:  ffffffff00000000  000000c4200476a0 
000000c420047648:  0100000500000000  0000000000000000 
000000c420047658:  00000000035c14e9  0000000000000000 
000000c420047668:  0000000000000000  000000c420047728 
000000c420047678:  000000c42004769c  0000000000000000 
000000c420047688:  0000000000000000  0000000000000000 
000000c420047698:  0000000000000000  0000000000000000 
000000c4200476a8:  0000000000000000  0000000000000000 
000000c4200476b8:  0000000000000000  0000000000000000 
000000c4200476c8:  0000000000000000  0000000000000000 
000000c4200476d8:  0000000000000000  0000000000000000 
000000c4200476e8:  0000000000000000  0000000000000000 
000000c4200476f8: <0000000000000000  0000000000000000 
000000c420047708:  0000000000000000 !0000000000000000 
000000c420047718: >0000000000000000  0000000000000000 
000000c420047728:  0000000000000000  0000000000000000 
000000c420047738:  0000000000000000  0000000000000000 
000000c420047748:  0000000000000000  0000000000000000 
000000c420047758:  0000000000000000  0000000000000000 
000000c420047768:  0000000000000000  0000000000000000 
000000c420047778:  0000000000000000  0000000000000000 
000000c420047788:  0000000000000000  0000000000000000 
000000c420047798:  0000000000000000  0000000000000000 
000000c4200477a8:  0000000000000000  0000000000000000 
000000c4200477b8:  0000000000000000  0000000000000000 
000000c4200477c8:  0000000003a64639  000000c420047ee8 
000000c4200477d8:  00003c6d9e1d7e74  0000000000000001 
000000c4200477e8:  000000c42014d260  0000000000000001 
000000c4200477f8:  00007fca880165e0 
fatal error: unknown caller pc

runtime stack:
runtime.throw(0x460c20e, 0x11)
        /usr/local/go/src/runtime/panic.go:619 +0x83
runtime.gentraceback(0xffffffffffffffff, 0xffffffffffffffff, 0x0, 0xc420001680, 0x0, 0x0, 0x7fffffff, 0x7fc9f17f9a88, 0x0, 0x0, ...)
        /usr/local/go/src/runtime/traceback.go:292 +0x1bdd
runtime.scanstack(0xc420001680, 0xc420037770)
        /usr/local/go/src/runtime/mgcmark.go:786 +0x14f
runtime.scang(0xc420001680, 0xc420037770)
        /usr/local/go/src/runtime/proc.go:888 +0x1f8
runtime.markroot.func1()
        /usr/local/go/src/runtime/mgcmark.go:264 +0x71
runtime.markroot(0xc420037770, 0xc400000009)
        /usr/local/go/src/runtime/mgcmark.go:245 +0x2f8
runtime.gcDrain(0xc420037770, 0x5)
        /usr/local/go/src/runtime/mgcmark.go:934 +0x115
runtime.gcBgMarkWorker.func2()
        /usr/local/go/src/runtime/mgc.go:1842 +0x82
runtime.systemstack(0x0)
        /usr/local/go/src/runtime/asm_amd64.s:409 +0x72
runtime.mstart()
        /usr/local/go/src/runtime/proc.go:1170

goroutine 21 [GC worker (idle)]:
runtime.systemstack_switch()
        /usr/local/go/src/runtime/asm_amd64.s:363 fp=0xc4201fff48 sp=0xc4201fff40 pc=0x1b41cb0
runtime.gcBgMarkWorker(0xc420036500)
        /usr/local/go/src/runtime/mgc.go:1829 +0x1f6 fp=0xc4201fffd8 sp=0xc4201fff48 pc=0x1b07236
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:2361 +0x1 fp=0xc4201fffe0 sp=0xc4201fffd8 pc=0x1b44851
created by runtime.gcBgMarkStartWorkers
        /usr/local/go/src/runtime/mgc.go:1723 +0x7b

goroutine 17 [runnable, locked to thread]:
syscall.Syscall(0x3, 0x29, 0x0, 0x0, 0x0, 0x0, 0x0)
        /usr/local/go/src/syscall/asm_linux_amd64.s:18 +0x5
syscall.Close(0x29, 0xc4200678b8, 0x1b80d1d)
        /usr/local/go/src/syscall/zsyscall_linux_amd64.go:280 +0x42
internal/poll.(*FD).destroy(0xc42025be00, 0xc420067801, 0x0)
        /usr/local/go/src/internal/poll/fd_unix.go:71 +0x50
internal/poll.(*FD).decref(0xc42025be00, 0x1, 0x2)
        /usr/local/go/src/internal/poll/fd_mutex.go:211 +0x46
internal/poll.(*FD).Close(0xc42025be00, 0x0, 0xc420020f00)
        /usr/local/go/src/internal/poll/fd_unix.go:93 +0x9b
os.(*file).close(0xc42025be00, 0xc420216000, 0x10)
        /usr/local/go/src/os/file_unix.go:211 +0x3a
os.(*File).Close(0xc42000e688, 0xc420067a18, 0xc4203fac90)
        /usr/local/go/src/os/file_unix.go:203 +0x35
os/exec.(*Cmd).closeDescriptors(0xc4200cec60, 0xc4200779c0, 0x3, 0x4)
        /usr/local/go/src/os/exec/exec.go:284 +0x43
os/exec.(*Cmd).Start(0xc4200cec60, 0xc420216001, 0xc42025bdb0)
        /usr/local/go/src/os/exec/exec.go:386 +0x54a
os/exec.(*Cmd).Run(0xc4200cec60, 0xc42025bdb0, 0x5)
        /usr/local/go/src/os/exec/exec.go:297 +0x2d
os/exec.(*Cmd).Output(0xc4200cec60, 0x10, 0xc420067b60, 0x5, 0x5, 0xc4200cec60)
        /usr/local/go/src/os/exec/exec.go:495 +0xf7
libs/ads_gobridge/gosrc/vendor/github.com/maxxant/mxdisk.NewUdevadmInfo(0xc420243998, 0x8, 0x0)
        /builds/ads/ads-rep/src/libs/ads_gobridge/gosrc/vendor/github.com/maxxant/mxdisk/udev.go:36 +0xd2
libs/ads_gobridge/gosrc/vendor/github.com/maxxant/mxdisk.DiskMap.FillDevIDs(0xc42041f8f0)
        /builds/ads/ads-rep/src/libs/ads_gobridge/gosrc/vendor/github.com/maxxant/mxdisk/diskmap.go:126 +0x11e
libs/ads_gobridge/gosrc/hal.GetLastJSONData(0xc420077540, 0x8)
        /builds/ads/ads-rep/src/libs/ads_gobridge/gosrc/hal/hal_disk.go:55 +0x1bc
main.goHalStorageGetJSON(0xc400000008)
        /builds/ads/ads-rep/src/libs/ads_gobridge/gosrc/reg_hal.go:43 +0x24
main._cgoexpwrap_17a8fac53b30_goHalStorageGetJSON(0x0)
        _cgo_gotypes.go:184 +0x53

goroutine 6 [select]:
libs/ads_gobridge/gosrc/vendor/github.com/maxxant/mxdisk.Watch.func3(0xc4201a6d50, 0xc42000e4c8, 0xc42000e4a0, 0xc42000e4b8, 0xc4201c2000, 0xc4201a5220, 0xc420182000, 0xc420024460, 0xc420083f80, 0xc42000e530, ...)
        /builds/ads/ads-rep/src/libs/ads_gobridge/gosrc/vendor/github.com/maxxant/mxdisk/mxdisk.go:71 +0x31d
created by libs/ads_gobridge/gosrc/vendor/github.com/maxxant/mxdisk.Watch
        /builds/ads/ads-rep/src/libs/ads_gobridge/gosrc/vendor/github.com/maxxant/mxdisk/mxdisk.go:47 +0x440
@davecheney

This comment has been minimized.

Contributor

davecheney commented Feb 8, 2018

This looks like memory corruption. Have you tried running your program under the race detector? See https://blog.golang.org/race-detector .

@maxxant

This comment has been minimized.

maxxant commented Feb 8, 2018

Dave, maybe you're right.
My golang parts tested with -race flag of course (coverage ~80). But after i'm build full application (C++ gcc 5.4.0) with golang lib with -race flag:

go build -race -buildmode=c-shared  -o ...

and run application i have next message:

==23640==ERROR: ThreadSanitizer failed to allocate 0x27a1000 (41553920) bytes at address 21fd99ad40000 (errno: 12)

unexpected fault address 0x0
fatal error: fault

[signal SIGSEGV: segmentation violation code=0x80 addr=0x0 pc=0x7ff66648cd1d]

goroutine 1 [running, locked to thread]:
runtime: unexpected return pc for runtime.throw called from 0x7ff63fb9abb8
stack: frame={sp:0x7ff63fb9ab30, fp:0x7ff63fb9ab50} stack=[0xc42007c000,0xc42007e000)

runtime.throw(0x7ff665e3d74d, 0x7ff665e3d223)
	/usr/local/go/src/runtime/panic.go:619 +0x83 fp=0x7ff63fb9ab50 sp=0x7ff63fb9ab30 pc=0x7ff665e3d223

I will dig deeper.

@aclements

This comment has been minimized.

Member

aclements commented Feb 8, 2018

@maxxant, are those trace backs complete? It's odd that none of the goroutines are marked as being scanned, but maybe it's a system goroutine that's causing the problem.

Could you reproduce with GOTRACEBACK=crash set? That should print the system goroutines along with some additional information.

@xiaoyulei

This comment has been minimized.

xiaoyulei commented Feb 9, 2018

@aclements @maxxant I open race and set GOTRACEBACK=crash, it have same message.

go build -race -buildmode=c-shared src/wrapper.go src/callback.go
gcc -fPIC -shared src/client.cpp  -I./src -I../dep/gtest/include/ -I./include -lstdc++ -L../dep/gtest/lib/ -lgtest -lgtest_main -lpthread  -std=c++11 -o libclient.so
mv  wrapper lib/libwrapper.so
mv libclient.so lib

gcc unit_test/client_test.cpp -I./src -I../dep/gtest/include/ -I./include -lstdc++ -L../dep/gtest/lib/ -lgtest -lgtest_main -lpthread  -std=c++11 -L./lib  -lclient -lwrapper -o unit_test/client_test
./unit_test/client_test
[==========] Running 15 tests from 2 test cases.
[----------] Global test environment set-up.
[----------] 1 test from InitClientTest
[ RUN      ] InitClientTest.newMultiClient
==92339==ERROR: ThreadSanitizer failed to allocate 0x27b9000 (41652224) bytes at address 21e0b28940000 (errno: 12)
unexpected fault address 0x0
fatal error: fault
[signal SIGSEGV: segmentation violation code=0x80 addr=0x0 pc=0x7f82c9bdb04d]

goroutine 1 [running, locked to thread]:
runtime: unexpected return pc for runtime.throw called from 0x7f82c8acee78
stack: frame={sp:0x7f82c8acedf0, fp:0x7f82c8acee10} stack=[0xc4200ae000,0xc4200b0000)

runtime.throw(0x7f82c960eead, 0x7f82c960e983)
	/mnt/hdd1/tools/go/src/runtime/panic.go:619 +0x83 fp=0x7f82c8acee10 sp=0x7f82c8acedf0 pc=0x7f82c960e983

goroutine 2 [force gc (idle)]:
runtime.gopark(0x7f82ca0266b8, 0x7f82ca298840, 0x7f82c9befde2, 0xf, 0x7f82ca026514, 0x1)
	/mnt/hdd1/tools/go/src/runtime/proc.go:291 +0xfb fp=0xc42008c768 sp=0xc42008c738 pc=0x7f82c961069b
runtime.goparkunlock(0x7f82ca298840, 0x7f82c9befde2, 0xf, 0x14, 0x1)
	/mnt/hdd1/tools/go/src/runtime/proc.go:297 +0x60 fp=0xc42008c7a8 sp=0xc42008c768 pc=0x7f82c9610750
runtime.forcegchelper()
	/mnt/hdd1/tools/go/src/runtime/proc.go:248 +0xd0 fp=0xc42008c7e0 sp=0xc42008c7a8 pc=0x7f82c9610500
runtime.goexit()
	/mnt/hdd1/tools/go/src/runtime/asm_amd64.s:2361 +0x1 fp=0xc42008c7e8 sp=0xc42008c7e0 pc=0x7f82c963e551
created by runtime.init.4
	/mnt/hdd1/tools/go/src/runtime/proc.go:237 +0x37

goroutine 3 [GC sweep wait]:
runtime.gopark(0x7f82ca0266b8, 0x7f82ca298980, 0x7f82c9beed3a, 0xd, 0x7f82c9602714, 0x1)
	/mnt/hdd1/tools/go/src/runtime/proc.go:291 +0xfb fp=0xc42008cf58 sp=0xc42008cf28 pc=0x7f82c961069b
runtime.goparkunlock(0x7f82ca298980, 0x7f82c9beed3a, 0xd, 0x14, 0x1)
	/mnt/hdd1/tools/go/src/runtime/proc.go:297 +0x60 fp=0xc42008cf98 sp=0xc42008cf58 pc=0x7f82c9610750
runtime.bgsweep(0xc4200bc000)
	/mnt/hdd1/tools/go/src/runtime/mgcsweep.go:52 +0xae fp=0xc42008cfd8 sp=0xc42008cf98 pc=0x7f82c96027ee
runtime.goexit()
	/mnt/hdd1/tools/go/src/runtime/asm_amd64.s:2361 +0x1 fp=0xc42008cfe0 sp=0xc42008cfd8 pc=0x7f82c963e551
created by runtime.gcenable
	/mnt/hdd1/tools/go/src/runtime/mgc.go:216 +0x5a
make: *** [test] Aborted (core dumped)
@maxxant

This comment has been minimized.

maxxant commented Feb 9, 2018

For testing i wrote an application in pure Go that does roughly the same thing but it still running (no crashes), and the race detector is also silent. But for the test needs more time, of course.

C++ & go lib crashed again.
@aclements, next trace with GOTRACEBACK=crash for go 1.10rc1:

runtime: unexpected return pc for runtime.gopark called from 0x0
stack: frame={sp:0xc420046f40, fp:0xc420046f60} stack=[0xc420046800,0xc420047000)
000000c420046e40:  000000c420046ea0  0100000500000000 
000000c420046e50:  0000000000000000  00000000035c14e9 
000000c420046e60:  0000000000000000  000000c420046f28 
000000c420046e70:  000000c420046f28  000000c420046e9c 
000000c420046e80:  0000000001b40960 <runtime.goready.func1+0>  0000000020000180 
000000c420046e90:  0000000000000004  000000c420046ed0 
000000c420046ea0:  0000000000000000  0000000000000000 
000000c420046eb0:  0000000000000000  0000000000000000 
000000c420046ec0:  0000000000000000  0000000000000000 
000000c420046ed0:  0000000000000000  0000000000000000 
000000c420046ee0:  0000000000000000  0000000000000000 
000000c420046ef0:  0000000000000000  0000000000000000 
000000c420046f00:  0000000000000000  0000000000000000 
000000c420046f10:  0000000000000000  0000000000000000 
000000c420046f20:  0000000000000000  0000000000000000 
000000c420046f30:  0000000000000000  0000000000000000 
000000c420046f40: <0000000000000000  0000000000000000 
000000c420046f50:  0000000000000000 !0000000000000000 
000000c420046f60: >0000000000000000  0000000000000000 
000000c420046f70:  0000000000000000  0000000000000000 
000000c420046f80:  0000000000000000  0000000000000000 
000000c420046f90:  0000000000000000  0000000000000000 
000000c420046fa0:  0000000000000000  0000000000000000 
000000c420046fb0:  0000000000000000  0000000000000014 
000000c420046fc0:  0000000000000001  000000003b7c79b1 
000000c420046fd0:  000000c4200476e8  00007ec035c7fc25 
000000c420046fe0:  0000000000000001  000000c420192000 
000000c420046ff0:  0000000000000001  00007f6a372c05e0 
fatal error: unknown caller pc

runtime stack:
runtime.throw(0x460c20e, 0x11)
	/usr/local/go/src/runtime/panic.go:619 +0x83 fp=0x7f698b7fd600 sp=0x7f698b7fd5e0 pc=0x1b17eb3
runtime.gentraceback(0xffffffffffffffff, 0xffffffffffffffff, 0x0, 0xc420000f00, 0x0, 0x0, 0x7fffffff, 0x7f698b7fda88, 0x0, 0x0, ...)
	/usr/local/go/src/runtime/traceback.go:292 +0x1bdd fp=0x7f698b7fd918 sp=0x7f698b7fd600 pc=0x1b39dad
runtime.scanstack(0xc420000f00, 0xc420035270)
	/usr/local/go/src/runtime/mgcmark.go:786 +0x14f fp=0x7f698b7fdab0 sp=0x7f698b7fd918 pc=0x1b0a00f
runtime.scang(0xc420000f00, 0xc420035270)
	/usr/local/go/src/runtime/proc.go:888 +0x1f8 fp=0x7f698b7fdb00 sp=0x7f698b7fdab0 pc=0x1b1bb38
runtime.markroot.func1()
	/usr/local/go/src/runtime/mgcmark.go:264 +0x71 fp=0x7f698b7fdb40 sp=0x7f698b7fdb00 pc=0x1b3fa01
runtime.markroot(0xc420035270, 0xc400000008)
	/usr/local/go/src/runtime/mgcmark.go:245 +0x2f8 fp=0x7f698b7fdbc0 sp=0x7f698b7fdb40 pc=0x1b09018
runtime.gcDrain(0xc420035270, 0x5)
	/usr/local/go/src/runtime/mgcmark.go:934 +0x115 fp=0x7f698b7fdc18 sp=0x7f698b7fdbc0 pc=0x1b0ab05
runtime.gcBgMarkWorker.func2()
	/usr/local/go/src/runtime/mgc.go:1842 +0x82 fp=0x7f698b7fdc58 sp=0x7f698b7fdc18 pc=0x1b3f842
runtime.systemstack(0x0)
	/usr/local/go/src/runtime/asm_amd64.s:409 +0x72 fp=0x7f698b7fdc60 sp=0x7f698b7fdc58 pc=0x1b41d32
runtime.mstart()
	/usr/local/go/src/runtime/proc.go:1170 fp=0x7f698b7fdc68 sp=0x7f698b7fdc60 pc=0x1b1c560

goroutine 68 [GC worker (idle)]:
runtime.systemstack_switch()
	/usr/local/go/src/runtime/asm_amd64.s:363 fp=0xc42032c748 sp=0xc42032c740 pc=0x1b41cb0
runtime.gcBgMarkWorker(0xc420034000)
	/usr/local/go/src/runtime/mgc.go:1829 +0x1f6 fp=0xc42032c7d8 sp=0xc42032c748 pc=0x1b07236
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:2361 +0x1 fp=0xc42032c7e0 sp=0xc42032c7d8 pc=0x1b44851
created by runtime.gcBgMarkStartWorkers
	/usr/local/go/src/runtime/mgc.go:1723 +0x7b

goroutine 4 [syscall]:
runtime.notetsleepg(0x5f5be60, 0x3bb79ea5, 0x0)
	/usr/local/go/src/runtime/lock_futex.go:227 +0x46 fp=0xc420047760 sp=0xc420047730 pc=0x1afd116
runtime.timerproc(0x5f5be40)
	/usr/local/go/src/runtime/time.go:261 +0x2eb fp=0xc4200477d8 sp=0xc420047760 pc=0x1b3590b
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:2361 +0x1 fp=0xc4200477e0 sp=0xc4200477d8 pc=0x1b44851
created by runtime.(*timersBucket).addtimerLocked
	/usr/local/go/src/runtime/time.go:160 +0x109

goroutine 17 [runnable, locked to thread]:
syscall.Syscall6(0xf7, 0x1, 0x385, 0xc42006d8a8, 0x1000004, 0x0, 0x0, 0x0, 0xc42006d8a8, 0x0)
	/usr/local/go/src/syscall/asm_linux_amd64.s:44 +0x5 fp=0xc42006d850 sp=0xc42006d848 pc=0x1b58405
os.(*Process).blockUntilWaitable(0xc4200a3ad0, 0x0, 0x0, 0x6)
	/usr/local/go/src/os/wait_waitid.go:31 +0x9a fp=0xc42006d950 sp=0xc42006d850 pc=0x1b85eea
os.(*Process).wait(0xc4200a3ad0, 0xc420291740, 0xc420292398, 0xc420292398)
	/usr/local/go/src/os/exec_unix.go:22 +0x3e fp=0xc42006d9c8 sp=0xc42006d950 pc=0x1b8126e
os.(*Process).Wait(0xc4200a3ad0, 0x5361670, 0x5361678, 0x5361668)
	/usr/local/go/src/os/exec.go:118 +0x2d fp=0xc42006d9f8 sp=0xc42006d9c8 pc=0x1b8081d
os/exec.(*Cmd).Wait(0xc4202922c0, 0x0, 0x0)
	/usr/local/go/src/os/exec/exec.go:456 +0x5e fp=0xc42006da70 sp=0xc42006d9f8 pc=0x1f4a64e
os/exec.(*Cmd).Run(0xc4202922c0, 0xc420297720, 0x5)
	/usr/local/go/src/os/exec/exec.go:300 +0x5e fp=0xc42006da98 sp=0xc42006da70 pc=0x1f49b2e
os/exec.(*Cmd).Output(0xc4202922c0, 0x10, 0xc42006db60, 0x5, 0x5, 0xc4202922c0)
	/usr/local/go/src/os/exec/exec.go:495 +0xf7 fp=0xc42006dae8 sp=0xc42006da98 pc=0x1f4aa47
libs/ads_gobridge/gosrc/vendor/github.com/maxxant/mxdisk.NewUdevadmInfo(0xc4202984f8, 0x8, 0x0)
	/builds/ads/ads-rep/src/libs/ads_gobridge/gosrc/vendor/github.com/maxxant/mxdisk/udev.go:36 +0xd2 fp=0xc42006dbc0 sp=0xc42006dae8 pc=0x1f53412
libs/ads_gobridge/gosrc/vendor/github.com/maxxant/mxdisk.DiskMap.FillDevIDs(0xc420464240)
	/builds/ads/ads-rep/src/libs/ads_gobridge/gosrc/vendor/github.com/maxxant/mxdisk/diskmap.go:126 +0x11e fp=0xc42006dd60 sp=0xc42006dbc0 pc=0x1f4f4ce
libs/ads_gobridge/gosrc/hal.GetLastJSONData(0xc4200ae040, 0x10)
	/builds/ads/ads-rep/src/libs/ads_gobridge/gosrc/hal/hal_disk.go:55 +0x1bc fp=0xc42006de58 sp=0xc42006dd60 pc=0x1f564ac
main.goHalStorageGetJSON(0xc400000008)
	/builds/ads/ads-rep/src/libs/ads_gobridge/gosrc/reg_hal.go:43 +0x24 fp=0xc42006de80 sp=0xc42006de58 pc=0x1f59954
main._cgoexpwrap_17a8fac53b30_goHalStorageGetJSON(0x0)
	_cgo_gotypes.go:184 +0x53 fp=0xc42006dea8 sp=0xc42006de80 pc=0x1f59063
runtime.call32(0x0, 0x7ffefdec08a0, 0x7ffefdec0930, 0x8)
	/usr/local/go/src/runtime/asm_amd64.s:573 +0x3d fp=0xc42006ded8 sp=0xc42006dea8 pc=0x1b4206d
runtime.cgocallbackg1(0x0)
	/usr/local/go/src/runtime/cgocall.go:316 +0x1a0 fp=0xc42006df58 sp=0xc42006ded8 pc=0x1aef580
runtime.cgocallbackg(0x0)
	/usr/local/go/src/runtime/cgocall.go:194 +0xe0 fp=0xc42006dfc0 sp=0xc42006df58 pc=0x1aef340
runtime.cgocallback_gofunc(0x0, 0x0, 0x0, 0x0)
	/usr/local/go/src/runtime/asm_amd64.s:826 +0x9a fp=0xc42006dfe0 sp=0xc42006dfc0 pc=0x1b4367a
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:2361 +0x1 fp=0xc42006dfe8 sp=0xc42006dfe0 pc=0x1b44851

goroutine 2 [force gc (idle)]:
runtime.gopark(0x53619f8, 0x5f573f0, 0x460ad89, 0xf, 0x5361814, 0x1)
	/usr/local/go/src/runtime/proc.go:291 +0x120 fp=0xc420046768 sp=0xc420046748 pc=0x1b19ba0
runtime.goparkunlock(0x5f573f0, 0x460ad89, 0xf, 0x14, 0x1)
	/usr/local/go/src/runtime/proc.go:297 +0x60 fp=0xc4200467a8 sp=0xc420046768 pc=0x1b19c60
runtime.forcegchelper()
	/usr/local/go/src/runtime/proc.go:248 +0xd0 fp=0xc4200467e0 sp=0xc4200467a8 pc=0x1b199e0
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:2361 +0x1 fp=0xc4200467e8 sp=0xc4200467e0 pc=0x1b44851
created by runtime.init.4
	/usr/local/go/src/runtime/proc.go:237 +0x37

goroutine 3 [GC sweep wait (scan)]:
runtime: unexpected return pc for runtime.gopark called from 0x0
stack: frame={sp:0xc420046f40, fp:0xc420046f60} stack=[0xc420046800,0xc420047000)
000000c420046e40:  000000c420046ea0  0100000500000000 
000000c420046e50:  0000000000000000  00000000035c14e9 
000000c420046e60:  0000000000000000  000000c420046f28 
000000c420046e70:  000000c420046f28  000000c420046e9c 
000000c420046e80:  0000000001b40960 <runtime.goready.func1+0>  0000000020000180 
000000c420046e90:  0000000000000004  000000c420046ed0 
000000c420046ea0:  0000000000000000  0000000000000000 
000000c420046eb0:  0000000000000000  0000000000000000 
000000c420046ec0:  0000000000000000  0000000000000000 
000000c420046ed0:  0000000000000000  0000000000000000 
000000c420046ee0:  0000000000000000  0000000000000000 
000000c420046ef0:  0000000000000000  0000000000000000 
000000c420046f00:  0000000000000000  0000000000000000 
000000c420046f10:  0000000000000000  0000000000000000 
000000c420046f20:  0000000000000000  0000000000000000 
000000c420046f30:  0000000000000000  0000000000000000 
000000c420046f40: <0000000000000000  0000000000000000 
000000c420046f50:  0000000000000000 !0000000000000000 
000000c420046f60: >0000000000000000  0000000000000000 
000000c420046f70:  0000000000000000  0000000000000000 
000000c420046f80:  0000000000000000  0000000000000000 
000000c420046f90:  0000000000000000  0000000000000000 
000000c420046fa0:  0000000000000000  0000000000000000 
000000c420046fb0:  0000000000000000  0000000000000014 
000000c420046fc0:  0000000000000001  000000003b7c79b1 
000000c420046fd0:  000000c4200476e8  00007ec035c7fc25 
000000c420046fe0:  0000000000000001  000000c420192000 
000000c420046ff0:  0000000000000001  00007f6a372c05e0 
runtime.gopark(0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
	/usr/local/go/src/runtime/proc.go:291 +0x120 fp=0xc420046f60 sp=0xc420046f40 pc=0x1b19ba0
created by runtime.gcenable
	/usr/local/go/src/runtime/mgc.go:216 +0x5a

goroutine 18 [finalizer wait]:
runtime.gopark(0x53619f8, 0x6160e60, 0x460a62f, 0xe, 0x14, 0x1)
	/usr/local/go/src/runtime/proc.go:291 +0x120 fp=0xc420042718 sp=0xc4200426f8 pc=0x1b19ba0
runtime.goparkunlock(0x6160e60, 0x460a62f, 0xe, 0x14, 0x1)
	/usr/local/go/src/runtime/proc.go:297 +0x60 fp=0xc420042758 sp=0xc420042718 pc=0x1b19c60
runtime.runfinq()
	/usr/local/go/src/runtime/mfinal.go:175 +0xb1 fp=0xc4200427e0 sp=0xc420042758 pc=0x1b03551
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:2361 +0x1 fp=0xc4200427e8 sp=0xc4200427e0 pc=0x1b44851
created by runtime.createfing
	/usr/local/go/src/runtime/mfinal.go:156 +0x64

goroutine 22 [syscall]:
runtime.notetsleepg(0x5f5bee0, 0x3b9ab9d8, 0x16)
	/usr/local/go/src/runtime/lock_futex.go:227 +0x46 fp=0xc420043760 sp=0xc420043730 pc=0x1afd116
runtime.timerproc(0x5f5bec0)
	/usr/local/go/src/runtime/time.go:261 +0x2eb fp=0xc4200437d8 sp=0xc420043760 pc=0x1b3590b
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:2361 +0x1 fp=0xc4200437e0 sp=0xc4200437d8 pc=0x1b44851
created by runtime.(*timersBucket).addtimerLocked
	/usr/local/go/src/runtime/time.go:160 +0x109

goroutine 21 [runnable]:
strings.Index(0xc420433ef0, 0x23, 0x4603c0b, 0x1, 0x8)
	/usr/local/go/src/strings/strings_amd64.go:27 +0x3cb fp=0xc42006b508 sp=0xc42006b500 pc=0x1c1924b
strings.genSplit(0xc420433ef0, 0x23, 0x4603c0b, 0x1, 0x0, 0x6, 0x1af4e0f, 0x53227a0, 0xc4204646c0)
	/usr/local/go/src/strings/strings.go:250 +0x114 fp=0xc42006b568 sp=0xc42006b508 pc=0x1c16684
strings.Split(0xc420433ee4, 0x2f, 0x4603c0b, 0x1, 0x4, 0xc42006b628, 0x1c16b8d)
	/usr/local/go/src/strings/strings.go:298 +0x5d fp=0xc42006b5c0 sp=0xc42006b568 pc=0x1c1695d
libs/ads_gobridge/gosrc/vendor/github.com/maxxant/go-fstab.parseOptions(0xc420433ee4, 0x2f, 0xc4204b41e0)
	/builds/ads/ads-rep/src/libs/ads_gobridge/gosrc/vendor/github.com/maxxant/go-fstab/mount.go:44 +0x61 fp=0xc42006b638 sp=0xc42006b5c0 pc=0x1f47081
libs/ads_gobridge/gosrc/vendor/github.com/maxxant/go-fstab.ParseLine(0xc420433ec0, 0x57, 0x57, 0xac0, 0xc420433ec0)
	/builds/ads/ads-rep/src/libs/ads_gobridge/gosrc/vendor/github.com/maxxant/go-fstab/mount.go:191 +0x14e fp=0xc42006b6d0 sp=0xc42006b638 pc=0x1f4834e
libs/ads_gobridge/gosrc/vendor/github.com/maxxant/go-fstab.Parse(0x53641a0, 0xc42029a150, 0xc42029a150, 0x0, 0x0, 0x1af4e0f, 0x53227a0)
	/builds/ads/ads-rep/src/libs/ads_gobridge/gosrc/vendor/github.com/maxxant/go-fstab/fstab.go:66 +0x14c fp=0xc42006b800 sp=0xc42006b6d0 pc=0x1f46d4c
libs/ads_gobridge/gosrc/vendor/github.com/maxxant/go-fstab.ParseFile(0x46094e7, 0xc, 0x0, 0x0, 0x0, 0x0, 0x0)
	/builds/ads/ads-rep/src/libs/ads_gobridge/gosrc/vendor/github.com/maxxant/go-fstab/fstab.go:54 +0xb6 fp=0xc42006b850 sp=0xc42006b800 pc=0x1f46b76
libs/ads_gobridge/gosrc/vendor/github.com/maxxant/mxdisk.mapMntFile(0x46094e7, 0xc, 0xc4203af680, 0x5)
	/builds/ads/ads-rep/src/libs/ads_gobridge/gosrc/vendor/github.com/maxxant/mxdisk/mnt.go:38 +0x63 fp=0xc42006baa0 sp=0xc42006b850 pc=0x1f517a3
libs/ads_gobridge/gosrc/vendor/github.com/maxxant/mxdisk.Watch.func3.1()
	/builds/ads/ads-rep/src/libs/ads_gobridge/gosrc/vendor/github.com/maxxant/mxdisk/mxdisk.go:67 +0x106 fp=0xc42006baf0 sp=0xc42006baa0 pc=0x1f543d6
libs/ads_gobridge/gosrc/vendor/github.com/maxxant/mxdisk.Watch.func3(0xc4201bccc0, 0xc4200aa4b8, 0xc4200aa490, 0xc4200aa4a8, 0xc4201da000, 0xc4201bb210, 0xc420192000, 0xc4200a8200, 0xc420091f80, 0xc4200aa520, ...)
	/builds/ads/ads-rep/src/libs/ads_gobridge/gosrc/vendor/github.com/maxxant/mxdisk/mxdisk.go:74 +0x589 fp=0xc42006bf78 sp=0xc42006baf0 pc=0x1f549e9
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:2361 +0x1 fp=0xc42006bf80 sp=0xc42006bf78 pc=0x1b44851
created by libs/ads_gobridge/gosrc/vendor/github.com/maxxant/mxdisk.Watch
	/builds/ads/ads-rep/src/libs/ads_gobridge/gosrc/vendor/github.com/maxxant/mxdisk/mxdisk.go:47 +0x440

goroutine 35 [timer goroutine (idle)]:
runtime.gopark(0x53619f8, 0x5f5bf40, 0x460eb48, 0x16, 0x20fd37114, 0x1)
	/usr/local/go/src/runtime/proc.go:291 +0x120 fp=0xc420048720 sp=0xc420048700 pc=0x1b19ba0
runtime.goparkunlock(0x5f5bf40, 0x460eb48, 0x16, 0x14, 0x1)
	/usr/local/go/src/runtime/proc.go:297 +0x60 fp=0xc420048760 sp=0xc420048720 pc=0x1b19c60
runtime.timerproc(0x5f5bf40)
	/usr/local/go/src/runtime/time.go:253 +0x271 fp=0xc4200487d8 sp=0xc420048760 pc=0x1b35891
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:2361 +0x1 fp=0xc4200487e0 sp=0xc4200487d8 pc=0x1b44851
created by runtime.(*timersBucket).addtimerLocked
	/usr/local/go/src/runtime/time.go:160 +0x109

goroutine 54 [syscall]:
runtime.notetsleepg(0x5f5bfe0, 0x772ff9b7, 0x16)
	/usr/local/go/src/runtime/lock_futex.go:227 +0x46 fp=0xc42032ef60 sp=0xc42032ef30 pc=0x1afd116
runtime.timerproc(0x5f5bfc0)
	/usr/local/go/src/runtime/time.go:261 +0x2eb fp=0xc42032efd8 sp=0xc42032ef60 pc=0x1b3590b
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:2361 +0x1 fp=0xc42032efe0 sp=0xc42032efd8 pc=0x1b44851
created by runtime.(*timersBucket).addtimerLocked
	/usr/local/go/src/runtime/time.go:160 +0x109

goroutine 58 [GC worker (idle)]:
runtime.gopark(0x5361880, 0xc420299ab0, 0x460b272, 0x10, 0xc42039f114, 0x0)
	/usr/local/go/src/runtime/proc.go:291 +0x120 fp=0xc420049f48 sp=0xc420049f28 pc=0x1b19ba0
runtime.gcBgMarkWorker(0xc420038a00)
	/usr/local/go/src/runtime/mgc.go:1775 +0x136 fp=0xc420049fd8 sp=0xc420049f48 pc=0x1b07176
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:2361 +0x1 fp=0xc420049fe0 sp=0xc420049fd8 pc=0x1b44851
created by runtime.gcBgMarkStartWorkers
	/usr/local/go/src/runtime/mgc.go:1723 +0x7b

goroutine 59 [GC worker (idle)]:
runtime.gopark(0x5361880, 0xc420299ac0, 0x460b272, 0x10, 0xc42039f114, 0x0)
	/usr/local/go/src/runtime/proc.go:291 +0x120 fp=0xc42032a748 sp=0xc42032a728 pc=0x1b19ba0
runtime.gcBgMarkWorker(0xc42003af00)
	/usr/local/go/src/runtime/mgc.go:1775 +0x136 fp=0xc42032a7d8 sp=0xc42032a748 pc=0x1b07176
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:2361 +0x1 fp=0xc42032a7e0 sp=0xc42032a7d8 pc=0x1b44851
created by runtime.gcBgMarkStartWorkers
	/usr/local/go/src/runtime/mgc.go:1723 +0x7b

goroutine 69 [GC worker (idle)]:
runtime.gopark(0x5361880, 0xc420299aa0, 0x460b272, 0x10, 0xc42032af14, 0x0)
	/usr/local/go/src/runtime/proc.go:291 +0x120 fp=0xc42032af48 sp=0xc42032af28 pc=0x1b19ba0
runtime.gcBgMarkWorker(0xc420036500)
	/usr/local/go/src/runtime/mgc.go:1775 +0x136 fp=0xc42032afd8 sp=0xc42032af48 pc=0x1b07176
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:2361 +0x1 fp=0xc42032afe0 sp=0xc42032afd8 pc=0x1b44851
created by runtime.gcBgMarkStartWorkers
	/usr/local/go/src/runtime/mgc.go:1723 +0x7b

goroutine 23 [runnable]:
syscall.Syscall(0x0, 0x3b, 0xc420352200, 0x200, 0x200, 0x200, 0x0)
	/usr/local/go/src/syscall/asm_linux_amd64.s:18 +0x5 fp=0xc42032b450 sp=0xc42032b448 pc=0x1b58385
syscall.read(0x3b, 0xc420352200, 0x200, 0x200, 0xffffffffffffff01, 0x0, 0x0)
	/usr/local/go/src/syscall/zsyscall_linux_amd64.go:749 +0x61 fp=0xc42032b4b0 sp=0xc42032b450 pc=0x1b55fd1
syscall.Read(0x3b, 0xc420352200, 0x200, 0x200, 0xffffffffffffffff, 0x5364dc0, 0x56439d8)
	/usr/local/go/src/syscall/syscall_unix.go:162 +0x4b fp=0xc42032b4f8 sp=0xc42032b4b0 pc=0x1b546ab
internal/poll.(*FD).Read(0xc4202977c0, 0xc420352200, 0x200, 0x200, 0x0, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_unix.go:153 +0x11a fp=0xc42032b548 sp=0xc42032b4f8 pc=0x1b7befa
os.(*File).read(0xc42029a120, 0xc420352200, 0x200, 0x200, 0xc420352200, 0x0, 0x0)
	/usr/local/go/src/os/file_unix.go:226 +0x50 fp=0xc42032b590 sp=0xc42032b548 pc=0x1b84510
os.(*File).Read(0xc42029a120, 0xc420352200, 0x200, 0x200, 0xc42032b658, 0x1b7bace, 0xc42032b660)
	/usr/local/go/src/os/file.go:107 +0x6c fp=0xc42032b600 sp=0xc42032b590 pc=0x1b8199c
bytes.(*Buffer).ReadFrom(0xc420295110, 0x53641a0, 0xc42029a120, 0x7f6a3e4590f8, 0xc420295110, 0xc42032b601)
	/usr/local/go/src/bytes/buffer.go:205 +0xa2 fp=0xc42032b670 sp=0xc42032b600 pc=0x1c08bf2
io.copyBuffer(0x5363940, 0xc420295110, 0x53641a0, 0xc42029a120, 0x0, 0x0, 0x0, 0xc420391360, 0x0, 0x0)
	/usr/local/go/src/io/io.go:386 +0x31c fp=0xc42032b6e0 sp=0xc42032b670 pc=0x1b66adc
io.Copy(0x5363940, 0xc420295110, 0x53641a0, 0xc42029a120, 0x1af1325, 0xc42039f440, 0xc42032b7b0)
	/usr/local/go/src/io/io.go:362 +0x5c fp=0xc42032b740 sp=0xc42032b6e0 pc=0x1b6668c
os/exec.(*Cmd).writerDescriptor.func1(0xc42039f440, 0xc42032b7b0)
	/usr/local/go/src/os/exec/exec.go:275 +0x4f fp=0xc42032b7a0 sp=0xc42032b740 pc=0x1f4cd1f
os/exec.(*Cmd).Start.func1(0xc4202922c0, 0xc42029ca20)
	/usr/local/go/src/os/exec/exec.go:391 +0x29 fp=0xc42032b7d0 sp=0xc42032b7a0 pc=0x1f4cd99
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:2361 +0x1 fp=0xc42032b7d8 sp=0xc42032b7d0 pc=0x1b44851
created by os/exec.(*Cmd).Start
	/usr/local/go/src/os/exec/exec.go:390 +0x5e1

goroutine 24 [runnable]:
runtime.gopark(0x53619f8, 0x5f65940, 0x4608c0a, 0xa, 0x1b55f19, 0x4)
	/usr/local/go/src/runtime/proc.go:291 +0x120 fp=0xc42032bc98 sp=0xc42032bc78 pc=0x1b19ba0
runtime.goparkunlock(0x5f65940, 0x4608c0a, 0xa, 0x19, 0x4)
	/usr/local/go/src/runtime/proc.go:297 +0x60 fp=0xc42032bcd8 sp=0xc42032bc98 pc=0x1b19c60
runtime.semacquire1(0x5f587c4, 0x1b3fb00, 0x0)
	/usr/local/go/src/runtime/sema.go:144 +0x1d8 fp=0xc42032bd48 sp=0xc42032bcd8 pc=0x1b29d58
runtime.semacquire(0x5f587c4)
	/usr/local/go/src/runtime/sema.go:95 +0x3b fp=0xc42032bd70 sp=0xc42032bd48 pc=0x1b29b6b
runtime.gcStart(0x0, 0x1, 0x0, 0x7f6a00000000)
	/usr/local/go/src/runtime/mgc.go:1268 +0xdb fp=0xc42032bda0 sp=0xc42032bd70 pc=0x1b05ccb
runtime.mallocgc(0x8000, 0x5278e00, 0x1, 0x0)
	/usr/local/go/src/runtime/malloc.go:803 +0x425 fp=0xc42032be40 sp=0xc42032bda0 pc=0x1afdf85
runtime.makeslice(0x5278e00, 0x8000, 0x8000, 0x0, 0x0, 0xc42032be00)
	/usr/local/go/src/runtime/slice.go:61 +0x79 fp=0xc42032be70 sp=0xc42032be40 pc=0x1b2da99
io.copyBuffer(0x53642a0, 0xc420297720, 0x53641a0, 0xc42029a138, 0x0, 0x0, 0x0, 0xc420391400, 0x0, 0x0)
	/usr/local/go/src/io/io.go:397 +0x2be fp=0xc42032bee0 sp=0xc42032be70 pc=0x1b66a7e
io.Copy(0x53642a0, 0xc420297720, 0x53641a0, 0xc42029a138, 0x1af1325, 0xc42039f440, 0xc42032bfb0)
	/usr/local/go/src/io/io.go:362 +0x5c fp=0xc42032bf40 sp=0xc42032bee0 pc=0x1b6668c
os/exec.(*Cmd).writerDescriptor.func1(0xc42039f440, 0xc42032bfb0)
	/usr/local/go/src/os/exec/exec.go:275 +0x4f fp=0xc42032bfa0 sp=0xc42032bf40 pc=0x1f4cd1f
os/exec.(*Cmd).Start.func1(0xc4202922c0, 0xc42029ca60)
	/usr/local/go/src/os/exec/exec.go:391 +0x29 fp=0xc42032bfd0 sp=0xc42032bfa0 pc=0x1f4cd99
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:2361 +0x1 fp=0xc42032bfd8 sp=0xc42032bfd0 pc=0x1b44851
created by os/exec.(*Cmd).Start
	/usr/local/go/src/os/exec/exec.go:390 +0x5e1
@maxxant

This comment has been minimized.

maxxant commented Feb 17, 2018

In my case everything C/C++ ::system() calls was removed from the code which used for execute miscellaneous tools and replaced with https://github.com/qt/qtbase/tree/5.11/src/3rdparty/forkfd (which internally used in Qt::QProcess). These ::system() calls are not related in the code with golang runtime and worked independently from C/C++ directly.

Now the crashes are gone

@YuleiXiao FYI

@aclements

This comment has been minimized.

Member

aclements commented Jun 21, 2018

TL;DR: @maxxant, do you have (or can you create) more crash tracebacks so I can look for patterns?

The two tracebacks with "ThreadSanitizer failed to allocate" errors are something very different. I'm not sure if it's supported to build in both race mode and c-shared mode.

The stack in #23561 (comment) is starting to reveal something. Here we can see goroutine 68 in a GC background worker scanning goroutine 3's stack. Goroutine 3 is the bgsweep system goroutine. It's parked in state "GC sweep wait", which is completely reasonable. But the arguments to gopark are not: runtime.gopark(0x0, 0x0, 0x0, 0x0, 0x0, 0x0).

The SP (0xc420046f40) is pretty close to the top of the stack (0xc420047000), which is reasonable for bgsweep. However, the whole stack is weird:

000000c420046f40: <0000000000000000
000000c420046f48:  0000000000000000 
000000c420046f50:  0000000000000000
000000c420046f58: !0000000000000000 return PC (should be in goparkunlock)
000000c420046f60: >0000000000000000 vv parkunlock_c (should be a go.func.*+X pointer)
000000c420046f68:  0000000000000000    should be &sweep.lock
000000c420046f70:  0000000000000000    should be pointer to "GC sweep wait" string
000000c420046f78:  0000000000000000    should be 0x0d
000000c420046f80:  0000000000000000    should be 0x14
000000c420046f88:  0000000000000000    should be 0x01
000000c420046f90:  0000000000000000 ^^ 0x38 byte frame for goparkunlock; saved BP (should be 000000c420046fd0)
000000c420046f98:  0000000000000000 return PC (should be in bgsweep)
000000c420046fa0:  0000000000000000 vv &sweep.lock argument (should be a global pointer)
000000c420046fa8:  0000000000000000    should be pointer to "GC sweep wait" string
000000c420046fb0:  0000000000000000    should be 0x0d
000000c420046fb8:  0000000000000014    should be 0x14 (correct!)
000000c420046fc0:  0000000000000001    should be 0x01 (correct!)
000000c420046fc8:  000000003b7c79b1 ^^ 0x30 byte frame for bgsweep; saved BP
000000c420046fd0:  000000c4200476e8 return PC (should be goexit)
000000c420046fd8:  00007ec035c7fc25 chan pointer argument (should be a heap pointer)
000000c420046fe0:  0000000000000001 vv
000000c420046fe8:  000000c420192000 
000000c420046ff0:  0000000000000001  
000000c420046ff8:  00007f6a372c05e0 ^^ 32 bytes of padding added by newproc

So, two of the words are correct, and nearly every other word is clearly incorrect. The two words that are correct make me think that we do have the right stack and it's just terribly corrupted.

This almost looks like an overflow from the next stack in memory. This stack ends at 0xc420047000, which is half-way into a stack span, which will contain a total of 4 2K stacks. The 0xc4200476e8 on the stack is particularly concerning, since that points into what would be the next stack in the span.

In fact, the next stack is goroutine 4's stack, which is timerproc. (The fact that goroutine 4's stack immediately follows goroutine 3's stack is evidence that neither stack has grown from the original allocation, so this is unlikely to be related to stack copying.) But timerproc doesn't do anything especially interesting.

At this point I'm kind of stuck, but getting more samples may help.

@ianlancetaylor ianlancetaylor modified the milestones: Go1.11, Go1.12 Jul 10, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment