Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

runtime: unexpected return pc for runtime.gopark called from 0x0 #66393

Open
zaisongz opened this issue Mar 19, 2024 · 1 comment
Open

runtime: unexpected return pc for runtime.gopark called from 0x0 #66393

zaisongz opened this issue Mar 19, 2024 · 1 comment
Labels
compiler/runtime Issues related to the Go compiler and/or runtime. WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided.

Comments

@zaisongz
Copy link

Go version

go1.16.15

Output of go env in your module/workspace:

we used official etcd version 3.5.3
https://github.com/etcd-io/etcd/releases/download/v3.5.3/etcd-v3.5.3-linux-amd64.tar.gz

Do not find the go env info.

What did you do?

One of the ETCD server(let's say pod-1) ran into crash loopback for almost 3 days. Then we watched pod-0 fires panic.
I did not want to say there is relationship between the crash loopback and panic. But that was what i observed.

What did you see happen?

2024-03-09T01:22:42.174324508Z
runtime: unexpected return pc for runtime.gopark called from 0x0
stack: frame={sp:0xc005b31d38, fp:0xc005b31d58} stack=[0xc005b2a000,0xc005b32000)
000000c005b31c38: 0000000000000000 0000000000000000
000000c005b31c48: 0000000000000000 0000000000000000
000000c005b31c58: 0000000000000000 0000000000000000
000000c005b31c68: 0000000000000000 0000000000000000
000000c005b31c78: 0000000000000000 0000000000000000
000000c005b31c88: 0000000000000000 0000000000000000
000000c005b31c98: 0000000000000000 0000000000000000
000000c005b31ca8: 0000000000000000 0000000000000000
000000c005b31cb8: 0000000000000000 0000000000000000
000000c005b31cc8: 0000000000000000 0000000000000000
000000c005b31cd8: 0000000000000000 0000000000000000
000000c005b31ce8: 0000000000000000 0000000000000000
000000c005b31cf8: 0000000000000000 0000000000000000
000000c005b31d08: 0000000000000000 0000000000000000
000000c005b31d18: 0000000000000000 0000000000000000
000000c005b31d28: 0000000000000000 0000000000000000
000000c005b31d38: \u003c4040404040404040 4040404040404040
000000c005b31d48: 0000000000000000 !0000000000000000
000000c005b31d58: \u003e0000000000000000 2525252525252525
000000c005b31d68: 2525252525252525 0000000000000000
000000c005b31d78: 0000000000000000 0000000000000000
000000c005b31d88: 0000000000000000 0000000000000000
000000c005b31d98: 0000000000000000 ffffff0000000000
000000c005b31da8: 0000000000000000 0000000000000000
000000c005b31db8: 0000000000000000 0000000000000001
000000c005b31dc8: 0000000000000001 0000000000000002
000000c005b31dd8: 0000000000000002 0000000000000011
000000c005b31de8: 000000c002fd7e48 000000c004d22040
000000c005b31df8: 000000c003d82a80 000000c0024d8000
000000c005b31e08: 000000c0024ea120 000000c0024ea120
000000c005b31e18: 000000c0024ea120 0000000000000000
000000c005b31e28: 000000c003d82b08 000000c002fd7ed4
000000c005b31e38: 000000c003d82bb8 000000c002fd7ed4
000000c005b31e48: 0000000000000002 0000000000000002
fatal error: unknown caller pc

2024-03-09T01:22:42.178142044Z
runtime stack:
runtime.throw(0x1213bd8, 0x11)
\u0009/go/gos/go1.16.15/src/runtime/panic.go:1117 +0x72
runtime.gentraceback(0xffffffffffffffff, 0xffffffffffffffff, 0x0, 0xc003d82a80, 0x0, 0x0, 0x7fffffff, 0x12695e0, 0xc0009afaf8, 0x0, ...)
\u0009/go/gos/go1.16.15/src/runtime/traceback.go:261 +0x1a56
runtime.copystack(0xc003d82a80, 0x8000)
\u0009/go/gos/go1.16.15/src/runtime/stack.go:908 +0x2f1
runtime.shrinkstack(0xc003d82a80)
\u0009/go/gos/go1.16.15/src/runtime/stack.go:1180 +0x13d
runtime.scanstack(0xc003d82a80, 0xc00006fe98)
\u0009/go/gos/go1.16.15/src/runtime/mgcmark.go:720 +0x58e
runtime.markroot.func1()
\u0009/go/gos/go1.16.15/src/runtime/mgcmark.go:233 +0xc6
runtime.markroot(0xc00006fe98, 0x1ca)
\u0009/go/gos/go1.16.15/src/runtime/mgcmark.go:206 +0x33e
runtime.gcDrain(0xc00006fe98, 0x7)
\u0009/go/gos/go1.16.15/src/runtime/mgcmark.go:1014 +0x118
runtime.gcBgMarkWorker.func2()
\u0009/go/gos/go1.16.15/src/runtime/mgc.go:2003 +0x17e
runtime.systemstack(0xc000644000)
\u0009/go/gos/go1.16.15/src/runtime/asm_amd64.s:379 +0x66
runtime.mstart()
\u0009/go/gos/go1.16.15/src/runtime/proc.go:1246

2024-03-09T01:22:42.178333233Z
goroutine 74 [GC worker (idle)]:
runtime.systemstack_switch()
\u0009/go/gos/go1.16.15/src/runtime/asm_amd64.s:339 fp=0xc000612f60 sp=0xc000612f58 pc=0x4705a0
runtime.gcBgMarkWorker()
\u0009/go/gos/go1.16.15/src/runtime/mgc.go:1967 +0x1c7 fp=0xc000612fe0 sp=0xc000612f60 pc=0x41dde7
runtime.goexit()
\u0009/go/gos/go1.16.15/src/runtime/asm_amd64.s:1371 +0x1 fp=0xc000612fe8 sp=0xc000612fe0 pc=0x4722a1
created by runtime.gcBgMarkStartWorkers
\u0009/go/gos/go1.16.15/src/runtime/mgc.go:1835 +0x37

2024-03-09T01:22:42.178404246Z
goroutine 1 [select, 10822 minutes]:
go.etcd.io/etcd/server/v3/etcdmain.startEtcdOrProxyV2(0xc000226000, 0x15, 0x16)
\u0009/go/src/go.etcd.io/etcd/release/etcd/server/etcdmain/etcd.go:216 +0x1472
go.etcd.io/etcd/server/v3/etcdmain.Main(0xc000226000, 0x15, 0x16)
\u0009/go/src/go.etcd.io/etcd/release/etcd/server/etcdmain/main.go:40 +0x13f
main.main()
\u0009/go/src/go.etcd.io/etcd/release/etcd/server/main.go:32 +0x45

2024-03-09T01:22:42.178600358Z
goroutine 601 [select]:
google.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc0003a6460, 0x1, 0x0, 0x0, 0x0, 0x0)
\u0009/go/pkg/mod/google.golang.org/grpc@v1.38.0/internal/transport/controlbuf.go:407 +0xff
google.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc00289a0c0, 0x0, 0x0)
\u0009/go/pkg/mod/google.golang.org/grpc@v1.38.0/internal/transport/controlbuf.go:527 +0x1dd
google.golang.org/grpc/internal/transport.newHTTP2Server.func2(0xc000521380)
\u0009/go/pkg/mod/google.golang.org/grpc@v1.38.0/internal/transport/http2_server.go:292 +0xd7
created by google.golang.org/grpc/internal/transport.newHTTP2Server
\u0009/go/pkg/mod/google.golang.org/grpc@v1.38.0/internal/transport/http2_server.go:289 +0x1110

2024-03-09T01:22:42.178649053Z
goroutine 370 [chan receive, 10822 minutes]:
go.etcd.io/etcd/server/v3/etcdserver/api/rafthttp.(*streamHandler).ServeHTTP(0xc000670bc0, 0x139f0d0, 0xc000154460, 0xc0005c8500)
\u0009/go/src/go.etcd.io/etcd/release/etcd/server/etcdserver/api/rafthttp/http.go:451 +0x1a4e
net/http.(*ServeMux).ServeHTTP(0xc000670c40, 0x139f0d0, 0xc000154460, 0xc0005c8500)
\u0009/go/gos/go1.16.15/src/net/http/server.go:2429 +0x1ad
net/http.(*ServeMux).ServeHTTP(0xc000670c80, 0x139f0d0, 0xc000154460, 0xc0005c8500)
\u0009/go/gos/go1.16.15/src/net/http/server.go:2429 +0x1ad
go.etcd.io/etcd/server/v3/embed.grpcHandlerFunc.func2(0x139f0d0, 0xc000154460, 0xc0005c8500)
\u0009/go/src/go.etcd.io/etcd/release/etcd/server/embed/serve.go:210 +0x5c
net/http.HandlerFunc.ServeHTTP(0xc000c81d80, 0x139f0d0, 0xc000154460, 0xc0005c8500)
\u0009/go/gos/go1.16.15/src/net/http/server.go:2050 +0x44
net/http.serverHandler.ServeHTTP(0xc0001541c0, 0x139f0d0, 0xc000154460, 0xc0005c8500)
\u0009/go/gos/go1.16.15/src/net/http/server.go:2868 +0xa3
net/http.(*conn).serve(0xc0002fa000, 0x13a1cb8, 0xc0015f27c0)
\u0009/go/gos/go1.16.15/src/net/http/server.go:1933 +0x8cd
created by net/http.(*Server).Serve
\u0009/go/gos/go1.16.15/src/net/http/server.go:2994 +0x39b

What did you expect to see?

Can anybody help explain what is happening here, and do we already fixed this kind of problem? i did a search and found that there is several reports similar to this one, but not exactly the same.

@gopherbot gopherbot added the compiler/runtime Issues related to the Go compiler and/or runtime. label Mar 19, 2024
@ianlancetaylor
Copy link
Contributor

Go 1.16 is no longer supported. Is it possible for you to try the current release, Go 1.22?

In any case, this looks like some sort of memory corruption. If it only happened once and can't be recreated, there may not be any way to figure it out.

@dr2chase dr2chase added the WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided. label Mar 19, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
compiler/runtime Issues related to the Go compiler and/or runtime. WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided.
Projects
Status: No status
Development

No branches or pull requests

4 participants