x/debug/cmd/viewcore: error while trying to run command "objects"

[chenk008]

Go version

go version go1.20.11 linux/amd64

What operating system and processor architecture are you using (go env)?

GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/root/.cache/go-build"
GOENV="/root/.config/go/env"
GOEXE=""
GOEXPERIMENT=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOINSECURE=""
GOMODCACHE="/root/go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/root/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/local/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"
GOVCS=""
GOVERSION="go1.20.11"
GCCGO="gccgo"
GOAMD64="v1"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/root/debug/go.mod"
GOWORK=""
CGO_CFLAGS="-O2 -g"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-O2 -g"
CGO_FFLAGS="-O2 -g"
CGO_LDFLAGS="-O2 -g"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -Wl,--no-gc-sections -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build559913390=/tmp/go-build -gno-record-gcc-switches"

What did you do?

1. user gcore to dump kubelet , saved to core.70006
2. run viewcore ~/core.70006 --exe /usr/bin/kubelet and objects

Error while trying to run command "objects": address 3b79634f6c6c754e is not mapped in the core file
Stack: goroutine 1 [running]:
runtime/debug.Stack()
	/usr/local/go/src/runtime/debug/stack.go:24 +0x65
main.capturePanic.func1()
	/root/debug/cmd/viewcore/main.go:372 +0x45
panic({0x7e3a00, 0xc01cd31600})
	/usr/local/go/src/runtime/panic.go:884 +0x213
golang.org/x/debug/internal/core.(*Process).ReadUint64(0x80d3e0?, 0x9b0000c0265230f0?)
	/root/debug/internal/core/read.go:81 +0x1b3
golang.org/x/debug/internal/core.(*Process).ReadUintptr(0x30?, 0xc01450aeb8?)
	/root/debug/internal/core/read.go:120 +0x36
golang.org/x/debug/internal/gocore.region.Uintptr({0xc008ce6000?, 0xc019637410?, 0xc0000c8000?})
	/root/debug/internal/gocore/region.go:40 +0x36
golang.org/x/debug/internal/gocore.(*Process).runtimeType2Type(0xc008ce6000, 0x3b79634f6c6c754e, 0x852ff4?)
	/root/debug/internal/gocore/type.go:148 +0x1aa
golang.org/x/debug/internal/gocore.(*Process).typeObject(0xc008ce6000, 0xc0014b42c8, 0xc00c5e0a50, {0x8f56d0, 0xc008cdc000}, 0xc026523590)
	/root/debug/internal/gocore/type.go:630 +0x98f
golang.org/x/debug/internal/gocore.(*Process).typeObject(0xc008ce6000, 0xc0014b42a0, 0xc00c5e0d20, {0x8f56d0, 0xc008cdc000}, 0xc026523590)
	/root/debug/internal/gocore/type.go:740 +0x51e
golang.org/x/debug/internal/gocore.(*Process).typeHeap.func1()
	/root/debug/internal/gocore/type.go:523 +0x3f4
sync.(*Once).doSlow(0xc01cd28d20?, 0x4100c5?)
	/usr/local/go/src/sync/once.go:74 +0xc2
sync.(*Once).Do(...)
	/usr/local/go/src/sync/once.go:65
golang.org/x/debug/internal/gocore.(*Process).typeHeap(0x413464?)
	/root/debug/internal/gocore/type.go:377 +0x47
golang.org/x/debug/internal/gocore.(*Process).Type(0xc008ce6000, 0xc011ecc030?)
	/root/debug/internal/gocore/object.go:242 +0x25
main.typeName(0xc008ce6000, 0xc000000000)
	/root/debug/cmd/viewcore/main.go:768 +0xac
main.runObjects.func1(0x7eae40?)
	/root/debug/cmd/viewcore/main.go:612 +0x2c
golang.org/x/debug/internal/gocore.(*Process).ForEachObject(0xc008ce6000, 0xc026523a08)
	/root/debug/internal/gocore/object.go:202 +0xfa
main.runObjects(0xb5c100?, {0xb926c0?, 0x0?, 0x0?})
	/root/debug/cmd/viewcore/main.go:611 +0x87
github.com/spf13/cobra.(*Command).execute(0xb5c100, {0xb926c0, 0x0, 0x0})
	/root/go/pkg/mod/github.com/spf13/cobra@v0.0.3/command.go:766 +0x663
github.com/spf13/cobra.(*Command).ExecuteC(0xc000c8a000)
	/root/go/pkg/mod/github.com/spf13/cobra@v0.0.3/command.go:852 +0x2e5
github.com/spf13/cobra.(*Command).Execute(...)
	/root/go/pkg/mod/github.com/spf13/cobra@v0.0.3/command.go:800
main.runRoot.func2()
	/root/debug/cmd/viewcore/main.go:361 +0x8a
main.capturePanic(0x8f3e20?)
	/root/debug/cmd/viewcore/main.go:376 +0x62
main.runRoot(0xb5aba0, {0xc000038240?, 0x2?, 0x2?})
	/root/debug/cmd/viewcore/main.go:358 +0x6e5
github.com/spf13/cobra.(*Command).execute(0xb5aba0, {0xc0000240a0, 0x2, 0x2})
	/root/go/pkg/mod/github.com/spf13/cobra@v0.0.3/command.go:766 +0x663
github.com/spf13/cobra.(*Command).ExecuteC(0xb5aba0)
	/root/go/pkg/mod/github.com/spf13/cobra@v0.0.3/command.go:852 +0x2e5
github.com/spf13/cobra.(*Command).Execute(...)
	/root/go/pkg/mod/github.com/spf13/cobra@v0.0.3/command.go:800
main.main()
	/root/debug/cmd/viewcore/main.go:244 +0x12a

What did you expect to see?

viewcore can analyze this coredump

What did you see instead?

1. viewcore panic and fail
2. I have tried multiple golang versions and kubelet versions, it always paniced on address 3b79634f6c6c754e

[chenk008]

Maybe it is related to how kubelet is built?

[chenk008]

Similar issue #64442.

After run object, the next command object works.

[chenk008]

initTypeHeap run once at the first command is running.
When the next command runs, gocore.Process is reused because of coreCache. The panic is absence.

The type which failed to parse is KindIface k8s.io/kubernetes/pkg/kubelet/volumemanager/cache.DesiredStateOfWorld, it is an interface.

[cagedmantis]

cc @golang/runtime