Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Shoot-dns-service panics when the API discovery fails for core apiservices #45

Open
Kristian-ZH opened this issue Sep 29, 2022 · 0 comments
Labels
kind/bug Bug lifecycle/rotten Nobody worked on this for 12 months (final aging stage)

Comments

@Kristian-ZH
Copy link

What happened:
We found an shoot-dns-service pod which was constantly failing with panic:

2022-09-29 12:03:23	{"log":"panic: runtime error: invalid memory address or nil pointer dereference"}
2022-09-29 12:03:23	{"log":"[signal SIGSEGV: segmentation violation code=0x1 addr=0x38 pc=0x12a4624]"}
2022-09-29 12:03:23	{"log":"goroutine 1 [running]:"}
2022-09-29 12:03:23	{"log":"github.com/gardener/controller-manager-library/pkg/resources.(*ResourceInfos).doUpdateGroup.func1(0xc0008c06c0, {{{0x0, 0x0}, {0x0, 0x0}}, {0xc000c36210, 0x11}, {0xc00059a800, 0x2, 0x4}, ...}, ...)"}
2022-09-29 12:03:23	{"log":"\t/build/vendor/github.com/gardener/controller-manager-library/pkg/resources/info.go:201 +0x1e4"}
2022-09-29 12:03:23	{"log":"github.com/gardener/controller-manager-library/pkg/resources.(*ResourceInfos).doUpdateGroup(0x4e3?, 0xc000058cc0?, {{{0x0, 0x0}, {0x0, 0x0}}, {0xc000c36210, 0x11}, {0xc00059a800, 0x2, ...}, ...})"}
2022-09-29 12:03:23	{"log":"\t/build/vendor/github.com/gardener/controller-manager-library/pkg/resources/info.go:221 +0x19a"}
2022-09-29 12:03:23	{"log":"github.com/gardener/controller-manager-library/pkg/resources.(*ResourceInfos).update(0xc0008c06c0, 0xc0009be220)"}
2022-09-29 12:03:23	{"log":"\t/build/vendor/github.com/gardener/controller-manager-library/pkg/resources/info.go:162 +0x6ee"}
2022-09-29 12:03:23	{"log":"github.com/gardener/controller-manager-library/pkg/resources.NewResourceInfos({0x24cd720?, 0xc0008ae500}, 0xc000bbd800)"}
2022-09-29 12:03:23	{"log":"\t/build/vendor/github.com/gardener/controller-manager-library/pkg/resources/info.go:102 +0x117"}
2022-09-29 12:03:23	{"log":"github.com/gardener/controller-manager-library/pkg/resources.NewResourceContext({0x24c47b8, 0xc000bb84b0}, {0x24cd720?, 0xc0008ae500}, 0xc0002858f0, 0x0)"}
2022-09-29 12:03:23	{"log":"\t/build/vendor/github.com/gardener/controller-manager-library/pkg/resources/context.go:55 +0x154"}
2022-09-29 12:03:23	{"log":"github.com/gardener/controller-manager-library/pkg/controllermanager/cluster.(*_Cluster).setup(0xc0008ae500, {0x7?, 0x0?})"}
2022-09-29 12:03:23	{"log":"\t/build/vendor/github.com/gardener/controller-manager-library/pkg/controllermanager/cluster/cluster.go:187 +0x50"}
2022-09-29 12:03:23	{"log":"github.com/gardener/controller-manager-library/pkg/controllermanager/cluster.CreateClusterForScheme({0x24c47b8?, 0xc000bb84b0}, {0x24ce940, 0xc00050f230}, {0x24cd780, 0xc000bb2500}, {0x7ffc656340b9, 0x53}, 0xc000bcc000, 0x0)"}
2022-09-29 12:03:23	{"log":"\t/build/vendor/github.com/gardener/controller-manager-library/pkg/controllermanager/cluster/cluster.go:267 +0x615"}
2022-09-29 12:03:23	{"log":"github.com/gardener/controller-manager-library/pkg/controllermanager/cluster.CreateCluster({0x24c47b8, 0xc000bb84b0}, {0x24ce940, 0xc00050f230}, {0x24cd780?, 0xc000bb2500?}, {0x7ffc656340b9, 0x53}, 0xc0001ea770)"}
2022-09-29 12:03:23	{"log":"\t/build/vendor/github.com/gardener/controller-manager-library/pkg/controllermanager/cluster/cluster.go:248 +0x4e7"}
2022-09-29 12:03:23	{"log":"github.com/gardener/controller-manager-library/pkg/controllermanager/cluster.(*_Definitions).create(0xc000447860, {0x24c47b8, 0xc000bb84b0}, {0x24ce940?, 0xc00050f230?}, 0xc0001ea770, {0x24cd780, 0xc0003ed360})"}
2022-09-29 12:03:23	{"log":"\t/build/vendor/github.com/gardener/controller-manager-library/pkg/controllermanager/cluster/definition.go:130 +0x305"}
2022-09-29 12:03:23	{"log":"github.com/gardener/controller-manager-library/pkg/controllermanager/cluster.(*_Definitions).handleCluster(0xc000447860, {0x24c47b8, 0xc000bb84b0}, {0x24ce940?, 0xc00050f230?}, 0xc00048ed20, 0xc000bb1000, 0x26?, {0x20c4faa, 0x7})"}
2022-09-29 12:03:23	{"log":"\t/build/vendor/github.com/gardener/controller-manager-library/pkg/controllermanager/cluster/definition.go:202 +0x349"}
2022-09-29 12:03:23	{"log":"github.com/gardener/controller-manager-library/pkg/controllermanager/cluster.(*_Definitions).CreateClusters(0xc000447860, {0x24c47b8, 0xc000bb84b0}, {0x24ce940?, 0xc00050f230?}, 0x20e1f45?, {0x24c4940?, 0xc000bb7e20?}, 0xc000bb84e0)"}
2022-09-29 12:03:23	{"log":"\t/build/vendor/github.com/gardener/controller-manager-library/pkg/controllermanager/cluster/definition.go:166 +0x57a"}
2022-09-29 12:03:23	{"log":"github.com/gardener/controller-manager-library/pkg/controllermanager.NewControllerManager({0x24c47b8, 0xc000a34270?}, {0x24cb070?, 0xc0003ed6d0})"}
2022-09-29 12:03:23	{"log":"\t/build/vendor/github.com/gardener/controller-manager-library/pkg/controllermanager/controllermanager.go:149 +0xe28"}
2022-09-29 12:03:23	{"log":"github.com/gardener/controller-manager-library/pkg/controllermanager.runCM.func1()"}
2022-09-29 12:03:23	{"log":"\t/build/vendor/github.com/gardener/controller-manager-library/pkg/controllermanager/main.go:140 +0xf2"}
2022-09-29 12:03:23	{"log":"github.com/gardener/controller-manager-library/pkg/run.Run({0x24c47b8, 0xc000a34270}, 0xc0009bf918)"}
2022-09-29 12:03:23	{"log":"\t/build/vendor/github.com/gardener/controller-manager-library/pkg/run/run.go:45 +0x299"}
2022-09-29 12:03:23	{"log":"github.com/gardener/controller-manager-library/pkg/controllermanager.runCM({0x24c47b8?, 0xc000a34270?}, {0x24cb070?, 0xc0003ed6d0?})"}
2022-09-29 12:03:23	{"log":"\t/build/vendor/github.com/gardener/controller-manager-library/pkg/controllermanager/main.go:138 +0x65"}
2022-09-29 12:03:23	{"log":"github.com/gardener/controller-manager-library/pkg/controllermanager.NewCommand.func1(0xc000a3af00?, {0x20c5d02?, 0x12?, 0x13?})"}
2022-09-29 12:03:23	{"log":"\t/build/vendor/github.com/gardener/controller-manager-library/pkg/controllermanager/main.go:125 +0x193"}
2022-09-29 12:03:23	{"log":"github.com/spf13/cobra.(*Command).execute(0xc000a3af00, {0xc0001a2010, 0x12, 0x13})"}
2022-09-29 12:03:23	{"log":"\t/build/vendor/github.com/spf13/cobra/command.go:856 +0x67c"}
2022-09-29 12:03:23	{"log":"github.com/spf13/cobra.(*Command).ExecuteC(0xc000a3af00)"}
2022-09-29 12:03:23	{"log":"\t/build/vendor/github.com/spf13/cobra/command.go:974 +0x3bd"}
2022-09-29 12:03:23	{"log":"github.com/spf13/cobra.(*Command).Execute(...)"}
2022-09-29 12:03:23	{"log":"\t/build/vendor/github.com/spf13/cobra/command.go:902"}
2022-09-29 12:03:23	{"log":"github.com/gardener/controller-manager-library/pkg/controllermanager.Configuration.Start({{0x20e1064, 0x16}, {0x20c55d8, 0x7}, {0x24c4a58, 0xc0004a9a60}, {0x24b6fc8, 0xc000407538}, {0x0}, 0xc0004477a0, ...}, ...)"}
2022-09-29 12:03:23	{"log":"\t/build/vendor/github.com/gardener/controller-manager-library/pkg/controllermanager/main.go:98 +0x30e"}
2022-09-29 12:03:23	{"log":"github.com/gardener/controller-manager-library/pkg/controllermanager.Start({0x20e1064, 0x16}, {0x20e1038, 0x16}, {0x20c55d8?, 0x406719?}, {0xc0009bff58, 0x1, 0x0?})"}
2022-09-29 12:03:23	{"log":"\t/build/vendor/github.com/gardener/controller-manager-library/pkg/controllermanager/main.go:38 +0x158"}
2022-09-29 12:03:23	{"log":"main.main()"}
2022-09-29 12:03:23	{"log":"\t/build/cmd/compound/main.go:89 +0xf0"}

The kubectl api-resources was failing with:

error: unable to retrieve the complete list of server APIs: admissionregistration.k8s.io/v1beta1: the server could not find the requested resource, apiextensions.k8s.io/v1beta1: the server could not find the requested resource, authentication.k8s.io/v1beta1: the server could not find the requested resource, authorization.k8s.io/v1beta1: the server could not find the requested resource, certificates.k8s.io/v1beta1: the server could not find the requested resource, coordination.k8s.io/v1beta1: the server could not find the requested resource, extensions/v1beta1: the server could not find the requested resource, networking.k8s.io/v1beta1: the server could not find the requested resource, rbac.authorization.k8s.io/v1beta1: the server could not find the requested resource, scheduling.k8s.io/v1beta1: the server could not find the requested resource

This issue was workarounded by restarting the kube-apiserver.

What you expected to happen:
Shoot-dns-service pod should not panic in this situation .

How to reproduce it (as minimally and precisely as possible):
We were not able to reproduce it locally.

Anything else we need to know:

Environment:

/cc @dimitar-kostadinov @ialidzhikov

@Kristian-ZH Kristian-ZH added the kind/bug Bug label Sep 29, 2022
@gardener-robot gardener-robot added the lifecycle/stale Nobody worked on this for 6 months (will further age) label Jun 8, 2023
@gardener-robot gardener-robot added lifecycle/rotten Nobody worked on this for 12 months (final aging stage) and removed lifecycle/stale Nobody worked on this for 6 months (will further age) labels Feb 15, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind/bug Bug lifecycle/rotten Nobody worked on this for 12 months (final aging stage)
Projects
None yet
Development

No branches or pull requests

2 participants