Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Get rid of the debug logging of Shoots in the shootUpdate func #4046

Merged
merged 1 commit into from
May 12, 2021
Merged

Get rid of the debug logging of Shoots in the shootUpdate func #4046

merged 1 commit into from
May 12, 2021

Conversation

ialidzhikov
Copy link
Member

/area quality
/kind bug
/kind cleanup

We observe some interesting panics with gardenlet@v1.22.2 such as:

E0512 10:36:12.474585       1 runtime.go:78] Observed a panic: &reflect.ValueError{Method:"reflect.Value.Type", Kind:0x0} (reflect: call of reflect.Value.Type on zero Value)
goroutine 655 [running]:
k8s.io/apimachinery/pkg/util/runtime.logPanic(0x1f5b220, 0xc01565d548)
	/go/src/github.com/gardener/gardener/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:74 +0x95
k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0)
	/go/src/github.com/gardener/gardener/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:48 +0x86
panic(0x1f5b220, 0xc01565d548)
	/usr/local/go/src/runtime/panic.go:965 +0x1b9
encoding/json.(*encodeState).marshal.func1(0xc00ae37ba0)
	/usr/local/go/src/encoding/json/encode.go:328 +0x8d
panic(0x1f5b220, 0xc01565d548)
	/usr/local/go/src/runtime/panic.go:965 +0x1b9
reflect.Value.Type(0x0, 0x0, 0x0, 0xc00fa44c50, 0x0)
	/usr/local/go/src/reflect/value.go:1908 +0x189
encoding/json.stringEncoder(0xc00e68ce00, 0x0, 0x0, 0x0, 0xc00fa40100)
	/usr/local/go/src/encoding/json/encode.go:620 +0x4c
encoding/json.mapEncoder.encode(0x23417e8, 0xc00e68ce00, 0x1f6ddc0, 0xc0105100b8, 0x195, 0x100)
	/usr/local/go/src/encoding/json/encode.go:813 +0x366
encoding/json.structEncoder.encode(0xc00086e000, 0x10, 0x10, 0xc0008338f0, 0xc00e68ce00, 0x21d7020, 0xc010510020, 0x199, 0x100)
	/usr/local/go/src/encoding/json/encode.go:761 +0x2ab
encoding/json.structEncoder.encode(0xc00a69bb00, 0x5, 0x8, 0xc013a574d0, 0xc00e68ce00, 0x20c4c20, 0xc010510000, 0x199, 0x1f60100)
	/usr/local/go/src/encoding/json/encode.go:761 +0x2ab
encoding/json.ptrEncoder.encode(0xc013a57500, 0xc00e68ce00, 0x221c080, 0xc010510000, 0x16, 0x2210100)
	/usr/local/go/src/encoding/json/encode.go:944 +0x125
encoding/json.(*encodeState).reflectValue(0xc00e68ce00, 0x221c080, 0xc010510000, 0x16, 0xc00ae30100)
	/usr/local/go/src/encoding/json/encode.go:360 +0x82
encoding/json.(*encodeState).marshal(0xc00e68ce00, 0x221c080, 0xc010510000, 0xc015dc0100, 0x0, 0x0)
	/usr/local/go/src/encoding/json/encode.go:332 +0xf9
encoding/json.Marshal(0x221c080, 0xc010510000, 0xc00ae37c98, 0x1, 0x1, 0x0, 0x0)
	/usr/local/go/src/encoding/json/encode.go:161 +0x52
github.com/gardener/gardener/pkg/gardenlet/controller/shoot.(*Controller).shootUpdate(0xc005613080, 0x221c080, 0xc010510000, 0x221c080, 0xc0159d0700)
	/go/src/github.com/gardener/gardener/pkg/gardenlet/controller/shoot/shoot_control.go:76 +0x70
k8s.io/client-go/tools/cache.ResourceEventHandlerFuncs.OnUpdate(...)
	/go/src/github.com/gardener/gardener/vendor/k8s.io/client-go/tools/cache/controller.go:238
k8s.io/client-go/tools/cache.FilteringResourceEventHandler.OnUpdate(0xc005af9050, 0x2550380, 0xc0056f4858, 0x221c080, 0xc010510000, 0x221c080, 0xc0159d0700)
	/go/src/github.com/gardener/gardener/vendor/k8s.io/client-go/tools/cache/controller.go:273 +0x122
k8s.io/client-go/tools/cache.(*processorListener).run.func1()
	/go/src/github.com/gardener/gardener/vendor/k8s.io/client-go/tools/cache/shared_informer.go:775 +0x1c5
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0xc000846f60)
	/go/src/github.com/gardener/gardener/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:155 +0x5f
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc00ae37f60, 0x2505440, 0xc00557a540, 0x1f47201, 0xc00923c8a0)
	/go/src/github.com/gardener/gardener/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:156 +0x9b
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc000846f60, 0x3b9aca00, 0x0, 0x431b01, 0xc00923c8a0)
	/go/src/github.com/gardener/gardener/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133 +0x98
k8s.io/apimachinery/pkg/util/wait.Until(...)
	/go/src/github.com/gardener/gardener/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:90
k8s.io/client-go/tools/cache.(*processorListener).run(0xc006072200)
	/go/src/github.com/gardener/gardener/vendor/k8s.io/client-go/tools/cache/shared_informer.go:771 +0x95
k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1(0xc000400680, 0xc0056c5f40)
	/go/src/github.com/gardener/gardener/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:73 +0x51
created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start
	/go/src/github.com/gardener/gardener/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:71 +0x65
time="2021-05-12T10:36:12Z" level=info msg="Found secrets in namespace \"seed-az-us1\": internal domain secret \"internal-domain-internal-canary-k8s-ondemand-com\" for domain \"internal.canary.k8s.ondemand.com\", default domain secret \"default-domain-shoot-canary-k8s-hana-ondemand-com\" for domain \"shoot.canary.k8s-hana.ondemand.com\", alerting secret \"alerting-0\", OpenVPN Diffie Hellman secret \"openvpn-diffie-hellman-key\", monitoring basic auth secret \"monitoring-ingress-credentials\""
panic: reflect: call of reflect.Value.Type on zero Value [recovered]
	panic: reflect: call of reflect.Value.Type on zero Value [recovered]
	panic: reflect: call of reflect.Value.Type on zero Value

goroutine 655 [running]:
k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0)
	/go/src/github.com/gardener/gardener/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:55 +0x109
panic(0x1f5b220, 0xc01565d548)
	/usr/local/go/src/runtime/panic.go:965 +0x1b9
encoding/json.(*encodeState).marshal.func1(0xc00ae37ba0)
	/usr/local/go/src/encoding/json/encode.go:328 +0x8d
panic(0x1f5b220, 0xc01565d548)
	/usr/local/go/src/runtime/panic.go:965 +0x1b9
reflect.Value.Type(0x0, 0x0, 0x0, 0xc00fa44c50, 0x0)
	/usr/local/go/src/reflect/value.go:1908 +0x189
encoding/json.stringEncoder(0xc00e68ce00, 0x0, 0x0, 0x0, 0xc00fa40100)
	/usr/local/go/src/encoding/json/encode.go:620 +0x4c
encoding/json.mapEncoder.encode(0x23417e8, 0xc00e68ce00, 0x1f6ddc0, 0xc0105100b8, 0x195, 0x100)
	/usr/local/go/src/encoding/json/encode.go:813 +0x366
encoding/json.structEncoder.encode(0xc00086e000, 0x10, 0x10, 0xc0008338f0, 0xc00e68ce00, 0x21d7020, 0xc010510020, 0x199, 0x100)
	/usr/local/go/src/encoding/json/encode.go:761 +0x2ab
encoding/json.structEncoder.encode(0xc00a69bb00, 0x5, 0x8, 0xc013a574d0, 0xc00e68ce00, 0x20c4c20, 0xc010510000, 0x199, 0x1f60100)
	/usr/local/go/src/encoding/json/encode.go:761 +0x2ab
encoding/json.ptrEncoder.encode(0xc013a57500, 0xc00e68ce00, 0x221c080, 0xc010510000, 0x16, 0x2210100)
	/usr/local/go/src/encoding/json/encode.go:944 +0x125
encoding/json.(*encodeState).reflectValue(0xc00e68ce00, 0x221c080, 0xc010510000, 0x16, 0xc00ae30100)
	/usr/local/go/src/encoding/json/encode.go:360 +0x82
encoding/json.(*encodeState).marshal(0xc00e68ce00, 0x221c080, 0xc010510000, 0xc015dc0100, 0x0, 0x0)
	/usr/local/go/src/encoding/json/encode.go:332 +0xf9
encoding/json.Marshal(0x221c080, 0xc010510000, 0xc00ae37c98, 0x1, 0x1, 0x0, 0x0)
	/usr/local/go/src/encoding/json/encode.go:161 +0x52
github.com/gardener/gardener/pkg/gardenlet/controller/shoot.(*Controller).shootUpdate(0xc005613080, 0x221c080, 0xc010510000, 0x221c080, 0xc0159d0700)
	/go/src/github.com/gardener/gardener/pkg/gardenlet/controller/shoot/shoot_control.go:76 +0x70
k8s.io/client-go/tools/cache.ResourceEventHandlerFuncs.OnUpdate(...)
	/go/src/github.com/gardener/gardener/vendor/k8s.io/client-go/tools/cache/controller.go:238
k8s.io/client-go/tools/cache.FilteringResourceEventHandler.OnUpdate(0xc005af9050, 0x2550380, 0xc0056f4858, 0x221c080, 0xc010510000, 0x221c080, 0xc0159d0700)
	/go/src/github.com/gardener/gardener/vendor/k8s.io/client-go/tools/cache/controller.go:273 +0x122
k8s.io/client-go/tools/cache.(*processorListener).run.func1()
	/go/src/github.com/gardener/gardener/vendor/k8s.io/client-go/tools/cache/shared_informer.go:775 +0x1c5
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0xc000846f60)
	/go/src/github.com/gardener/gardener/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:155 +0x5f
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc00ae37f60, 0x2505440, 0xc00557a540, 0x1f47201, 0xc00923c8a0)
	/go/src/github.com/gardener/gardener/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:156 +0x9b
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc000846f60, 0x3b9aca00, 0x0, 0x431b01, 0xc00923c8a0)
	/go/src/github.com/gardener/gardener/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133 +0x98
k8s.io/apimachinery/pkg/util/wait.Until(...)
	/go/src/github.com/gardener/gardener/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:90
k8s.io/client-go/tools/cache.(*processorListener).run(0xc006072200)
	/go/src/github.com/gardener/gardener/vendor/k8s.io/client-go/tools/cache/shared_informer.go:771 +0x95
k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1(0xc000400680, 0xc0056c5f40)
	/go/src/github.com/gardener/gardener/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:73 +0x51
created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start
	/go/src/github.com/gardener/gardener/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:71 +0x65

As a short term mitigation we can remove the corresponding code that is causing the trouble. The logging was with debug level so it was not enabled by default and it was not quite useful in large landscape as it generates tons of debug logs.
As an alternative audit logging can be used or other means.

Release note:

Apply a mitigation that will prevent gardenlet to panic under certain circumstances.

@ialidzhikov ialidzhikov requested a review from a team as a code owner May 12, 2021 11:17
@gardener-robot gardener-robot added area/quality Output qualification (tests, checks, scans, automation in general, etc.) related kind/bug Bug kind/cleanup Something that is not needed anymore and can be cleaned up needs/review size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. labels May 12, 2021
@ialidzhikov ialidzhikov changed the title Get rid of the debug logging of Shoots in the shootUpdate Get rid of the debug logging of Shoots in the shootUpdate func May 12, 2021
@rfranzke
Copy link
Member

Can you also remove https://github.com/gardener/gardener/blob/master/pkg/gardenlet/controller/seed/seed_control.go#L313?

@ialidzhikov
Get rid of the debug logging of Shoots in the shootUpdate func
89083a3 
Signed-off-by: ialidzhikov <i.alidjikov@gmail.com>
rfranzke
rfranzke approved these changes May 12, 2021
View reviewed changes
Copy link
Member

@rfranzke rfranzke left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/lgtm

@ialidzhikov ialidzhikov mentioned this pull request May 12, 2021
Merged
vpnachev
vpnachev approved these changes May 12, 2021
View reviewed changes
Copy link
Member

@vpnachev vpnachev left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/lgtm

@rfranzke rfranzke merged commit a0c0f7b into gardener:master May 12, 2021
@ialidzhikov ialidzhikov deleted the mitigate/gardenlet-panic branch May 13, 2021 07:22
@ialidzhikov ialidzhikov added this to the v1.23 milestone May 14, 2021
krgostev pushed a commit to krgostev/gardener that referenced this pull request Apr 21, 2022
@ialidzhikov
Get rid of the debug logging of Shoots in the shootUpdate func (garde…
0f92a52 
…ner#4046)

Signed-off-by: ialidzhikov <i.alidjikov@gmail.com>
krgostev pushed a commit to krgostev/gardener that referenced this pull request Jul 5, 2022
@ialidzhikov
Get rid of the debug logging of Shoots in the shootUpdate func (garde…
7eae22f 
…ner#4046)

Signed-off-by: ialidzhikov <i.alidjikov@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@vpnachev vpnachev vpnachev approved these changes

@rfranzke rfranzke rfranzke approved these changes

Assignees
No one assigned
Labels
area/quality Output qualification (tests, checks, scans, automation in general, etc.) related kind/bug Bug kind/cleanup Something that is not needed anymore and can be cleaned up size/XS Denotes a PR that changes 0-9 lines, ignoring generated files.
Projects
None yet
Milestone
v1.23
Development

Successfully merging this pull request may close these issues.

None yet
6 participants
@ialidzhikov @rfranzke @vpnachev @gardener-robot-ci-2 @gardener-robot-ci-3 @gardener-robot