Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Turn of metrics for reflectors created by watch managers #73624

Closed
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
6 changes: 6 additions & 0 deletions pkg/kubelet/kubelet.go
Expand Up @@ -544,6 +544,12 @@ func NewMainKubelet(kubeCfg *kubeletconfiginternal.KubeletConfiguration,
klet.cloudResourceSyncManager = cloudresource.NewSyncManager(klet.cloud, nodeName, klet.nodeStatusUpdateFrequency)
}

// Set the noop metrics provider for all reflectors that will be created
// after this point, so that watch-based managers (if used) will not
// blow the Kubelet memory usage if there are huge churn of secrets and/or
// configmaps attached to pods running on this node.
cache.SetReflectorMetricsProvider(manager.NoopMetricsProvider{})
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually it won't work, because SetReflectorMetricsProvider is using sync.Once underneath (so once initialized, we can't change it).

That said, I realized that I don't understand why this is happening, because the:

  • metrics factory is defaulted to noop
  • the prometheus metrics need to be explicitly linked and initialized (as they are in apiserver e.g.:
    _ "k8s.io/kubernetes/pkg/util/reflector/prometheus" // for reflector metric registration
    )

So I'm not sure what the exact problem actually is...


var secretManager secret.Manager
var configMapManager configmap.Manager
switch kubeCfg.ConfigMapAndSecretChangeDetectionStrategy {
Expand Down
1 change: 1 addition & 0 deletions pkg/kubelet/util/manager/BUILD
Expand Up @@ -5,6 +5,7 @@ go_library(
srcs = [
"cache_based_manager.go",
"manager.go",
"noop_reflector_metrics.go",
"watch_based_manager.go",
],
importpath = "k8s.io/kubernetes/pkg/kubelet/util/manager",
Expand Down
57 changes: 57 additions & 0 deletions pkg/kubelet/util/manager/noop_reflector_metrics.go
@@ -0,0 +1,57 @@
/*
Copyright 2019 The Kubernetes Authors.

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/

package manager

import (
"k8s.io/client-go/tools/cache"
)

type noopMetric struct{}

func (noopMetric) Inc() {}
func (noopMetric) Dec() {}
func (noopMetric) Observe(float64) {}
func (noopMetric) Set(float64) {}

// NoopMetricsProvider implements interface of reflector metrics
// by faking all the metrics.
type NoopMetricsProvider struct{}

func (NoopMetricsProvider) NewListsMetric(name string) cache.CounterMetric {
return noopMetric{}
}
func (NoopMetricsProvider) NewListDurationMetric(name string) cache.SummaryMetric {
return noopMetric{}
}
func (NoopMetricsProvider) NewItemsInListMetric(name string) cache.SummaryMetric {
return noopMetric{}
}
func (NoopMetricsProvider) NewWatchesMetric(name string) cache.CounterMetric {
return noopMetric{}
}
func (NoopMetricsProvider) NewShortWatchesMetric(name string) cache.CounterMetric {
return noopMetric{}
}
func (NoopMetricsProvider) NewWatchDurationMetric(name string) cache.SummaryMetric {
return noopMetric{}
}
func (NoopMetricsProvider) NewItemsInWatchMetric(name string) cache.SummaryMetric {
return noopMetric{}
}
func (NoopMetricsProvider) NewLastResourceVersionMetric(name string) cache.GaugeMetric {
return noopMetric{}
}