Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remove reflector metrics since they are causing a memory leak #74636

Merged
merged 1 commit into from Feb 27, 2019

Conversation

@logicalhan
Copy link
Contributor

logicalhan commented Feb 27, 2019

What type of PR is this?

/kind bug

What this PR does / why we need it:

Currently, our reflector metrics are a memory leak. While this has always been true (for these metrics), this has been exacerbated by the increased watches introduced by #64752. For now, let's just remove these metrics since (I think) no one wants a memory leak. This should be safe because it would not have been possible to have set up reliable monitoring around these metrics since the label names are unstable (as documented).

Which issue(s) this PR fixes:

Fixes #73587

Does this PR introduce a user-facing change?:

This PR removes the following metrics:

  reflector_items_per_list
  reflector_items_per_watch
  reflector_last_resource_version
  reflector_list_duration_seconds
  reflector_lists_total
  reflector_short_watches_total
  reflector_watch_duration_seconds
  reflector_watches_total

While this is a backwards-incompatible change, it would have been impossible to setup reliable monitoring around these metrics since the labels were not stable. 

/sig api-machinery instrumentation
/cc @wojtek-t @lavalamp @liggitt

@smarterclayton

This comment has been minimized.

Copy link
Contributor

smarterclayton commented Feb 27, 2019

I agree, we need to be more judicious because these are generally low value relative to other metrics and are not targeted like our other metrics.

@liggitt

This comment has been minimized.

Copy link
Member

liggitt commented Feb 27, 2019

I'm in favor as well. If there are targeted metrics we want for fixed informers, we could add them in the future, but the impact of these (e.g. #73587) far outweighs any benefit.

cc @kubernetes/sig-instrumentation-pr-reviews

@danielqsj

This comment has been minimized.

Copy link
Member

danielqsj commented Feb 27, 2019

/cc @brancz

@k8s-ci-robot k8s-ci-robot requested a review from brancz Feb 27, 2019

@logicalhan logicalhan force-pushed the logicalhan:reflector-metrics branch from 72707b9 to ca096f8 Feb 27, 2019

@logicalhan

This comment has been minimized.

Copy link
Contributor Author

logicalhan commented Feb 27, 2019

/test pull-kubernetes-bazel-test pull-kubernetes-integration

@@ -403,9 +383,4 @@ func (r *Reflector) setLastSyncResourceVersion(v string) {
r.lastSyncResourceVersionMutex.Lock()
defer r.lastSyncResourceVersionMutex.Unlock()
r.lastSyncResourceVersion = v

rv, err := strconv.Atoi(v)

This comment has been minimized.

@lavalamp

lavalamp Feb 27, 2019

Member

Wow, this removed code is even worse than expected. Good call.

@lavalamp

This comment has been minimized.

Copy link
Member

lavalamp commented Feb 27, 2019

/approve
/lgtm

@k8s-ci-robot

This comment has been minimized.

Copy link
Contributor

k8s-ci-robot commented Feb 27, 2019

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: lavalamp, logicalhan

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@logicalhan

This comment has been minimized.

Copy link
Contributor Author

logicalhan commented Feb 27, 2019

/test pull-kubernetes-integration

@wojtek-t

This comment has been minimized.

Copy link
Member

wojtek-t commented Feb 27, 2019

/lgtm
/retest

Thanks!

@brancz

This comment has been minimized.

Copy link
Member

brancz commented Feb 27, 2019

This is the second time these have caused a leak, so I welcome this 👍

@k8s-ci-robot k8s-ci-robot merged commit a514fa0 into kubernetes:master Feb 27, 2019

16 checks passed

cla/linuxfoundation logicalhan authorized
Details
pull-kubernetes-bazel-build Job succeeded.
Details
pull-kubernetes-bazel-test Job succeeded.
Details
pull-kubernetes-cross Skipped.
pull-kubernetes-e2e-gce Job succeeded.
Details
pull-kubernetes-e2e-gce-100-performance Job succeeded.
Details
pull-kubernetes-e2e-gce-device-plugin-gpu Job succeeded.
Details
pull-kubernetes-godeps Job succeeded.
Details
pull-kubernetes-integration Job succeeded.
Details
pull-kubernetes-kubemark-e2e-gce-big Job succeeded.
Details
pull-kubernetes-local-e2e Skipped.
pull-kubernetes-node-e2e Job succeeded.
Details
pull-kubernetes-typecheck Job succeeded.
Details
pull-kubernetes-verify Job succeeded.
Details
pull-publishing-bot-validate Skipped.
tide In merge pool.
Details
@jrake-revelant

This comment has been minimized.

Copy link

jrake-revelant commented Feb 27, 2019

Thank you for fixing this!

@logicalhan logicalhan deleted the logicalhan:reflector-metrics branch Feb 27, 2019

k8s-ci-robot added a commit that referenced this pull request Feb 28, 2019

Merge pull request #74673 from logicalhan/automated-cherry-pick-of-#7…
…4636-upstream-release-1.12

Automated cherry pick of #74636: Remove reflector metrics as they currently cause a memory

openshift-publish-robot pushed a commit to openshift/kubernetes that referenced this pull request Mar 1, 2019

UPSTREAM: 74636: Remove reflector metrics as they currently cause a m…
…emory leak

ref: kubernetes#73587
ref: kubernetes#74636

Origin-commit: 01380498b02d6dee75e52d9ce54e9a5dffef24fb

openshift-publish-robot pushed a commit to openshift/kubernetes-client-go that referenced this pull request Mar 1, 2019

UPSTREAM: 74636: Remove reflector metrics as they currently cause a m…
…emory leak

ref: kubernetes/kubernetes#73587
ref: kubernetes/kubernetes#74636

Origin-commit: 01380498b02d6dee75e52d9ce54e9a5dffef24fb

Kubernetes-commit: f77a2c16c80223249ead526ca12caa6962117888

k8s-ci-robot added a commit that referenced this pull request Mar 5, 2019

Merge pull request #74672 from logicalhan/automated-cherry-pick-of-#7…
…4636-upstream-release-1.13

Automated cherry pick of #74636: Remove reflector metrics as they currently cause a memory

deads2k added a commit to deads2k/kubernetes that referenced this pull request Mar 14, 2019

UPSTREAM: 74636: Remove reflector metrics as they currently cause a m…
…emory leak

ref: kubernetes#73587
ref: kubernetes#74636

Origin-commit: 01380498b02d6dee75e52d9ce54e9a5dffef24fb

deads2k added a commit to deads2k/kubernetes that referenced this pull request Mar 14, 2019

UPSTREAM: 74636: Remove reflector metrics as they currently cause a m…
…emory leak

ref: kubernetes#73587
ref: kubernetes#74636

Origin-commit: 01380498b02d6dee75e52d9ce54e9a5dffef24fb

openshift-publish-robot pushed a commit to openshift/kubernetes-client-go that referenced this pull request Mar 15, 2019

UPSTREAM: 74636: Remove reflector metrics as they currently cause a m…
…emory leak

ref: kubernetes/kubernetes#73587
ref: kubernetes/kubernetes#74636

Origin-commit: 01380498b02d6dee75e52d9ce54e9a5dffef24fb

Kubernetes-commit: fd85bbcb7e0922b8889c85fad1f5f2d4ca7a3fa7
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.