Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

proxy: add some useful metrics #74027

Merged
merged 1 commit into from May 15, 2019

Conversation

@squeed
Copy link
Contributor

commented Feb 13, 2019

/kind feature

What this PR does / why we need it:
This adds some useful metrics around pending changes and last successful
sync time.

The goal is for administrators to be able to alert on proxies that, for
whatever reason, are quite stale.

Does this PR introduce a user-facing change?:

The metric `kube_proxy_sync_proxy_rules_last_timestamp_seconds` is now available, indicating the last time that kube-proxy successfully applied proxying rules.

Signed-off-by: Casey Callendrello cdc@redhat.com

@k8s-ci-robot

This comment has been minimized.

Copy link
Contributor

commented Feb 13, 2019

Hi @squeed. Thanks for your PR.

I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@squeed

This comment has been minimized.

Copy link
Contributor Author

commented Feb 13, 2019

/sig network
/sig instrumentation

@@ -1359,6 +1359,7 @@ func (proxier *Proxier) syncProxyRules() {
if proxier.healthzServer != nil {
proxier.healthzServer.UpdateTimestamp()
}
metrics.SyncProxyRulesLastTimestamp.Set(time.Now().Unix())

This comment has been minimized.

Copy link
@SuperQ

SuperQ Feb 13, 2019

Contributor

There is a handy .SetToCurrentTime() method.

This comment has been minimized.

Copy link
@squeed

squeed Feb 13, 2019

Author Contributor

Ah, thanks!

@squeed squeed force-pushed the squeed:kube-proxy-metrics branch from cb0e915 to 592f251 Feb 13, 2019

@squeed

This comment has been minimized.

Copy link
Contributor Author

commented Feb 13, 2019

cc @mm4tt, who seems to be working on this too.

@mm4tt

This comment has been minimized.

Copy link
Contributor

commented Feb 14, 2019

Thanks for cc'ing me. I was working on something slightly different, i.e. adding the network programming latency metric, but the metrics you're adding might be useful in debugging our scalability tests.
@kubernetes/sig-scalability

@squeed

This comment has been minimized.

Copy link
Contributor Author

commented Feb 14, 2019

The end-to-end latency metric is very clever! That's definitely the most useful.

These are just a few simpler metrics I wish I had when diagnosing customer issues.

@squeed

This comment has been minimized.

Copy link
Contributor Author

commented Feb 20, 2019

Can I get an ok-to-test, please? Thanks.

@dims

This comment has been minimized.

Copy link
Member

commented Feb 20, 2019

/ok-to-test

@squeed

This comment has been minimized.

Copy link
Contributor Author

commented Feb 20, 2019

/retest

@squeed

This comment has been minimized.

Copy link
Contributor Author

commented Feb 26, 2019

This is all green, and ready for review.

@squeed squeed force-pushed the squeed:kube-proxy-metrics branch from 592f251 to 1605e3c Mar 7, 2019

@squeed squeed force-pushed the squeed:kube-proxy-metrics branch from 1605e3c to a2f8aa7 Apr 17, 2019

@squeed

This comment has been minimized.

Copy link
Contributor Author

commented Apr 17, 2019

/retest

@squeed

This comment has been minimized.

Copy link
Contributor Author

commented Apr 18, 2019

@kubernetes/sig-network-pr-reviews This is all green - can I get a review?

prometheus.CounterOpts{
Subsystem: kubeProxySubsystem,
Name: "sync_proxy_rules_service_changes_total",
Help: "Cumulative proxy rules Endpoint changes",

This comment has been minimized.

Copy link
@dcbw

dcbw May 2, 2019

Member

Service

prometheus.GaugeOpts{
Subsystem: kubeProxySubsystem,
Name: "sync_proxy_rules_service_changes_pending",
Help: "Pending proxy rules Endpoint changes",

This comment has been minimized.

Copy link
@dcbw

dcbw May 2, 2019

Member

Service

@dcbw

This comment has been minimized.

Copy link
Member

commented May 2, 2019

/lgtm cancel

@k8s-ci-robot k8s-ci-robot removed the lgtm label May 2, 2019

@dcbw

This comment has been minimized.

Copy link
Member

commented May 2, 2019

Just a small doc fixup required.

proxy: add some useful metrics
This adds some useful metrics around pending changes and last successful
sync time.

The goal is for administrators to be able to alert on proxies that, for
whatever reason, are quite stale.

Signed-off-by: Casey Callendrello <cdc@redhat.com>

@squeed squeed force-pushed the squeed:kube-proxy-metrics branch from a2f8aa7 to 017f57a May 7, 2019

@squeed

This comment has been minimized.

Copy link
Contributor Author

commented May 7, 2019

@dcbw good catch, fixed.

@squeed

This comment has been minimized.

Copy link
Contributor Author

commented May 8, 2019

OK, this is all green. @thockin @matchstick could I get an approval?

@dcbw

This comment has been minimized.

Copy link
Member

commented May 8, 2019

/lgtm

@k8s-ci-robot k8s-ci-robot added the lgtm label May 8, 2019

@dcbw

This comment has been minimized.

Copy link
Member

commented May 13, 2019

/priority important-soon

@thockin
Copy link
Member

left a comment

These seem fine.

It's probably worthwhile to think/write about what sorts of questions we want to ask about kube-proxy, to make sure we have the right metrics in the first place.

@wojtek-t has spent some time thinking about this

Thanks!

/lgtm
/approve

@k8s-ci-robot

This comment has been minimized.

Copy link
Contributor

commented May 15, 2019

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: squeed, thockin

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-ci-robot k8s-ci-robot merged commit 7474379 into kubernetes:master May 15, 2019

20 checks passed

cla/linuxfoundation squeed authorized
Details
pull-kubernetes-bazel-build Job succeeded.
Details
pull-kubernetes-bazel-test Job succeeded.
Details
pull-kubernetes-conformance-image-test Skipped.
pull-kubernetes-cross Skipped.
pull-kubernetes-dependencies Job succeeded.
Details
pull-kubernetes-e2e-gce Job succeeded.
Details
pull-kubernetes-e2e-gce-100-performance Job succeeded.
Details
pull-kubernetes-e2e-gce-csi-serial Skipped.
pull-kubernetes-e2e-gce-device-plugin-gpu Job succeeded.
Details
pull-kubernetes-e2e-gce-storage-slow Skipped.
pull-kubernetes-godeps Skipped.
pull-kubernetes-integration Job succeeded.
Details
pull-kubernetes-kubemark-e2e-gce-big Job succeeded.
Details
pull-kubernetes-local-e2e Skipped.
pull-kubernetes-node-e2e Job succeeded.
Details
pull-kubernetes-typecheck Job succeeded.
Details
pull-kubernetes-verify Job succeeded.
Details
pull-publishing-bot-validate Skipped.
tide In merge pool.
Details
k8s-ci-robot added a commit that referenced this pull request Jun 20, 2019
Merge pull request #78602 from paulgmiller/release-1.14
cherry pick of #74027: proxy: add some useful metrics
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.