New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Change proxy metrics to conform metrics guidelines #72334

Merged
merged 2 commits into from Jan 26, 2019

Conversation

@danielqsj
Copy link
Member

danielqsj commented Dec 26, 2018

What type of PR is this?

/kind feature
/sig instrumentation

What this PR does / why we need it:

As part of kubernetes metrics overhaul, change proxy metrics to conform Kubernetes metrics instrumentation guidelines.

Which issue(s) this PR fixes:

Fixes #

Special notes for your reviewer:

This patch does not remove the existing metrics but mark them as deprecated.
We need 2 releases for users to convert monitoring configuration.

Does this PR introduce a user-facing change?:

Change proxy metrics to conform metrics guidelines.
The metrics `sync_proxy_rules_latency_microseconds` is deprecated, and will be removed in a future release, please convert to metrics`sync_proxy_rules_latency_seconds`.
@danielqsj

This comment has been minimized.

Copy link
Member Author

danielqsj commented Dec 26, 2018

/assign @brancz

@logicalhan
Copy link
Contributor

logicalhan left a comment

/lgtm

prometheus.HistogramOpts{
Subsystem: kubeProxySubsystem,
Name: "sync_proxy_rules_latency_microseconds",
Help: "SyncProxyRules latency",
Help: "SyncProxyRules latency in microseconds",

This comment has been minimized.

@logicalhan

logicalhan Dec 26, 2018

Contributor

Should we output the deprecation notice in the help?

This comment has been minimized.

@danielqsj

danielqsj Dec 27, 2018

Author Member

Great idea. This kind of notice should be applied in all deprecation metrics. I will mark all of them later.

@k8s-ci-robot k8s-ci-robot removed the lgtm label Jan 8, 2019

@danielqsj

This comment has been minimized.

Copy link
Member Author

danielqsj commented Jan 8, 2019

@brancz @logicalhan
Added Deprecated in metrics description. PTAL

@danielqsj

This comment has been minimized.

Copy link
Member Author

danielqsj commented Jan 8, 2019

/retest

@brancz

This comment has been minimized.

Copy link
Member

brancz commented Jan 10, 2019

/lgtm
/approve

@k8s-ci-robot k8s-ci-robot added the lgtm label Jan 10, 2019

@brancz

This comment has been minimized.

Copy link
Member

brancz commented Jan 10, 2019

/assign @thockin

@thockin

This comment has been minimized.

Copy link
Member

thockin commented Jan 11, 2019

Regarding removal - do we consider metrics part of the admin CLI or part of the API? Metrics are not covered by the deprecation policy, but maybe they should be>

@thockin
Copy link
Member

thockin left a comment

One question to make sure I understand. assuming that is right, LGTM

@@ -643,7 +643,8 @@ func (proxier *Proxier) syncProxyRules() {

start := time.Now()
defer func() {
metrics.SyncProxyRulesLatency.Observe(metrics.SinceInMicroseconds(start))
metrics.SyncProxyRulesLatency.Observe(metrics.SinceInSeconds(start))

This comment has been minimized.

@thockin

thockin Jan 11, 2019

Member

Is this dropping precision? why? I expect that most of the time this takes < 1 second, butthe difference between 0.2 seconds and 0.9 seconds is kind of meaningful. It seems to be float so maybe it's just weird naming?

This comment has been minimized.

@danielqsj

danielqsj Jan 14, 2019

Author Member

@thockin In fact, SinceInSeconds (nanoseconds level) have higher precision than SinceInMicroseconds (microseconds level).
For example:
The result of this case:

fmt.Printf("SinceInMicroseconds: %v\nSinceInSeconds: %v\n", SinceInMicroseconds(time.Now().Add(-time.Second)), SinceInSeconds(time.Now().Add(-time.Second)))

is

SinceInMicroseconds: 1.000001e+06
SinceInSeconds: 1.000000806

The name SinceInSeconds is used to show its unit as seconds.
But we can discuss about this naming. cc @brancz

This comment has been minimized.

@brancz

brancz Jan 14, 2019

Member

We're not loosing precision as it's still a float64, this is just a consistency change, to make sure all metrics use the base unit, which is the best practice so it is easier for different metrics to be used with each other (this is part of the Prometheus instrumentation best practices, which among others includes using seconds as time base unit. Complying to the Prometheus best practices is one of the motivations for the metrics overhaul KEP, which this is part of.

@brancz

This comment has been minimized.

Copy link
Member

brancz commented Jan 14, 2019

@thockin as the KEP mentions:

While uncertain at this point, once cleaned up, this effort may put us a step closer to having stability guarantees for Kubernetes around metrics. Currently metrics are excluded from any kind of stability requirements.

We absolutely want to have deprecation/stability guarantees around metrics, but are not comfortable doing so at the moment, this KEP is in preparation for that 🙂 . It is sig-instrumentation's plan however, and after we've done the cleanup and put tooling in place to ensure all metrics do comply to the guidelines we intend to put stability guarantees in place.

@brancz

This comment has been minimized.

Copy link
Member

brancz commented Jan 17, 2019

@thockin ping 🙂

@thockin

This comment has been minimized.

Copy link
Member

thockin commented Jan 26, 2019

I look forward to a discussion of deprecation of metrics :)

/lgtm
/approve

@k8s-ci-robot

This comment has been minimized.

Copy link
Contributor

k8s-ci-robot commented Jan 26, 2019

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: brancz, danielqsj, thockin

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-ci-robot k8s-ci-robot merged commit b8d6de3 into kubernetes:master Jan 26, 2019

19 checks passed

cla/linuxfoundation danielqsj authorized
Details
pull-kubernetes-bazel-build Job succeeded.
Details
pull-kubernetes-bazel-test Job succeeded.
Details
pull-kubernetes-cross Skipped
pull-kubernetes-e2e-gce Job succeeded.
Details
pull-kubernetes-e2e-gce-100-performance Job succeeded.
Details
pull-kubernetes-e2e-gce-device-plugin-gpu Job succeeded.
Details
pull-kubernetes-e2e-gke Skipped
pull-kubernetes-e2e-kops-aws Job succeeded.
Details
pull-kubernetes-e2e-kubeadm-gce Skipped
pull-kubernetes-godeps Skipped
pull-kubernetes-integration Job succeeded.
Details
pull-kubernetes-kubemark-e2e-gce-big Job succeeded.
Details
pull-kubernetes-local-e2e Skipped
pull-kubernetes-local-e2e-containerized Skipped
pull-kubernetes-node-e2e Job succeeded.
Details
pull-kubernetes-typecheck Job succeeded.
Details
pull-kubernetes-verify Job succeeded.
Details
tide In merge pool.
Details
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment