Change kubelet metrics to conform metrics guidelines #72470

danielqsj · 2019-01-02T03:10:53Z

What type of PR is this?

/kind feature
/sig instrumentation

What this PR does / why we need it:

As part of kubernetes metrics overhaul, change kubelet metrics to conform Kubernetes metrics instrumentation guidelines.
Change kubelet metrics to histogram for better aggregation.

Which issue(s) this PR fixes:

Fixes #

Special notes for your reviewer:

This patch does not remove the existing metrics but mark them as deprecated.
We need 2 releases for users to convert monitoring configuration.

Does this PR introduce a user-facing change?:

Change kubelet metrics to conform metrics guidelines.
The following metrics are deprecated, and will be removed in a future release:
* `kubelet_pod_worker_latency_microseconds`
* `kubelet_pod_start_latency_microseconds`
* `kubelet_cgroup_manager_latency_microseconds`
* `kubelet_pod_worker_start_latency_microseconds`
* `kubelet_pleg_relist_latency_microseconds`
* `kubelet_pleg_relist_interval_microseconds`
* `kubelet_eviction_stats_age_microseconds`
* `kubelet_runtime_operations`
* `kubelet_runtime_operations_latency_microseconds`
* `kubelet_runtime_operations_errors`
* `kubelet_device_plugin_registration_count`
* `kubelet_device_plugin_alloc_latency_microseconds`
Please convert to the following metrics:
* `kubelet_pod_worker_duration_seconds`
* `kubelet_pod_start_duration_seconds`
* `kubelet_cgroup_manager_duration_seconds`
* `kubelet_pod_worker_start_duration_seconds`
* `kubelet_pleg_relist_duration_seconds`
* `kubelet_pleg_relist_interval_seconds`
* `kubelet_eviction_stats_age_seconds`
* `kubelet_runtime_operations_total`
* `kubelet_runtime_operations_duration_seconds`
* `kubelet_runtime_operations_errors_total`
* `kubelet_device_plugin_registration_total`
* `kubelet_device_plugin_alloc_duration_seconds`

danielqsj · 2019-01-02T03:11:02Z

/assign @brancz

brancz · 2019-01-08T09:27:47Z

Could you add the deprecation hint in the description here as well? Thanks!

danielqsj · 2019-01-08T10:01:41Z

@brancz Added Deprecated in metrics description. PTAL

brancz · 2019-01-08T13:43:13Z

/retest

brancz · 2019-01-08T14:52:09Z

/retest

brancz · 2019-01-08T17:13:02Z

/retest

brancz · 2019-01-08T20:00:37Z

/retest

danielqsj · 2019-01-10T08:52:03Z

/retest

brancz · 2019-01-10T12:30:32Z

/lgtm
/assign @derekwaynecarr

danielqsj · 2019-01-17T09:20:39Z

@derekwaynecarr can you help review this? Thanks

danielqsj · 2019-01-28T01:40:32Z

@derekwaynecarr @vishh ping

danielqsj · 2019-02-02T01:49:05Z

@thockin @derekwaynecarr @vishh can you help review this? Thanks

danielqsj · 2019-02-07T13:02:43Z

rebased.
@thockin @derekwaynecarr @vishh : this is the last task of #72333 . Can you help review it then I can close the tracking issue?

brancz · 2019-02-07T15:13:39Z

/retest

brancz · 2019-02-07T16:51:18Z

/retest

ehashman · 2019-02-07T17:50:50Z

I believe this fixes #66791.

danielqsj · 2019-02-18T06:11:34Z

PR rebased.

@tallclair : could you help review it ? thanks

brancz · 2019-02-18T09:23:46Z

pkg/kubelet/metrics/metrics.go

+	KubeletSubsystem                     = "kubelet"
+	NodeNameKey                          = "node_name"
+	NodeLabelKey                         = "node"
+	PodWorkerLatencyKey                  = "pod_worker_latency_seconds"


I just noticed this on another PR, these should all be duration instead of latency

Does this looks good ? 79a3eb8

So we should do this change to all other metrics ?

Yes we should.

sure, I will change them.

brancz · 2019-02-18T09:43:37Z

Thanks for the quick update!

/lgtm

brancz · 2019-02-18T10:05:10Z

/cc @thockin @smarterclayton @derekwaynecarr

smarterclayton · 2019-02-19T01:38:45Z

/approve

k8s-ci-robot · 2019-02-19T01:44:43Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: danielqsj, smarterclayton

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~pkg/kubelet/OWNERS~~ [smarterclayton]
~~test/OWNERS~~ [smarterclayton]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

fejta-bot · 2019-02-19T04:00:09Z

/retest
This bot automatically retries jobs that failed/flaked on approved PRs (send feedback to fejta).

Review the full test history for this PR.

Silence the bot with an /lgtm cancel or /hold comment for consistent failures.

mikedanese · 2019-09-06T20:43:50Z

I just noticed this on another PR, these should all be duration instead of latency

@brancz this is probably worth adding to https://github.com/kubernetes/community/blob/81ec4af0ed02b4c5c0917a16563250b2f45250c2/contributors/devel/sig-instrumentation/instrumentation.md#naming

brancz · 2019-09-09T11:25:37Z

@mikedanese You're totally right. There is this already, but we don't link to it yet. Let me fix that.

k8s-ci-robot assigned brancz Jan 2, 2019

k8s-ci-robot added area/kubelet sig/node Categorizes an issue or PR as relevant to SIG Node. sig/testing Categorizes an issue or PR as relevant to SIG Testing. labels Jan 2, 2019

k8s-ci-robot requested review from dims and enj January 2, 2019 03:11

danielqsj mentioned this pull request Jan 2, 2019

Change existing metrics to conform metrics guidelines #72333

Closed

9 tasks

k8s-ci-robot assigned derekwaynecarr Jan 10, 2019

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Jan 10, 2019

danielqsj force-pushed the kl branch from 469afa5 to 5f110b4 Compare February 7, 2019 12:58

k8s-ci-robot removed the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Feb 7, 2019

k8s-ci-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Feb 17, 2019

danielqsj added 4 commits February 18, 2019 14:01

Change kubelet metrics to conform guideline

9fd99a4

Move kubelet metrics to histogram metrics

0e9515c

Mark deprecated in related kubelet metrics

4fa0ee7

add default buckets for histogram metrics

0bfe4c2

danielqsj force-pushed the kl branch from 5f110b4 to 0bfe4c2 Compare February 18, 2019 06:08

k8s-ci-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Feb 18, 2019

brancz reviewed Feb 18, 2019

View reviewed changes

rename latency to duration in metrics

79a3eb8

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Feb 18, 2019

k8s-ci-robot requested review from derekwaynecarr, smarterclayton and thockin February 18, 2019 10:05

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Feb 19, 2019

k8s-ci-robot merged commit 0b01b9c into kubernetes:master Feb 19, 2019

danielqsj mentioned this pull request Feb 22, 2019

convert latency/latencies in metrics name to duration #74418

Merged

ehashman mentioned this pull request Mar 7, 2019

Add histogram metric for Pod start latency #66791

Closed

brancz mentioned this pull request Aug 7, 2019

Kubernetes Metrics Overhaul kubernetes/enhancements#1206

Closed

brancz mentioned this pull request Sep 9, 2019

contributors/devel/sig-instrumentation: Add further details on naming best practices kubernetes/community#4072

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Change kubelet metrics to conform metrics guidelines #72470

Change kubelet metrics to conform metrics guidelines #72470

danielqsj commented Jan 2, 2019 •

edited

Loading

danielqsj commented Jan 2, 2019

brancz commented Jan 8, 2019

danielqsj commented Jan 8, 2019

brancz commented Jan 8, 2019

brancz commented Jan 8, 2019

brancz commented Jan 8, 2019

brancz commented Jan 8, 2019

danielqsj commented Jan 10, 2019

brancz commented Jan 10, 2019

danielqsj commented Jan 17, 2019

danielqsj commented Jan 28, 2019

danielqsj commented Feb 2, 2019

danielqsj commented Feb 7, 2019

brancz commented Feb 7, 2019

brancz commented Feb 7, 2019

ehashman commented Feb 7, 2019

danielqsj commented Feb 18, 2019

brancz Feb 18, 2019

danielqsj Feb 18, 2019

danielqsj Feb 18, 2019

brancz Feb 18, 2019

danielqsj Feb 18, 2019

brancz commented Feb 18, 2019

brancz commented Feb 18, 2019

smarterclayton commented Feb 19, 2019

k8s-ci-robot commented Feb 19, 2019

fejta-bot commented Feb 19, 2019

mikedanese commented Sep 6, 2019

brancz commented Sep 9, 2019

Change kubelet metrics to conform metrics guidelines #72470

Change kubelet metrics to conform metrics guidelines #72470

Conversation

danielqsj commented Jan 2, 2019 • edited Loading

danielqsj commented Jan 2, 2019

brancz commented Jan 8, 2019

danielqsj commented Jan 8, 2019

brancz commented Jan 8, 2019

brancz commented Jan 8, 2019

brancz commented Jan 8, 2019

brancz commented Jan 8, 2019

danielqsj commented Jan 10, 2019

brancz commented Jan 10, 2019

danielqsj commented Jan 17, 2019

danielqsj commented Jan 28, 2019

danielqsj commented Feb 2, 2019

danielqsj commented Feb 7, 2019

brancz commented Feb 7, 2019

brancz commented Feb 7, 2019

ehashman commented Feb 7, 2019

danielqsj commented Feb 18, 2019

brancz Feb 18, 2019

Choose a reason for hiding this comment

danielqsj Feb 18, 2019

Choose a reason for hiding this comment

danielqsj Feb 18, 2019

Choose a reason for hiding this comment

brancz Feb 18, 2019

Choose a reason for hiding this comment

danielqsj Feb 18, 2019

Choose a reason for hiding this comment

brancz commented Feb 18, 2019

brancz commented Feb 18, 2019

smarterclayton commented Feb 19, 2019

k8s-ci-robot commented Feb 19, 2019

fejta-bot commented Feb 19, 2019

mikedanese commented Sep 6, 2019

brancz commented Sep 9, 2019

danielqsj commented Jan 2, 2019 •

edited

Loading