[receiver/kubeletstats] Start name change for cpu.utilization #25901

TylerHelmuth · 2023-08-18T20:55:51Z

Description:
Starts the name change processor for *.cpu.utilization metrics.

Link to tracking Issue:
Related to #24905
Related to #27885

Testing:

github-actions · 2023-09-05T05:20:45Z

This PR was marked stale due to lack of activity. It will be closed in 14 days.

github-actions · 2023-09-20T05:19:43Z

This PR was marked stale due to lack of activity. It will be closed in 14 days.

github-actions · 2023-10-05T05:21:40Z

This PR was marked stale due to lack of activity. It will be closed in 14 days.

github-actions · 2023-10-20T05:20:37Z

This PR was marked stale due to lack of activity. It will be closed in 14 days.

receiver/kubeletstatsreceiver/documentation.md

povilasv · 2023-10-24T07:56:06Z

Another thing I was thining about, is maybe we can actually compute the CPU utilization somehow?

I.e. instead of deprecating, we actually fix it to correctly report the values?

TylerHelmuth · 2023-10-24T14:53:47Z

@povilasv good questions, can you ask that question in the issue

github-actions · 2023-11-08T05:19:47Z

This PR was marked stale due to lack of activity. It will be closed in 14 days.

dmitryax

It's unclear how the new usage metrics are different from time metrics. Can you please clarify that they are average aggregations over time?

TylerHelmuth · 2024-01-09T22:24:16Z

@dmitryax I'm only as knowledgable as the comments on the struct:

// CPUStats contains data about CPU usage.
type CPUStats struct {
	// The time at which these stats were updated.
	Time metav1.Time `json:"time"`
	// Total CPU usage (sum of all cores) averaged over the sample window.
	// The "core" unit can be interpreted as CPU core-nanoseconds per second.
	// +optional
	UsageNanoCores *uint64 `json:"usageNanoCores,omitempty"`
	// Cumulative CPU usage (sum of all cores) since object creation.
	// +optional
	UsageCoreNanoSeconds *uint64 `json:"usageCoreNanoSeconds,omitempty"`
}

We use Time to set the *.cpu.time and we use UsageNanoCores to set *.cpu.utilization (and now *.cpu.usage). @jinja2 @povilasv do you know?

dmitryax · 2024-01-09T23:20:16Z

I was just asking to improve the description of the "usage" metrics to make it easier to differentiate it from the "time" metrics.

Usage can be described as "Total CPU usage (sum of all cores per second) averaged over the sample window".

Time can be "Total cumulative CPU time (sum of all cores) spent by the container/pod/node since its creation".

@TylerHelmuth @povilasv @jinja2 let me know if it's clear enough

dmitryax · 2024-01-09T23:21:46Z

receiver/kubeletstatsreceiver/metadata.yaml

+  k8s.node.cpu.usage:
+    enabled: false
+    description: "Node CPU usage"
+    unit: s


Not sure unit is s. Maybe {cpu}?

povilasv · 2024-01-10T13:50:14Z

Based on my prior research it looks like CpuNanoCores it returns nanoseconds. See #27885 (comment)

And I think that's why we are dividing by 1_000_000_000 to get seconds:

	value := float64(*s.UsageNanoCores) / 1_000_000_000
	cpuMetrics.Utilization(mb, currentTime, value)
	cpuMetrics.Usage(mb, currentTime, value)

I'll recheck and dig around in that code a bit more to try to figure it out what it's actually counting, as those docs are pretty confusing

…changing-cpu-utilization

dmitryax · 2024-01-11T00:27:26Z

Based on my prior research it looks like CpuNanoCores it returns nanoseconds

It is, but it's an average nanoseconds*cpu/second over a reporting interval. Dividing it by 1_000_000_000 ns should give us {cpu}

…changing-cpu-utilization

dmitryax

LGTM. @povilasv PTAL

povilasv

LGTM

…elemetry#25901) **Description:** Starts the name change processor for `*.cpu.utilization` metrics. **Link to tracking Issue:** Related to open-telemetry#24905 Related to open-telemetry#27885

The following metrics emitted by kubeletstats receiver were deprecated: - k8s.node.cpu.utilization - k8s.pod.cpu.utilization - container.cpu.utilization See open-telemetry/opentelemetry-collector-contrib#25901 for more details. Those metrics are already excluded by default in the signalfx exporter. So if we don't do anything, user will see the following confusing warnings: ``` 2024-02-06T19:21:44.773Z warn metadata/generated_metrics.go:2894 [WARNING] `container.cpu.utilization` should not be enabled: WARNING: This metric will be disabled in a future release. Use metric container.cpu.usage instead. {"kind": "receiver", "name": "kubeletstats", "data_type": "metrics"} 2024-02-06T19:21:44.773Z warn metadata/generated_metrics.go:2897 [WARNING] `k8s.node.cpu.utilization` should not be enabled: WARNING: This metric will be disabled in a future release. Use metric k8s.node.cpu.usage instead. {"kind": "receiver", "name": "kubeletstats", "data_type": "metrics"} 2024-02-06T19:21:44.773Z warn metadata/generated_metrics.go:2900 [WARNING] `k8s.pod.cpu.utilization` should not be enabled: This metric will be disabled in a future release. Use metric k8s.pod.cpu.usage instead. {"kind": "receiver", "name": "kubeletstats", "data_type": "metrics"} 2024-02-06T19:21:44.774Z warn metadata/generated_metrics.go:2894 [WARNING] `container.cpu.utilization` should not be enabled: WARNING: This metric will be disabled in a future release. Use metric container.cpu.usage instead. {"kind": "receiver", "name": "kubeletstats", "data_type": "metrics"} 2024-02-06T19:21:44.774Z warn metadata/generated_metrics.go:2897 [WARNING] `k8s.node.cpu.utilization` should not be enabled: WARNING: This metric will be disabled in a future release. Use metric k8s.node.cpu.usage instead. {"kind": "receiver", "name": "kubeletstats", "data_type": "metrics"} 2024-02-06T19:21:44.774Z warn metadata/generated_metrics.go:2900 [WARNING] `k8s.pod.cpu.utilization` should not be enabled: This metric will be disabled in a future release. Use metric k8s.pod.cpu.usage instead. {"kind": "receiver", "name": "kubeletstats", "data_type": "metrics"} 2024-02-06T19:21:44.774Z warn metadata/generated_metrics.go:2894 [WARNING] `container.cpu.utilization` should not be enabled: WARNING: This metric will be disabled in a future release. Use metric container.cpu.usage instead. {"kind": "receiver", "name": "kubeletstats", "data_type": "metrics"} 2024-02-06T19:21:44.774Z warn metadata/generated_metrics.go:2897 [WARNING] `k8s.node.cpu.utilization` should not be enabled: WARNING: This metric will be disabled in a future release. Use metric k8s.node.cpu.usage instead. {"kind": "receiver", "name": "kubeletstats", "data_type": "metrics"} 2024-02-06T19:21:44.774Z warn metadata/generated_metrics.go:2900 [WARNING] `k8s.pod.cpu.utilization` should not be enabled: This metric will be disabled in a future release. Use metric k8s.pod.cpu.usage instead. {"kind": "receiver", "name": "kubeletstats", "data_type": "metrics"} 2024-02-06T19:21:44.774Z warn metadata/generated_metrics.go:2894 [WARNING] `container.cpu.utilization` should not be enabled: WARNING: This metric will be disabled in a future release. Use metric container.cpu.usage instead. {"kind": "receiver", "name": "kubeletstats", "data_type": "metrics"} 2024-02-06T19:21:44.774Z warn metadata/generated_metrics.go:2897 [WARNING] `k8s.node.cpu.utilization` should not be enabled: WARNING: This metric will be disabled in a future release. Use metric k8s.node.cpu.usage instead. {"kind": "receiver", "name": "kubeletstats", "data_type": "metrics"} 2024-02-06T19:21:44.774Z warn metadata/generated_metrics.go:2900 [WARNING] `k8s.pod.cpu.utilization` should not be enabled: This metric will be disabled in a future release. Use metric k8s.pod.cpu.usage instead. {"kind": "receiver", "name": "kubeletstats", "data_type": "metrics"} ``` Potentially we couldn't just explicitly disable them in the default k8s configuration on the receiver side. The problem is that the exclusion on the signalfx exporter side potentially can be disabled by users. For those users, we actually want to show the warnings. So we need to add another converter until the metric is disabled on the kubelet receiver side by default. The converter looks at the signalfx exporter in user's config, and if metrics are not explicitly enabled on the exporter side, it disables them on the kubelet receiver side to supress the warnings. The converted required to copy `dpfilters` package from signal exporter. This will be removed along with the converter in a few releases.

) The following metrics emitted by kubeletstats receiver were deprecated: - k8s.node.cpu.utilization - k8s.pod.cpu.utilization - container.cpu.utilization See open-telemetry/opentelemetry-collector-contrib#25901 for more details. Those metrics are already excluded by default in the signalfx exporter. So if we don't do anything, user will see the following confusing warnings: ``` 2024-02-06T19:21:44.773Z warn metadata/generated_metrics.go:2894 [WARNING] `container.cpu.utilization` should not be enabled: WARNING: This metric will be disabled in a future release. Use metric container.cpu.usage instead. {"kind": "receiver", "name": "kubeletstats", "data_type": "metrics"} 2024-02-06T19:21:44.773Z warn metadata/generated_metrics.go:2897 [WARNING] `k8s.node.cpu.utilization` should not be enabled: WARNING: This metric will be disabled in a future release. Use metric k8s.node.cpu.usage instead. {"kind": "receiver", "name": "kubeletstats", "data_type": "metrics"} 2024-02-06T19:21:44.773Z warn metadata/generated_metrics.go:2900 [WARNING] `k8s.pod.cpu.utilization` should not be enabled: This metric will be disabled in a future release. Use metric k8s.pod.cpu.usage instead. {"kind": "receiver", "name": "kubeletstats", "data_type": "metrics"} 2024-02-06T19:21:44.774Z warn metadata/generated_metrics.go:2894 [WARNING] `container.cpu.utilization` should not be enabled: WARNING: This metric will be disabled in a future release. Use metric container.cpu.usage instead. {"kind": "receiver", "name": "kubeletstats", "data_type": "metrics"} 2024-02-06T19:21:44.774Z warn metadata/generated_metrics.go:2897 [WARNING] `k8s.node.cpu.utilization` should not be enabled: WARNING: This metric will be disabled in a future release. Use metric k8s.node.cpu.usage instead. {"kind": "receiver", "name": "kubeletstats", "data_type": "metrics"} 2024-02-06T19:21:44.774Z warn metadata/generated_metrics.go:2900 [WARNING] `k8s.pod.cpu.utilization` should not be enabled: This metric will be disabled in a future release. Use metric k8s.pod.cpu.usage instead. {"kind": "receiver", "name": "kubeletstats", "data_type": "metrics"} 2024-02-06T19:21:44.774Z warn metadata/generated_metrics.go:2894 [WARNING] `container.cpu.utilization` should not be enabled: WARNING: This metric will be disabled in a future release. Use metric container.cpu.usage instead. {"kind": "receiver", "name": "kubeletstats", "data_type": "metrics"} 2024-02-06T19:21:44.774Z warn metadata/generated_metrics.go:2897 [WARNING] `k8s.node.cpu.utilization` should not be enabled: WARNING: This metric will be disabled in a future release. Use metric k8s.node.cpu.usage instead. {"kind": "receiver", "name": "kubeletstats", "data_type": "metrics"} 2024-02-06T19:21:44.774Z warn metadata/generated_metrics.go:2900 [WARNING] `k8s.pod.cpu.utilization` should not be enabled: This metric will be disabled in a future release. Use metric k8s.pod.cpu.usage instead. {"kind": "receiver", "name": "kubeletstats", "data_type": "metrics"} 2024-02-06T19:21:44.774Z warn metadata/generated_metrics.go:2894 [WARNING] `container.cpu.utilization` should not be enabled: WARNING: This metric will be disabled in a future release. Use metric container.cpu.usage instead. {"kind": "receiver", "name": "kubeletstats", "data_type": "metrics"} 2024-02-06T19:21:44.774Z warn metadata/generated_metrics.go:2897 [WARNING] `k8s.node.cpu.utilization` should not be enabled: WARNING: This metric will be disabled in a future release. Use metric k8s.node.cpu.usage instead. {"kind": "receiver", "name": "kubeletstats", "data_type": "metrics"} 2024-02-06T19:21:44.774Z warn metadata/generated_metrics.go:2900 [WARNING] `k8s.pod.cpu.utilization` should not be enabled: This metric will be disabled in a future release. Use metric k8s.pod.cpu.usage instead. {"kind": "receiver", "name": "kubeletstats", "data_type": "metrics"} ``` Potentially we couldn't just explicitly disable them in the default k8s configuration on the receiver side. The problem is that the exclusion on the signalfx exporter side potentially can be disabled by users. For those users, we actually want to show the warnings. So we need to add another converter until the metric is disabled on the kubelet receiver side by default. The converter looks at the signalfx exporter in user's config, and if metrics are not explicitly enabled on the exporter side, it disables them on the kubelet receiver side to supress the warnings. The converted required to copy `dpfilters` package from signal exporter. This will be removed along with the converter in a few releases.

TylerHelmuth requested review from dmitryax and a team as code owners August 18, 2023 20:55

github-actions bot assigned codeboten Aug 18, 2023

github-actions bot added the receiver/kubeletstats label Aug 18, 2023

TylerHelmuth force-pushed the kubeletstats-start-changing-cpu-utilization branch from 6efff1c to 46d276d Compare August 18, 2023 20:57

github-actions bot added the Stale label Sep 5, 2023

TylerHelmuth removed the Stale label Sep 5, 2023

github-actions bot added the Stale label Sep 20, 2023

TylerHelmuth removed the Stale label Sep 20, 2023

github-actions bot added the Stale label Oct 5, 2023

TylerHelmuth removed the Stale label Oct 5, 2023

github-actions bot added the Stale label Oct 20, 2023

TylerHelmuth removed the Stale label Oct 20, 2023

TylerHelmuth added 2 commits October 20, 2023 08:11

Start name change for cpu.utilization

7c0296c

changelog

2dfdd5f

TylerHelmuth force-pushed the kubeletstats-start-changing-cpu-utilization branch from c00544d to 2dfdd5f Compare October 20, 2023 14:12

povilasv reviewed Oct 23, 2023

View reviewed changes

receiver/kubeletstatsreceiver/documentation.md Outdated Show resolved Hide resolved

Update units

f9306b8

povilasv reviewed Oct 24, 2023

View reviewed changes

receiver/kubeletstatsreceiver/documentation.md Outdated Show resolved Hide resolved

Update units

6c323c1

povilasv approved these changes Oct 24, 2023

View reviewed changes

github-actions bot added the Stale label Nov 8, 2023

TylerHelmuth removed the Stale label Nov 9, 2023

dmitryax removed the Stale label Jan 2, 2024

dmitryax mentioned this pull request Jan 9, 2024

Add container metric fields (from ECS) open-telemetry/semantic-conventions#282

Merged

dmitryax reviewed Jan 9, 2024

View reviewed changes

TylerHelmuth added 2 commits January 10, 2024 08:37

Update descriptions

c018ee0

Merge remote-tracking branch 'upstream/main' into kubeletstats-start-…

b22c8bf

…changing-cpu-utilization

TylerHelmuth added 4 commits January 11, 2024 09:15

Update unit

10f0f50

Update test expectation

d7ad181

Merge remote-tracking branch 'upstream/main' into kubeletstats-start-…

85431f0

…changing-cpu-utilization

Update descriptions

f9dd035

dmitryax approved these changes Jan 11, 2024

View reviewed changes

povilasv approved these changes Jan 12, 2024

View reviewed changes

dmitryax merged commit 74294cb into open-telemetry:main Jan 12, 2024
85 checks passed

github-actions bot added this to the next release milestone Jan 12, 2024

TylerHelmuth deleted the kubeletstats-start-changing-cpu-utilization branch January 12, 2024 17:09

dmitryax mentioned this pull request Feb 7, 2024

[chore] Adopt to the kubelet cpu utilization metrics deprecation signalfx/splunk-otel-collector#4269

Merged

a-thaler mentioned this pull request Feb 29, 2024

Fix and review cpu utilization metrics kyma-project/telemetry-manager#838

Closed

ChrsMark mentioned this pull request Mar 27, 2024

[receiver/kubeletstat] Review cpu.utilization naming #27885

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[receiver/kubeletstats] Start name change for cpu.utilization #25901

[receiver/kubeletstats] Start name change for cpu.utilization #25901

TylerHelmuth commented Aug 18, 2023 •

edited

github-actions bot commented Sep 5, 2023

github-actions bot commented Sep 20, 2023

github-actions bot commented Oct 5, 2023

github-actions bot commented Oct 20, 2023

povilasv commented Oct 24, 2023

TylerHelmuth commented Oct 24, 2023

github-actions bot commented Nov 8, 2023

dmitryax left a comment

TylerHelmuth commented Jan 9, 2024 •

edited

dmitryax commented Jan 9, 2024 •

edited

dmitryax Jan 9, 2024

povilasv commented Jan 10, 2024 •

edited

dmitryax commented Jan 11, 2024 •

edited

dmitryax left a comment

povilasv left a comment

[receiver/kubeletstats] Start name change for cpu.utilization #25901

[receiver/kubeletstats] Start name change for cpu.utilization #25901

Conversation

TylerHelmuth commented Aug 18, 2023 • edited

github-actions bot commented Sep 5, 2023

github-actions bot commented Sep 20, 2023

github-actions bot commented Oct 5, 2023

github-actions bot commented Oct 20, 2023

povilasv commented Oct 24, 2023

TylerHelmuth commented Oct 24, 2023

github-actions bot commented Nov 8, 2023

dmitryax left a comment

Choose a reason for hiding this comment

TylerHelmuth commented Jan 9, 2024 • edited

dmitryax commented Jan 9, 2024 • edited

dmitryax Jan 9, 2024

Choose a reason for hiding this comment

povilasv commented Jan 10, 2024 • edited

dmitryax commented Jan 11, 2024 • edited

dmitryax left a comment

Choose a reason for hiding this comment

povilasv left a comment

Choose a reason for hiding this comment

TylerHelmuth commented Aug 18, 2023 •

edited

TylerHelmuth commented Jan 9, 2024 •

edited

dmitryax commented Jan 9, 2024 •

edited

povilasv commented Jan 10, 2024 •

edited

dmitryax commented Jan 11, 2024 •

edited