podgc metrics should count all pod deletion behaviors #118480

carlory · 2023-06-06T00:54:07Z

What type of PR is this?

/kind feature

What this PR does / why we need it:

Literally, force_delete_pods_total and force_delete_pod_errors_total metrics should count all the behaviors of the gc controller that forcefully delete pods.

The reason label indicates the reason why the pod was forcibly deleted by GC

Special notes for your reviewer:

Does this PR introduce a user-facing change?

`force_delete_pods_total ` and `force_delete_pod_errors_total ` metrics count all pod deletion behaviors.

Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

- [KEP]: https://github.com/kubernetes/enhancements/issues/2268

carlory · 2023-06-06T01:03:37Z

/cc @xing-yang

xing-yang · 2023-06-06T01:18:30Z

pkg/controller/podgc/gc_controller_test.go

 	if err != nil {
 		t.Errorf("Error getting actualDeletingPodsTotal")
 	}
 	if actualDeletingPodsTotal != float64(inputDeletingPodsTotal) {
 		t.Errorf("Expected desiredDeletingPodsTotal to be %d, got %v", inputDeletingPodsTotal, actualDeletingPodsTotal)
 	}

-	actualDeletingPodsErrorTotal, err := metricstestutil.GetCounterMetricValue(deletingPodsErrorTotal.WithLabelValues())
+	actualDeletingPodsErrorTotal, err := metricstestutil.GetCounterMetricValue(deletingPodsErrorTotal.WithLabelValues("out-of-service"))
 	if err != nil {


Can you add unit tests for all the other labels?

carlory · 2023-06-12T01:47:11Z

/cc @xing-yang @jpbetz @gnufied

carlory · 2023-06-13T01:47:43Z

/triage accepted
/assign @janetkuo

carlory · 2023-06-14T03:44:16Z

/check-cla

carlory · 2023-06-14T03:44:40Z

/easycla

xing-yang · 2023-06-14T12:45:01Z

/assign @gnufied

gnufied · 2023-06-23T21:33:28Z

pkg/controller/podgc/gc_controller.go

@@ -179,11 +179,11 @@ func (gcc *PodGCController) gcTerminating(ctx context.Context, pods []*v1.Pod) {
 		wait.Add(1)
 		go func(pod *v1.Pod) {
 			defer wait.Done()
-			deletingPodsTotal.WithLabelValues().Inc()
+			deletingPodsTotal.WithLabelValues("out-of-service").Inc()


Can we move these dimensions such as out-of-service, terminated etc in well defined constants, so as it gets easier to reuse them without potentially having typos in the code?

@gnufied updated.

gnufied · 2023-06-23T21:45:47Z

pkg/controller/podgc/gc_controller.go

@@ -213,9 +213,11 @@ func (gcc *PodGCController) gcTerminated(ctx context.Context, pods []*v1.Pod) {
 		wait.Add(1)
 		go func(pod *v1.Pod) {
 			defer wait.Done()
+			deletingPodsTotal.WithLabelValues("terminated").Inc()


I am thinking, if we should emit this metric before or after the completion of delete operation. As it stands, we emit when we start deleting.

Also, I think we should add namespace to these metrics, so as if we force deleted pods from a particular namespace, a k8s admin can investigate why that happened. @msau42 what do you think?

Waiting for a feedback from @msau42

namespace would be good to add. I have a slight preference to add the metric after completion. Do we care about deletion errors?

@xing-yang what do you think？

carlory · 2023-07-03T02:33:40Z

@xing-yang @msau42 @gnufied the namespace label is added.

carlory · 2023-07-03T03:04:18Z

/test pull-kubernetes-node-e2e-containerd

xing-yang · 2023-07-05T17:55:06Z

pkg/controller/podgc/gc_controller_test.go

 	t.Helper()

-	actualDeletingPodsTotal, err := metricstestutil.GetCounterMetricValue(deletingPodsTotal.WithLabelValues())
+	actualDeletingPodsTotal, err := metricstestutil.GetCounterMetricValue(metrics.DeletingPodsTotal.WithLabelValues("", reason))


What is the namespace for these pods?

@xing-yang It's fixed in https://github.com/kubernetes/kubernetes/compare/04ac55e68117dff43adaaf3be47e39578e44dadc..322da7c1aae34496fda5ce5dd96966706d6513b4

gnufied · 2023-07-06T19:10:10Z

/lgtm

k8s-ci-robot · 2023-07-06T19:10:17Z

LGTM label has been added.

Git tree hash: b87e82ce5aab1c88b98e1351fc76a6dd201c19ed

carlory · 2023-07-07T03:05:17Z

/assign @janetkuo

xing-yang · 2023-07-10T02:59:07Z

/lgtm

xing-yang · 2023-07-10T02:59:18Z

/assign @andrewsykim

xing-yang · 2023-07-12T14:43:51Z

/assign @soltysh

soltysh

/approve

k8s-ci-robot · 2023-07-13T12:01:10Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: carlory, soltysh

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~pkg/controller/OWNERS~~ [soltysh]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

soltysh · 2023-07-13T12:01:21Z

/priority important-longterm

k8s-ci-robot requested review from cheftako and deads2k June 6, 2023 00:59

k8s-ci-robot added sig/apps Categorizes an issue or PR as relevant to SIG Apps. and removed do-not-merge/needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. labels Jun 6, 2023

carlory mentioned this pull request Jun 6, 2023

KEP-2268: move non-graceful node shutdown to GA kubernetes/enhancements#4029

Merged

carlory changed the title ~~gc metrics should count all pod deletion behaviors~~ podgc metrics should count all pod deletion behaviors Jun 6, 2023

k8s-ci-robot requested a review from xing-yang June 6, 2023 01:03

xing-yang reviewed Jun 6, 2023

View reviewed changes

carlory force-pushed the gc_metrics branch from 9159943 to 848174c Compare June 6, 2023 02:12

k8s-ci-robot added size/M Denotes a PR that changes 30-99 lines, ignoring generated files. and removed size/S Denotes a PR that changes 10-29 lines, ignoring generated files. labels Jun 6, 2023

k8s-ci-robot requested review from gnufied, jpbetz and xing-yang June 12, 2023 01:47

k8s-ci-robot assigned janetkuo Jun 13, 2023

k8s-ci-robot added triage/accepted Indicates an issue or PR is ready to be actively worked on. and removed needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. labels Jun 13, 2023

gnufied reviewed Jun 23, 2023

View reviewed changes

carlory force-pushed the gc_metrics branch 3 times, most recently from b09a043 to 04ac55e Compare July 3, 2023 02:32

xing-yang reviewed Jul 5, 2023

View reviewed changes

fix gc metrics

322da7c

carlory force-pushed the gc_metrics branch from 04ac55e to 322da7c Compare July 5, 2023 18:06

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Jul 6, 2023

k8s-ci-robot assigned xing-yang Jul 10, 2023

k8s-ci-robot assigned andrewsykim Jul 10, 2023

xing-yang mentioned this pull request Jul 11, 2023

Non-graceful node shutdown kubernetes/enhancements#2268

Closed

k8s-ci-robot assigned soltysh Jul 12, 2023

soltysh approved these changes Jul 13, 2023

View reviewed changes

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jul 13, 2023

k8s-ci-robot added priority/important-longterm Important over the long term, but may not be staffed and/or may need multiple releases to complete. and removed needs-priority Indicates a PR lacks a `priority/foo` label and requires one. labels Jul 13, 2023

k8s-ci-robot merged commit cd92159 into kubernetes:master Jul 13, 2023
11 of 12 checks passed

k8s-ci-robot added this to the v1.28 milestone Jul 13, 2023

YuikoTakada mentioned this pull request Aug 1, 2023

Add blog for non-graceful node shutdown kubernetes/website#41914

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

podgc metrics should count all pod deletion behaviors #118480

podgc metrics should count all pod deletion behaviors #118480

carlory commented Jun 6, 2023

carlory commented Jun 6, 2023

xing-yang Jun 6, 2023

carlory Jun 6, 2023

carlory commented Jun 12, 2023

carlory commented Jun 13, 2023

carlory commented Jun 14, 2023

carlory commented Jun 14, 2023

xing-yang commented Jun 14, 2023

gnufied Jun 23, 2023

carlory Jun 29, 2023

gnufied Jun 23, 2023

carlory Jun 29, 2023

msau42 Jun 29, 2023

carlory Jun 30, 2023

carlory commented Jul 3, 2023

carlory commented Jul 3, 2023

xing-yang Jul 5, 2023

carlory Jul 5, 2023

gnufied commented Jul 6, 2023

k8s-ci-robot commented Jul 6, 2023

carlory commented Jul 7, 2023

xing-yang commented Jul 10, 2023

xing-yang commented Jul 10, 2023

xing-yang commented Jul 12, 2023

soltysh left a comment

k8s-ci-robot commented Jul 13, 2023

soltysh commented Jul 13, 2023

podgc metrics should count all pod deletion behaviors #118480

podgc metrics should count all pod deletion behaviors #118480

Conversation

carlory commented Jun 6, 2023

What type of PR is this?

What this PR does / why we need it:

Special notes for your reviewer:

Does this PR introduce a user-facing change?

Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

carlory commented Jun 6, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

carlory commented Jun 12, 2023

carlory commented Jun 13, 2023

carlory commented Jun 14, 2023

carlory commented Jun 14, 2023

xing-yang commented Jun 14, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

carlory commented Jul 3, 2023

carlory commented Jul 3, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gnufied commented Jul 6, 2023

k8s-ci-robot commented Jul 6, 2023

carlory commented Jul 7, 2023

xing-yang commented Jul 10, 2023

xing-yang commented Jul 10, 2023

xing-yang commented Jul 12, 2023

soltysh left a comment

Choose a reason for hiding this comment

k8s-ci-robot commented Jul 13, 2023

soltysh commented Jul 13, 2023