Fix ReplicaSet deletion rank for multiple colocation nodes #102949

FillZpp · 2021-06-17T11:19:05Z

Signed-off-by: FillZpp FillZpp.pub@gmail.com

What type of PR is this?

/kind bug

What this PR does / why we need it:

Fix ReplicaSet deletion rank for multiple colocation nodes.

Which issue(s) this PR fixes:

Fixes #102948

Special notes for your reviewer:

Now for a ReplicaSet or Deployment which has five Pods on two Nodes:

node01: pod-a, pod-b, pod-c
node02: pod-d, pod-e

When replicas changed from 5 to 2, the deletion rank that controller will get for Pods:

node01:
- pod-a: 3
- pod-b: 3
- pod-c: 3
node02:
- pod-d: 2
- pod-e: 2

So ReplicaSet will choose all three Pods on node01 to delete.

This PR fixes the rank to be:

node01:
- pod-a: 3
- pod-b: 2
- pod-c: 1
node02:
- pod-d: 2
- pod-e: 1

So that controller will delete Pods more evenly.

Does this PR introduce a user-facing change?

No user-facing changed.

/sig apps

k8s-ci-robot · 2021-06-17T11:19:06Z

@FillZpp: Adding the "do-not-merge/release-note-label-needed" label because no release-note block was detected, please follow our release note process to remove it.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

k8s-ci-robot · 2021-06-17T11:19:12Z

@FillZpp: This issue is currently awaiting triage.

If a SIG or subproject determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

k8s-ci-robot · 2021-06-17T11:19:13Z

Hi @FillZpp. Thanks for your PR.

I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

FillZpp · 2021-06-17T11:22:54Z

/cc @janetkuo @kow3ns @soltysh

gjkim42 · 2021-06-17T13:23:32Z

/ok-to-test

lgtm

Hope folks at sig-apps can approve this.

https://kubernetes.io/docs/concepts/workloads/controllers/replicaset/#scaling-a-replicaset

alculquicondor · 2021-06-17T14:30:10Z

pkg/controller/replicaset/replica_set.go

@@ -819,7 +819,9 @@ func getPodsRankedByRelatedPodsOnSameNode(podsToRank, relatedPods []*v1.Pod) con
 	}
 	ranks := make([]int, len(podsToRank))
 	for i, pod := range podsToRank {
-		ranks[i] = podsOnNode[pod.Spec.NodeName]
+		if ranks[i] = podsOnNode[pod.Spec.NodeName]; ranks[i] > 0 {
+			podsOnNode[pod.Spec.NodeName]--


This is removing the usefulness of other criteria, like the time the pods have been running.
Can you think of an alternative to not lose that?

cc @damemi who has been looking into this.

Yeah, it would probably be better to try to tie this into the sort function that is already called on these podsWithRanks, as done in getPodsToDelete(). See

kubernetes/pkg/controller/controller_utils.go

Lines 820 to 894 in d8aad79

func (s ActivePodsWithRanks) Less(i, j int) bool {

// 1. Unassigned < assigned

// If only one of the pods is unassigned, the unassigned one is smaller

if s.Pods[i].Spec.NodeName != s.Pods[j].Spec.NodeName && (len(s.Pods[i].Spec.NodeName) == 0 || len(s.Pods[j].Spec.NodeName) == 0) {

return len(s.Pods[i].Spec.NodeName) == 0

}

// 2. PodPending < PodUnknown < PodRunning

if podPhaseToOrdinal[s.Pods[i].Status.Phase] != podPhaseToOrdinal[s.Pods[j].Status.Phase] {

return podPhaseToOrdinal[s.Pods[i].Status.Phase] < podPhaseToOrdinal[s.Pods[j].Status.Phase]

}

// 3. Not ready < ready

// If only one of the pods is not ready, the not ready one is smaller

if podutil.IsPodReady(s.Pods[i]) != podutil.IsPodReady(s.Pods[j]) {

return !podutil.IsPodReady(s.Pods[i])

}

// 4. higher pod-deletion-cost < lower pod-deletion cost

if utilfeature.DefaultFeatureGate.Enabled(features.PodDeletionCost) {

pi, _ := helper.GetDeletionCostFromPodAnnotations(s.Pods[i].Annotations)

pj, _ := helper.GetDeletionCostFromPodAnnotations(s.Pods[j].Annotations)

if pi != pj {

return pi < pj

}

}

// 5. Doubled up < not doubled up

// If one of the two pods is on the same node as one or more additional

// ready pods that belong to the same replicaset, whichever pod has more

// colocated ready pods is less

if s.Rank[i] != s.Rank[j] {

return s.Rank[i] > s.Rank[j]

}

// TODO: take availability into account when we push minReadySeconds information from deployment into pods,

// see https://github.com/kubernetes/kubernetes/issues/22065

// 6. Been ready for empty time < less time < more time

// If both pods are ready, the latest ready one is smaller

if podutil.IsPodReady(s.Pods[i]) && podutil.IsPodReady(s.Pods[j]) {

readyTime1 := podReadyTime(s.Pods[i])

readyTime2 := podReadyTime(s.Pods[j])

if !readyTime1.Equal(readyTime2) {

if !utilfeature.DefaultFeatureGate.Enabled(features.LogarithmicScaleDown) {

return afterOrZero(readyTime1, readyTime2)

} else {

if s.Now.IsZero() || readyTime1.IsZero() || readyTime2.IsZero() {

return afterOrZero(readyTime1, readyTime2)

}

rankDiff := logarithmicRankDiff(*readyTime1, *readyTime2, s.Now)

if rankDiff == 0 {

return s.Pods[i].UID < s.Pods[j].UID

}

return rankDiff < 0

}

}

}

// 7. Pods with containers with higher restart counts < lower restart counts

if maxContainerRestarts(s.Pods[i]) != maxContainerRestarts(s.Pods[j]) {

return maxContainerRestarts(s.Pods[i]) > maxContainerRestarts(s.Pods[j])

}

// 8. Empty creation time pods < newer pods < older pods

if !s.Pods[i].CreationTimestamp.Equal(&s.Pods[j].CreationTimestamp) {

if !utilfeature.DefaultFeatureGate.Enabled(features.LogarithmicScaleDown) {

return afterOrZero(&s.Pods[i].CreationTimestamp, &s.Pods[j].CreationTimestamp)

} else {

if s.Now.IsZero() || s.Pods[i].CreationTimestamp.IsZero() || s.Pods[j].CreationTimestamp.IsZero() {

return afterOrZero(&s.Pods[i].CreationTimestamp, &s.Pods[j].CreationTimestamp)

}

rankDiff := logarithmicRankDiff(s.Pods[i].CreationTimestamp, s.Pods[j].CreationTimestamp, s.Now)

if rankDiff == 0 {

return s.Pods[i].UID < s.Pods[j].UID

}

return rankDiff < 0

}

}

return false

}

One alternative I can think of is to do the sorting without ranks first, and then run the loop to add the ranks and finally do a second stable sort.

Or... we can simply remove the ranks. Now that we have randomized removals https://github.com/kubernetes/enhancements/tree/master/keps/sig-apps/2185-random-pod-select-on-replicaset-downscale, the nodes with higher number of pods are more likely to get pods removed :)

I think these two logic don't conflict. With this PR, pods with the same rank can still sort by randomized.

For example, there are 5 nodes and each of them has 2 pods on it. So, the rank of 5 pods on the different 5 nodes will be 1 and the other 5 pods will be 0. The random algorithm will sort the former 5 pods by randomized and the latter 5 pods by randomized.

@alculquicondor @damemi What do you think?

But there is no comparison between a-0 and a-1. You are directly assigning a preference without checking the timestamp.

You are directly assigning a preference without checking the timestamp.

Yes, that might need to sort the pods on same node by timestamp and then give them the right ranks?

Correct. I think it's only possible through two consecutive sort operations, as I proposed above. However, I think if we get rid of the rank entirely, we get the same result, probabilistically.

FillZpp · 2021-07-26T02:40:36Z

WDYT?

@alculquicondor Hey, sorry for this late reply, have gone through a busy time.. 😢

I just updated the code so that pods on the same node will have the right ranks. Would you please take a look again?
BTW, I still wonder if the random itself can get the same result. If we have some test data of this, I'm very happy to get it.

FillZpp · 2021-07-26T07:37:55Z

/retest

alculquicondor · 2021-07-26T13:45:52Z

pkg/controller/replicaset/replica_set.go

 		}
 	}
+	for _, podsOnNode := range podsOnNodes {
+		// sort pods on the same node with active and timestamp
+		sort.Sort(controller.ActivePods(podsOnNode))


This function doesn't sort with logarithmic comparisons

In my opinion, there is no need to sort pods on the same node with logarithmic comparisons, which are designed to solve the sort problem across different topologies (such as multiple failure domains).

The pods on different nodes that have the same rank will be sort by logarithmic in the later ActivePodsWithRanks.

alculquicondor · 2021-07-26T13:48:07Z

BTW, I still wonder if the random itself can get the same result. If we have some test data of this, I'm very happy to get it.

You might be able to build some sort of unit test that plays the scenario you want to solve and compare your PR with what I'm proposing (removing the ranks)

FillZpp · 2021-10-27T08:09:07Z

/remove-lifecycle stale

dims · 2022-01-04T18:19:45Z

/assign @soltysh

/retest

k8s-ci-robot · 2022-01-05T07:43:58Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: FillZpp
To complete the pull request process, please ask for approval from soltysh after the PR has been reviewed.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

pkg/controller/replicaset/OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

FillZpp · 2022-01-06T02:42:47Z

/retest

k8s-triage-robot · 2022-04-06T04:03:53Z

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

After 90d of inactivity, lifecycle/stale is applied
After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

Mark this issue or PR as fresh with /remove-lifecycle stale
Mark this issue or PR as rotten with /lifecycle rotten
Close this issue or PR with /close
Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

k8s-triage-robot · 2022-05-06T15:21:09Z

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

After 90d of inactivity, lifecycle/stale is applied
After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

Mark this issue or PR as fresh with /remove-lifecycle rotten
Close this issue or PR with /close
Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

Signed-off-by: FillZpp <FillZpp.pub@gmail.com>

FillZpp · 2022-05-13T07:35:47Z

/remove-lifecycle rotten

k8s-triage-robot · 2022-08-15T11:52:06Z

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

After 90d of inactivity, lifecycle/stale is applied
After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

Mark this issue or PR as fresh with /remove-lifecycle stale
Mark this issue or PR as rotten with /lifecycle rotten
Close this issue or PR with /close
Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

k8s-triage-robot · 2022-09-14T12:22:34Z

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

After 90d of inactivity, lifecycle/stale is applied
After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

Mark this issue or PR as fresh with /remove-lifecycle rotten
Close this issue or PR with /close
Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

k8s-triage-robot · 2022-10-14T13:20:44Z

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages PRs according to the following rules:

After 90d of inactivity, lifecycle/stale is applied
After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
After 30d of inactivity since lifecycle/rotten was applied, the PR is closed

You can:

Reopen this PR with /reopen
Mark this PR as fresh with /remove-lifecycle rotten
Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close

k8s-ci-robot · 2022-10-14T13:20:50Z

@k8s-triage-robot: Closed this PR.

In response to this:

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages PRs according to the following rules:

After 90d of inactivity, lifecycle/stale is applied

After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied

After 30d of inactivity since lifecycle/rotten was applied, the PR is closed

You can:

Reopen this PR with /reopen

Mark this PR as fresh with /remove-lifecycle rotten

Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

k8s-ci-robot added the needs-priority Indicates a PR lacks a `priority/foo` label and requires one. label Jun 17, 2021

k8s-ci-robot added needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. labels Jun 17, 2021

k8s-ci-robot requested review from enisoc and soltysh June 17, 2021 11:19

k8s-ci-robot requested review from janetkuo and kow3ns June 17, 2021 11:22

k8s-ci-robot added ok-to-test Indicates a non-member PR verified by an org member that is safe to test. and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Jun 17, 2021

alculquicondor reviewed Jun 17, 2021

View reviewed changes

alculquicondor mentioned this pull request Jun 23, 2021

Promote LogarithmicScaleDown to Beta #101767

Merged

FillZpp force-pushed the fix-replicaset-same-node-rank branch from 309b1ca to 2c0d5a1 Compare July 26, 2021 02:19

k8s-ci-robot added size/M Denotes a PR that changes 30-99 lines, ignoring generated files. and removed size/S Denotes a PR that changes 10-29 lines, ignoring generated files. labels Jul 26, 2021

FillZpp force-pushed the fix-replicaset-same-node-rank branch from 2c0d5a1 to 62b321e Compare July 26, 2021 06:26

alculquicondor reviewed Jul 26, 2021

View reviewed changes

FillZpp force-pushed the fix-replicaset-same-node-rank branch from 62b321e to c8b326c Compare July 27, 2021 07:34

k8s-ci-robot removed the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Oct 27, 2021

FillZpp mentioned this pull request Nov 16, 2021

REQUEST: New membership for FillZpp kubernetes/org#3100

Closed

7 tasks

k8s-ci-robot assigned soltysh Jan 4, 2022

FillZpp force-pushed the fix-replicaset-same-node-rank branch 2 times, most recently from 1d46db6 to b5bc898 Compare January 5, 2022 05:36

k8s-ci-robot added area/test sig/testing Categorizes an issue or PR as relevant to SIG Testing. labels Jan 5, 2022

FillZpp force-pushed the fix-replicaset-same-node-rank branch from b5bc898 to b8064ad Compare January 5, 2022 07:43

FillZpp force-pushed the fix-replicaset-same-node-rank branch from b8064ad to 4b8d762 Compare January 5, 2022 08:27

k8s-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Apr 6, 2022

k8s-ci-robot added lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. and removed lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. labels May 6, 2022

Fix replicaset deletion rank for multiple colocation nodes

7ce7e19

Signed-off-by: FillZpp <FillZpp.pub@gmail.com>

FillZpp force-pushed the fix-replicaset-same-node-rank branch from 4b8d762 to 7ce7e19 Compare May 13, 2022 07:27

k8s-ci-robot removed the lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. label May 13, 2022

k8s-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Aug 15, 2022

k8s-ci-robot added lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. and removed lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. labels Sep 14, 2022

k8s-ci-robot closed this Oct 14, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix ReplicaSet deletion rank for multiple colocation nodes #102949

Fix ReplicaSet deletion rank for multiple colocation nodes #102949

FillZpp commented Jun 17, 2021 •

edited

k8s-ci-robot commented Jun 17, 2021

k8s-ci-robot commented Jun 17, 2021

k8s-ci-robot commented Jun 17, 2021

FillZpp commented Jun 17, 2021

gjkim42 commented Jun 17, 2021 •

edited

alculquicondor Jun 17, 2021

alculquicondor Jun 17, 2021

damemi Jun 17, 2021

alculquicondor Jun 17, 2021

alculquicondor Jun 17, 2021

FillZpp Jun 24, 2021

alculquicondor Jun 24, 2021

FillZpp Jun 25, 2021

alculquicondor Jun 25, 2021

alculquicondor Jun 29, 2021

FillZpp commented Jul 26, 2021

FillZpp commented Jul 26, 2021

alculquicondor Jul 26, 2021

FillZpp Jul 27, 2021

alculquicondor commented Jul 26, 2021

FillZpp commented Oct 27, 2021

dims commented Jan 4, 2022

k8s-ci-robot commented Jan 5, 2022

FillZpp commented Jan 6, 2022

k8s-triage-robot commented Apr 6, 2022

k8s-triage-robot commented May 6, 2022

FillZpp commented May 13, 2022

k8s-triage-robot commented Aug 15, 2022

k8s-triage-robot commented Sep 14, 2022

k8s-triage-robot commented Oct 14, 2022

k8s-ci-robot commented Oct 14, 2022

	func (s ActivePodsWithRanks) Less(i, j int) bool {
	// 1. Unassigned < assigned
	// If only one of the pods is unassigned, the unassigned one is smaller
	if s.Pods[i].Spec.NodeName != s.Pods[j].Spec.NodeName && (len(s.Pods[i].Spec.NodeName) == 0 \|\| len(s.Pods[j].Spec.NodeName) == 0) {
	return len(s.Pods[i].Spec.NodeName) == 0
	}
	// 2. PodPending < PodUnknown < PodRunning
	if podPhaseToOrdinal[s.Pods[i].Status.Phase] != podPhaseToOrdinal[s.Pods[j].Status.Phase] {
	return podPhaseToOrdinal[s.Pods[i].Status.Phase] < podPhaseToOrdinal[s.Pods[j].Status.Phase]
	}
	// 3. Not ready < ready
	// If only one of the pods is not ready, the not ready one is smaller
	if podutil.IsPodReady(s.Pods[i]) != podutil.IsPodReady(s.Pods[j]) {
	return !podutil.IsPodReady(s.Pods[i])
	}

	// 4. higher pod-deletion-cost < lower pod-deletion cost
	if utilfeature.DefaultFeatureGate.Enabled(features.PodDeletionCost) {
	pi, _ := helper.GetDeletionCostFromPodAnnotations(s.Pods[i].Annotations)
	pj, _ := helper.GetDeletionCostFromPodAnnotations(s.Pods[j].Annotations)
	if pi != pj {
	return pi < pj
	}
	}

	// 5. Doubled up < not doubled up
	// If one of the two pods is on the same node as one or more additional
	// ready pods that belong to the same replicaset, whichever pod has more
	// colocated ready pods is less
	if s.Rank[i] != s.Rank[j] {
	return s.Rank[i] > s.Rank[j]
	}
	// TODO: take availability into account when we push minReadySeconds information from deployment into pods,
	// see https://github.com/kubernetes/kubernetes/issues/22065
	// 6. Been ready for empty time < less time < more time
	// If both pods are ready, the latest ready one is smaller
	if podutil.IsPodReady(s.Pods[i]) && podutil.IsPodReady(s.Pods[j]) {
	readyTime1 := podReadyTime(s.Pods[i])
	readyTime2 := podReadyTime(s.Pods[j])
	if !readyTime1.Equal(readyTime2) {
	if !utilfeature.DefaultFeatureGate.Enabled(features.LogarithmicScaleDown) {
	return afterOrZero(readyTime1, readyTime2)
	} else {
	if s.Now.IsZero() \|\| readyTime1.IsZero() \|\| readyTime2.IsZero() {
	return afterOrZero(readyTime1, readyTime2)
	}
	rankDiff := logarithmicRankDiff(readyTime1, readyTime2, s.Now)
	if rankDiff == 0 {
	return s.Pods[i].UID < s.Pods[j].UID
	}
	return rankDiff < 0
	}
	}
	}
	// 7. Pods with containers with higher restart counts < lower restart counts
	if maxContainerRestarts(s.Pods[i]) != maxContainerRestarts(s.Pods[j]) {
	return maxContainerRestarts(s.Pods[i]) > maxContainerRestarts(s.Pods[j])
	}
	// 8. Empty creation time pods < newer pods < older pods
	if !s.Pods[i].CreationTimestamp.Equal(&s.Pods[j].CreationTimestamp) {
	if !utilfeature.DefaultFeatureGate.Enabled(features.LogarithmicScaleDown) {
	return afterOrZero(&s.Pods[i].CreationTimestamp, &s.Pods[j].CreationTimestamp)
	} else {
	if s.Now.IsZero() \|\| s.Pods[i].CreationTimestamp.IsZero() \|\| s.Pods[j].CreationTimestamp.IsZero() {
	return afterOrZero(&s.Pods[i].CreationTimestamp, &s.Pods[j].CreationTimestamp)
	}
	rankDiff := logarithmicRankDiff(s.Pods[i].CreationTimestamp, s.Pods[j].CreationTimestamp, s.Now)
	if rankDiff == 0 {
	return s.Pods[i].UID < s.Pods[j].UID
	}
	return rankDiff < 0
	}
	}
	return false
	}

Fix ReplicaSet deletion rank for multiple colocation nodes #102949

Fix ReplicaSet deletion rank for multiple colocation nodes #102949

Conversation

FillZpp commented Jun 17, 2021 • edited

What type of PR is this?

What this PR does / why we need it:

Which issue(s) this PR fixes:

Special notes for your reviewer:

Does this PR introduce a user-facing change?

k8s-ci-robot commented Jun 17, 2021

k8s-ci-robot commented Jun 17, 2021

k8s-ci-robot commented Jun 17, 2021

FillZpp commented Jun 17, 2021

gjkim42 commented Jun 17, 2021 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

FillZpp commented Jul 26, 2021

FillZpp commented Jul 26, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

alculquicondor commented Jul 26, 2021

FillZpp commented Oct 27, 2021

dims commented Jan 4, 2022

k8s-ci-robot commented Jan 5, 2022

FillZpp commented Jan 6, 2022

k8s-triage-robot commented Apr 6, 2022

k8s-triage-robot commented May 6, 2022

FillZpp commented May 13, 2022

k8s-triage-robot commented Aug 15, 2022

k8s-triage-robot commented Sep 14, 2022

k8s-triage-robot commented Oct 14, 2022

k8s-ci-robot commented Oct 14, 2022

FillZpp commented Jun 17, 2021 •

edited

gjkim42 commented Jun 17, 2021 •

edited