Fix: remove workload finalizer if workload is finished #1523

achernevskii · 2023-12-27T17:57:13Z

What type of PR is this?

/kind bug

What this PR does / why we need it:

Workload will be finalized if it has a Finished condition and no OwnerReferences.

Which issue(s) this PR fixes:

Fixes #1450

Special notes for your reviewer:

Does this PR introduce a user-facing change?

Remove finalizer from Workloads that are orphaned (have no owners).

netlify · 2023-12-27T17:57:19Z

✅ Deploy Preview for kubernetes-sigs-kueue canceled.

Name	Link
🔨 Latest commit	`a5181ea`
🔍 Latest deploy log	https://app.netlify.com/sites/kubernetes-sigs-kueue/deploys/65a051d464cb780008a71433

tenzen-y · 2024-01-04T09:45:17Z

pkg/controller/core/workload_controller.go

@@ -133,8 +134,8 @@ func (r *WorkloadReconciler) Reconcile(ctx context.Context, req ctrl.Request) (c
 	ctx = ctrl.LoggerInto(ctx, log)
 	log.V(2).Info("Reconciling Workload")

-	if apimeta.IsStatusConditionTrue(wl.Status.Conditions, kueue.WorkloadFinished) {
-		return ctrl.Result{}, nil
+	if apimeta.IsStatusConditionTrue(wl.Status.Conditions, kueue.WorkloadFinished) && len(wl.ObjectMeta.OwnerReferences) == 0 {


Why do we need to check if the workload has ownerReference?
Will Argo remove ownerReference from the workload?

If workload has an owner reference, there's a possibility that the owner job should be stopped:

kueue/pkg/controller/jobframework/reconciler.go

Line 257 in bb69906

err := r.stopJob(ctx, job, wl, StopReasonWorkloadDeleted, "Workload is deleted")

We will introduce a race condition between the workload and job controller if we don't check for the owner references here.

And the workload is finalized here for the uncommon scenario, when the workload has no owner references, so the reconcile for respective job could not be queued.

I think it should look more like this:

if len(owners) == 0 && deletionTimestamp != nil { finalize() } if finished { return }

Note that workloads might not have owners in two cases:

When using prebuilt workload Prebuilt workload suport #1358

When using orphan cascade deletion https://kubernetes.io/docs/tasks/administer-cluster/use-cascading-deletion/#set-orphan-deletion-policy

@achernevskii Thank you for the clarifications!
As @alculquicondor mentioned, there are some positive situations when the workload doesn't have
ownerReference. So, I think that we should apply the Aldo's suggestion.

Changed the behavior in f0690dc

tenzen-y · 2024-01-04T09:55:03Z

pkg/controller/core/workload_controller_test.go

+					Type:   "Finished",
+					Status: "True",
+				}).
+				Obj(),
 		},


Should we add a case for the workload with ownerReference?

tenzen-y · 2024-01-04T10:07:25Z

pkg/controller/jobs/pod/pod_controller_test.go

+							Obj(),
+					).
+					Admitted(true).
+					Delete().


Should we verify in

kueue/pkg/controller/jobs/pod/pod_controller_test.go

Line 2153 in 6242c0f

}

if the workload has deletionTimestamp instead of adding such a deletionTimestamp?

tenzen-y · 2024-01-04T10:08:29Z

pkg/util/workload/workload.go

+	kueue "sigs.k8s.io/kueue/apis/kueue/v1beta1"
+)
+
+func RemoveWorkloadFinalizer(ctx context.Context, c client.Client, wl *kueue.Workload) error {


Should we move this function to the workload (pkg/workload) package?

Done in f0690dc

tenzen-y · 2024-01-04T10:08:59Z

@achernevskii Also, can you fix the CI error?

alculquicondor · 2024-01-04T14:50:18Z

/cc

alculquicondor · 2024-01-09T14:06:24Z

/assign @trasc

tenzen-y

Overall lgtm

tenzen-y · 2024-01-10T17:05:28Z

pkg/controller/core/workload_controller_test.go

+					Status: "True",
+				}).
+				OwnerReference(batchv1.SchemeGroupVersion.String(), "Job", "job", "test-uid", true, true).
+				Delete().


Comparing the deletionTimestamp would cause a flaky test?
So, can we verify if the workload has the deletionTImestamp in Line 427 instead of comparison?

Or you can use EquateApproxTime? or a AcyclicTransformer that only considers whether the values are nil.

We can probably use EquateApproxTime.

let's just provide the desired time to the wrapper

tenzen-y · 2024-01-10T17:06:15Z

pkg/util/testing/wrappers.go

+// Delete sets a deletion timestamp for the pod object
+func (w *WorkloadWrapper) Delete() *WorkloadWrapper {


Suggested change

// Delete sets a deletion timestamp for the pod object

func (w *WorkloadWrapper) Delete() *WorkloadWrapper {

// DeletionTimestamp sets a deletion timestamp for the pod object

func (w *WorkloadWrapper) DeletionTimestamp() *WorkloadWrapper {

tenzen-y · 2024-01-10T17:07:16Z

pkg/util/testing/wrappers.go

+	t := metav1.NewTime(time.Now()).Rfc3339Copy()
+	w.Workload.DeletionTimestamp = &t


Suggested change

t := metav1.NewTime(time.Now()).Rfc3339Copy()

w.Workload.DeletionTimestamp = &t

w.Workload.DeletionTimestamp := ptr.To(metav1.NewTime(time.Now()).Rfc3339Copy())

What is this Rfc3339Copy doing? truncating to second?

What is this Rfc3339Copy doing? truncating to second?

Correct, had to add it because of kubernetes/kubernetes#81026

alculquicondor · 2024-01-10T18:52:32Z

pkg/util/testing/wrappers.go

+	t := metav1.NewTime(time.Now()).Rfc3339Copy()
+	w.Workload.DeletionTimestamp = &t


What is this Rfc3339Copy doing? truncating to second?

alculquicondor · 2024-01-10T18:55:17Z

pkg/controller/jobframework/reconciler.go

-		}
-
-		if err == nil && wl != nil {
-			err = r.removeFinalizer(ctx, wl)


why change this? how will this workload be finalized?

alculquicondor · 2024-01-10T18:56:13Z

pkg/controller/core/workload_controller_test.go

+					Status: "True",
+				}).
+				OwnerReference(batchv1.SchemeGroupVersion.String(), "Job", "job", "test-uid", true, true).
+				Delete().


Or you can use EquateApproxTime? or a AcyclicTransformer that only considers whether the values are nil.

alculquicondor

/lgtm
/approve

k8s-ci-robot · 2024-01-11T15:30:40Z

LGTM label has been added.

Git tree hash: 1f5b261905a2aafebc4e6108ac5fc65f9e5d10dc

alculquicondor · 2024-01-11T15:30:49Z

/hold

alculquicondor · 2024-01-11T15:31:09Z

Please squash, so that cherry-picking is less cumbersome.

tenzen-y

/lgtm
/approve

Please squash, so that cherry-picking is less cumbersome.

@alculquicondor Isn't the default merge method squash?

https://github.com/kubernetes/test-infra/blob/c96b01283ef705e2a3c1f0b836b8ffeef0223c13/config/prow/config.yaml#L783

tenzen-y · 2024-01-11T15:58:02Z

/cherry-pick release-0.5

k8s-infra-cherrypick-robot · 2024-01-11T15:58:04Z

@tenzen-y: once the present PR merges, I will cherry-pick it on top of release-0.5 in a new PR and assign it to you.

In response to this:

/cherry-pick release-0.5

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

alculquicondor · 2024-01-11T15:59:25Z

It is, but the cherry-pick bot still goes by the original commits. Not sure if the individual commits could run into conflicts where the squashed commit wouldn't

tenzen-y · 2024-01-11T16:02:52Z

It is, but the cherry-pick bot still goes by the original commits. Not sure if the individual commits could run into conflicts where the squashed commit wouldn't

I see. I'm fine with manually squash.

trasc · 2024-01-11T20:42:22Z

squashed, but somehow it detected code changes due to rebase. Needs LGTM.

/unhold

trasc · 2024-01-11T20:58:00Z

/lgtm

k8s-ci-robot · 2024-01-11T20:58:07Z

LGTM label has been added.

Git tree hash: 352be45fd720ee5036953f279568a014bda84ee7

k8s-infra-cherrypick-robot · 2024-01-11T21:04:45Z

@tenzen-y: #1523 failed to apply on top of branch "release-0.5":

Applying: fix: remove finalizer if workload finished
Using index info to reconstruct a base tree...
M	pkg/controller/core/workload_controller.go
M	pkg/controller/jobframework/reconciler.go
M	pkg/controller/jobs/pod/pod_controller_test.go
M	pkg/util/testing/wrappers.go
M	pkg/workload/workload.go
Falling back to patching base and 3-way merge...
Auto-merging pkg/workload/workload.go
Auto-merging pkg/util/testing/wrappers.go
CONFLICT (content): Merge conflict in pkg/util/testing/wrappers.go
Auto-merging pkg/controller/jobs/pod/pod_controller_test.go
CONFLICT (content): Merge conflict in pkg/controller/jobs/pod/pod_controller_test.go
Auto-merging pkg/controller/jobframework/reconciler.go
CONFLICT (content): Merge conflict in pkg/controller/jobframework/reconciler.go
Auto-merging pkg/controller/core/workload_controller.go
error: Failed to merge in the changes.
hint: Use 'git am --show-current-patch=diff' to see the failed patch
Patch failed at 0001 fix: remove finalizer if workload finished
When you have resolved this problem, run "git am --continue".
If you prefer to skip this patch, run "git am --skip" instead.
To restore the original branch and stop patching, run "git am --abort".

In response to this:

/cherry-pick release-0.5

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

alculquicondor · 2024-01-11T21:53:38Z

@tenzen-y do you think it's that important to cherry-pick this?

It's a bit of an edge case.

tenzen-y · 2024-01-12T16:41:30Z

@tenzen-y do you think it's that important to cherry-pick this?

It's a bit of an edge case.

@alculquicondor I think that cherry-picking might be better since we have a chance to release a patch version (v0.5.2).
If we don't have any plan to release a patch version, I'm ok with either way.

Are you concerned about manyconflicts?

alculquicondor · 2024-01-12T16:56:09Z

@trasc can you try to get a manual cherry-pick?

Co-authored-by: Lukas Wöhrl <lukas.woehrl@plentymarkets.com>

trasc · 2024-01-15T13:01:31Z

@trasc can you try to get a manual cherry-pick?

done #1583

Co-authored-by: Aleksei Chernevskii <aleksei_chernevskii@epam.com> Co-authored-by: Lukas Wöhrl <lukas.woehrl@plentymarkets.com>

tenzen-y · 2024-01-15T16:11:04Z

/release-note-edit

Fix a bug in the pod integration that the pod's finalizer won't be removed even though the pod is finished

alculquicondor · 2024-01-18T19:16:07Z

/release-note-edit

Remove finalizer from Workloads that are orphaned (have no owners).

k8s-ci-robot added release-note-none Denotes a PR that doesn't merit a release note. kind/bug Categorizes issue or PR as related to a bug. labels Dec 27, 2023

k8s-ci-robot added the cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. label Dec 27, 2023

k8s-ci-robot requested review from denkensk and mimowo December 27, 2023 17:57

k8s-ci-robot added the size/L Denotes a PR that changes 100-499 lines, ignoring generated files. label Dec 27, 2023

achernevskii mentioned this pull request Dec 27, 2023

Update pod controller to reconcile not found pods #1512

Merged

woehrl01 mentioned this pull request Dec 28, 2023

fix: remove finalizer if workload finished #1454

Closed

tenzen-y mentioned this pull request Jan 4, 2024

Release v0.5.2 #1539

Closed

22 tasks

tenzen-y reviewed Jan 4, 2024

View reviewed changes

k8s-ci-robot requested a review from alculquicondor January 4, 2024 14:50

trasc force-pushed the fix/workload_finalizer_removal branch from f0690dc to 0a44617 Compare January 9, 2024 09:16

k8s-ci-robot assigned trasc Jan 9, 2024

tenzen-y reviewed Jan 10, 2024

View reviewed changes

alculquicondor reviewed Jan 10, 2024

View reviewed changes

trasc force-pushed the fix/workload_finalizer_removal branch from 160e42f to acbed0c Compare January 11, 2024 14:42

alculquicondor reviewed Jan 11, 2024

View reviewed changes

k8s-ci-robot assigned alculquicondor Jan 11, 2024

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Jan 11, 2024

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jan 11, 2024

k8s-ci-robot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Jan 11, 2024

tenzen-y reviewed Jan 11, 2024

View reviewed changes

k8s-ci-robot assigned tenzen-y Jan 11, 2024

fix: remove finalizer if workload finished

a5181ea

trasc force-pushed the fix/workload_finalizer_removal branch from acbed0c to a5181ea Compare January 11, 2024 20:38

k8s-ci-robot removed the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Jan 11, 2024

k8s-ci-robot requested review from alculquicondor, tenzen-y and trasc January 11, 2024 20:38

k8s-ci-robot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Jan 11, 2024

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Jan 11, 2024

k8s-ci-robot merged commit 336a3cb into kubernetes-sigs:main Jan 11, 2024
14 checks passed

k8s-ci-robot added this to the v0.6 milestone Jan 11, 2024

trasc pushed a commit to epam/kubernetes-kueue that referenced this pull request Jan 15, 2024

fix: remove finalizer if workload finished (kubernetes-sigs#1523)

0cc6423

Co-authored-by: Lukas Wöhrl <lukas.woehrl@plentymarkets.com>

trasc mentioned this pull request Jan 15, 2024

[release-0.5] Fix: remove workload finalizer if workload is finished #1583

Merged

k8s-ci-robot pushed a commit that referenced this pull request Jan 15, 2024

fix: remove finalizer if workload finished (#1523) (#1583)

f5a2cda

Co-authored-by: Aleksei Chernevskii <aleksei_chernevskii@epam.com> Co-authored-by: Lukas Wöhrl <lukas.woehrl@plentymarkets.com>

k8s-ci-robot added release-note Denotes a PR that will be considered when it comes time to generate release notes. and removed release-note-none Denotes a PR that doesn't merit a release note. labels Jan 15, 2024

trasc deleted the fix/workload_finalizer_removal branch March 12, 2024 08:55

		// Delete sets a deletion timestamp for the pod object
		func (w WorkloadWrapper) Delete() WorkloadWrapper {

		t := metav1.NewTime(time.Now()).Rfc3339Copy()
		w.Workload.DeletionTimestamp = &t

	t := metav1.NewTime(time.Now()).Rfc3339Copy()
	w.Workload.DeletionTimestamp = &t
	w.Workload.DeletionTimestamp := ptr.To(metav1.NewTime(time.Now()).Rfc3339Copy())

Fix: remove workload finalizer if workload is finished #1523

Fix: remove workload finalizer if workload is finished #1523

Conversation

achernevskii commented Dec 27, 2023 • edited by k8s-ci-robot

What type of PR is this?

What this PR does / why we need it:

Which issue(s) this PR fixes:

Special notes for your reviewer:

Does this PR introduce a user-facing change?

netlify bot commented Dec 27, 2023 • edited

✅ Deploy Preview for kubernetes-sigs-kueue canceled.

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tenzen-y commented Jan 4, 2024

alculquicondor commented Jan 4, 2024

alculquicondor commented Jan 9, 2024

tenzen-y left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

alculquicondor left a comment

Choose a reason for hiding this comment

k8s-ci-robot commented Jan 11, 2024

alculquicondor commented Jan 11, 2024

alculquicondor commented Jan 11, 2024

tenzen-y left a comment

Choose a reason for hiding this comment

tenzen-y commented Jan 11, 2024

k8s-infra-cherrypick-robot commented Jan 11, 2024

alculquicondor commented Jan 11, 2024

tenzen-y commented Jan 11, 2024

trasc commented Jan 11, 2024

trasc commented Jan 11, 2024

k8s-ci-robot commented Jan 11, 2024

k8s-infra-cherrypick-robot commented Jan 11, 2024

alculquicondor commented Jan 11, 2024

tenzen-y commented Jan 12, 2024

alculquicondor commented Jan 12, 2024

trasc commented Jan 15, 2024

tenzen-y commented Jan 15, 2024

alculquicondor commented Jan 18, 2024

achernevskii commented Dec 27, 2023 •

edited by k8s-ci-robot

netlify bot commented Dec 27, 2023 •

edited