CronJob: Respect ControllerRef #42177

enisoc · 2017-02-27T18:17:23Z

What this PR does / why we need it:

This is part of the completion of the ControllerRef proposal. It brings CronJob into compliance with ControllerRef. See the individual commit messages for details.

Which issue this PR fixes:

This ensures that other controllers do not fight over control of objects that a CronJob owns.

Special notes for your reviewer:

Release note:

CronJob controller now respects ControllerRef to avoid fighting with other controllers.

cc @erictune @kubernetes/sig-apps-pr-reviews

k8s-reviewable · 2017-02-27T18:17:35Z

This change is

soltysh

LGTM

soltysh · 2017-02-27T21:02:00Z

/approve

soltysh · 2017-02-27T21:03:37Z

/lgtm

kow3ns · 2017-02-27T22:15:40Z

Reviewed 9 of 9 files at r1.
Review status: all files reviewed at latest revision, 1 unresolved discussion, some commit checks broke.

pkg/controller/cronjob/cronjob_controller.go, line 58 at r1 (raw file):

// ControllerKind contains the schema.GroupVersionKind for this controller type.
var ControllerKind = batch.SchemeGroupVersion.WithKind("CronJob")

Probably should not be exported from package

Comments from Reviewable

calebamiles · 2017-03-02T01:05:58Z

@k8s-bot kops aws e2e test this
@k8s-bot cvm gce e2e test this
@k8s-bot gci gce e2e test this
@k8s-bot gce etcd3 e2e test this

enisoc · 2017-03-04T00:18:57Z

@k8s-bot node e2e test this
@k8s-bot gci gce e2e test this
@k8s-bot cvm gce e2e test this

enisoc · 2017-03-06T20:22:17Z

@soltysh Please take a look. I believe all comments are addressed.

soltysh · 2017-03-07T09:48:47Z

/lgtm

soltysh · 2017-03-07T13:29:41Z

@ethernetdan @calebamiles is this ok to merge for 1.6 release?

0xmichalis · 2017-04-14T18:26:22Z

pkg/controller/cronjob/cronjob_controller.go

+	// List children (Jobs) before parents (CronJob).
+	// This guarantees that if we see any Job that got orphaned by the GC orphan finalizer,
+	// we must also see that the parent CronJob has non-nil DeletionTimestamp (see #42639).
+	// Note that this only works because we are NOT using any caches here.
 	jl, err := jm.kubeClient.BatchV1().Jobs(metav1.NamespaceAll).List(metav1.ListOptions{})
 	if err != nil {
 		glog.Errorf("Error listing jobs")


The error is not printed here

0xmichalis · 2017-04-14T18:26:43Z

pkg/controller/cronjob/cronjob_controller.go

@@ -117,6 +116,14 @@ func (jm *CronJobController) syncAll() {
 	js := jl.Items
 	glog.V(4).Infof("Found %d jobs", len(js))

+	sjl, err := jm.kubeClient.BatchV2alpha1().CronJobs(metav1.NamespaceAll).List(metav1.ListOptions{})
+	if err != nil {
+		glog.Errorf("Error listing cronjobs: %v", err)


Can you change glog.Errorf calls to utilruntime.HandleError (the latter is ratelimited among other things)

0xmichalis · 2017-04-14T18:27:42Z

pkg/controller/cronjob/cronjob_controller.go

@@ -237,6 +244,18 @@ func syncOne(sj *batchv2alpha1.CronJob, js []batchv1.Job, now time.Time, jc jobC
 	}
 	*sj = *updatedSJ

+	if sj.DeletionTimestamp != nil {
+		// The CronJob is being deleted.
+		// Don't do anything other than updating status.


There is no status update.

Oh well it seems it has already happened above

0xmichalis · 2017-04-14T18:32:30Z

test/e2e/cronjob.go

+
+	// Adopt Jobs it owns that don't have ControllerRef yet.
+	// That is, the Jobs were created by a pre-v1.6.0 master.
+	It("should adopt Jobs it owns that don't have ControllerRef yet", func() {


This could also be an integration test, no?

This was mainly added because we've run into problems with RBAC before. Do integration tests cover that?

I would expect so, @deads2k ?

Now that CronJob adds ControllerRef to Jobs it creates, we need to set this default so legacy behavior is maintained.

This should only happen if the Jobs were created by an older version of the CronJob controller, since from now on we add ControllerRef upon creation. CronJob doesn't do actual adoption because it doesn't use label selectors to find its Jobs. However, we should apply ControllerRef for potential server-side cascading deletion, and to advise other controllers we own these objects.

janetkuo · 2017-04-18T22:24:24Z

pkg/controller/cronjob/utils.go

+			continue
+		}
+		job.OwnerReferences = append(job.OwnerReferences, sjControllerRef)
+		updatedJob, err := jc.UpdateJob(job.Namespace, job)


Now that we have more time, shall we switch to patch?

janetkuo · 2017-04-18T22:33:43Z

test/e2e/generated_clientset.go

@@ -324,7 +324,8 @@ var _ = framework.KubeDescribe("Generated release_1_5 clientset", func() {
 		observeCreation(w)

 		By("deleting the cronJob")
-		if err := cronJobClient.Delete(cronJob.Name, nil); err != nil {
+		// Use OrphanDependents=false so the delete is synchronous.
+		if err := cronJobClient.Delete(cronJob.Name, &metav1.DeleteOptions{OrphanDependents: new(bool)}); err != nil {


OrphanDependents is being deprecated in 1.7. Should we switch to use deleteOptions.propagationPolicy instead?

janetkuo · 2017-04-18T22:43:30Z

pkg/controller/cronjob/cronjob_controller.go

+	// List children (Jobs) before parents (CronJob).
+	// This guarantees that if we see any Job that got orphaned by the GC orphan finalizer,
+	// we must also see that the parent CronJob has non-nil DeletionTimestamp (see #42639).
+	// Note that this only works because we are NOT using any caches here.


@soltysh we'll need to take care of this when we change cronjob controller to use informer

Yes, we've talked about it during KubeCon. If I have a reasonable solution in place in openshift I'll make sure to fix cronjob controller.

janetkuo · 2017-04-18T22:58:35Z

Do we need to test that SJ won't adopt Jobs it doesn't own?

enisoc · 2017-04-19T21:41:12Z

Do we need to test that SJ won't adopt Jobs it doesn't own?

This is unit tested via TestGroupJobsByParent(). Jobs that the CronJob doesn't own should not even be passed into adoptJobs().

The e2e test is only to ensure surprises like RBAC don't prevent adopt calls from working. In this case, the correct action is to do nothing, so we don't need an e2e test to make sure other components don't interfere.

There is no "release/orphan" case since CronJob does not use a label selector to find its children.

This is needed now that the default is OrphanDependents.

This prevents a race with the GC while it orphans dependents.

Currently, an e2e test is the only way to ensure we have the proper RBAC permissions to adopt Jobs.

janetkuo

LGTM

k8s-github-robot · 2017-04-20T19:57:03Z

Automatic merge from submit-queue (batch tested with PRs 42177, 42176, 44721)

davidghiurco · 2018-06-13T20:24:53Z

Does this mean that whenever I delete a cronjob from a k8s cluster, all its jobs, and associated pods would get deleted too?
Because I am still seeing orphaned jobs and pods with
kubectl get jobs
and
kubectl get pods

after deleting their associated cronjob. See my comment on #42627

soltysh · 2018-06-14T09:59:37Z

Does this mean that whenever I delete a cronjob from a k8s cluster, all its jobs, and associated pods would get deleted too?

That depends how you remove the object, it is still possible to remove an object and orphan its dependents.

davidghiurco · 2018-06-20T19:29:23Z

I delete the object using helm.
e.g.
helm delete --purge <helm_release_name>

soltysh · 2018-06-21T10:55:32Z

Can you provide me the actual requests helm delete sends over to the server, I'm not familiar with helm.

enisoc added the release-note Denotes a PR that will be considered when it comes time to generate release notes. label Feb 27, 2017

enisoc added this to the v1.6 milestone Feb 27, 2017

enisoc assigned janetkuo and kow3ns Feb 27, 2017

k8s-ci-robot added the cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. label Feb 27, 2017

k8s-github-robot added the size/L Denotes a PR that changes 100-499 lines, ignoring generated files. label Feb 27, 2017

0xmichalis assigned soltysh Feb 27, 2017

enisoc force-pushed the controller-ref-cronjob branch 2 times, most recently from 4973031 to d471655 Compare February 27, 2017 18:42

soltysh approved these changes Feb 27, 2017

View reviewed changes

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Feb 27, 2017

enisoc force-pushed the controller-ref-cronjob branch from d471655 to 1083d91 Compare February 28, 2017 01:02

k8s-github-robot removed the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Feb 28, 2017

janetkuo assigned caesarxuchao Mar 1, 2017

enisoc added the kind/bug Categorizes issue or PR as related to a bug. label Mar 1, 2017

enisoc force-pushed the controller-ref-cronjob branch 2 times, most recently from 9586055 to 3e166bd Compare March 3, 2017 23:00

0xmichalis mentioned this pull request Mar 6, 2017

Implement ScheduledJob Reaper #31553

Closed

enisoc force-pushed the controller-ref-cronjob branch from 3e166bd to 7505b1f Compare March 6, 2017 19:18

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Mar 7, 2017

0xmichalis mentioned this pull request Mar 7, 2017

kubectl delete cronjob xxx do not delete all jobs related with the cron job #42627

Closed

0xmichalis reviewed Apr 14, 2017

View reviewed changes

enisoc added 5 commits April 18, 2017 13:59

CronJob: Don't launch if DeletionTimestamp is set.

4c954d6

CronJob: Add ControllerRef on all created Jobs.

d72eebd

CronJob: Set DefaultGarbageCollectionPolicy to OrphanDependents.

631d8bc

Now that CronJob adds ControllerRef to Jobs it creates, we need to set this default so legacy behavior is maintained.

CronJob: Always set BlockOwnerDeletion in ControllerRef.

4e682fd

enisoc force-pushed the controller-ref-cronjob branch from bb3305d to c168dad Compare April 18, 2017 21:18

janetkuo reviewed Apr 18, 2017

View reviewed changes

enisoc force-pushed the controller-ref-cronjob branch from c168dad to 01aded9 Compare April 19, 2017 22:31

enisoc added 5 commits April 19, 2017 15:42

CronJob: Use synchronous deletion for CronJobs in e2e.

5cef455

This is needed now that the default is OrphanDependents.

CronJob: Check ControllerRef Name and UID in unit test.

4e1b07d

CronJob: List children before parents.

1e14323

This prevents a race with the GC while it orphans dependents.

CronJob: Add e2e test for adoption.

33d7788

Currently, an e2e test is the only way to ensure we have the proper RBAC permissions to adopt Jobs.

CronJob: Use PATCH to adopt Jobs.

be1fe95

enisoc force-pushed the controller-ref-cronjob branch from 01aded9 to be1fe95 Compare April 19, 2017 22:42

janetkuo approved these changes Apr 19, 2017

View reviewed changes

enisoc added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Apr 20, 2017

k8s-github-robot merged commit f25a657 into kubernetes:master Apr 20, 2017

enisoc deleted the controller-ref-cronjob branch April 21, 2017 19:19

enisoc mentioned this pull request Apr 24, 2017

Implement controllerRef #24946

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

CronJob: Respect ControllerRef #42177

CronJob: Respect ControllerRef #42177

enisoc commented Feb 27, 2017 •

edited

Loading

k8s-reviewable commented Feb 27, 2017

soltysh left a comment

soltysh commented Feb 27, 2017

soltysh commented Feb 27, 2017

kow3ns commented Feb 27, 2017

calebamiles commented Mar 2, 2017

enisoc commented Mar 4, 2017

enisoc commented Mar 6, 2017

soltysh commented Mar 7, 2017

soltysh commented Mar 7, 2017

0xmichalis Apr 14, 2017

0xmichalis Apr 14, 2017

0xmichalis Apr 14, 2017 •

edited

Loading

0xmichalis Apr 14, 2017

0xmichalis Apr 14, 2017

enisoc Apr 18, 2017

0xmichalis Apr 19, 2017

janetkuo Apr 18, 2017

janetkuo Apr 18, 2017

enisoc Apr 19, 2017

janetkuo Apr 18, 2017

soltysh Apr 24, 2017

janetkuo commented Apr 18, 2017

enisoc commented Apr 19, 2017

janetkuo left a comment

k8s-github-robot commented Apr 20, 2017

davidghiurco commented Jun 13, 2018

soltysh commented Jun 14, 2018

davidghiurco commented Jun 20, 2018

soltysh commented Jun 21, 2018

CronJob: Respect ControllerRef #42177

CronJob: Respect ControllerRef #42177

Conversation

enisoc commented Feb 27, 2017 • edited Loading

k8s-reviewable commented Feb 27, 2017

soltysh left a comment

Choose a reason for hiding this comment

soltysh commented Feb 27, 2017

soltysh commented Feb 27, 2017

kow3ns commented Feb 27, 2017

calebamiles commented Mar 2, 2017

enisoc commented Mar 4, 2017

enisoc commented Mar 6, 2017

soltysh commented Mar 7, 2017

soltysh commented Mar 7, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

0xmichalis Apr 14, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

janetkuo commented Apr 18, 2017

enisoc commented Apr 19, 2017

janetkuo left a comment

Choose a reason for hiding this comment

k8s-github-robot commented Apr 20, 2017

davidghiurco commented Jun 13, 2018

soltysh commented Jun 14, 2018

davidghiurco commented Jun 20, 2018

soltysh commented Jun 21, 2018

enisoc commented Feb 27, 2017 •

edited

Loading

0xmichalis Apr 14, 2017 •

edited

Loading