Drop reapers #63979

soltysh · 2018-05-17T15:30:24Z

Release note:

kubectl delete does not use reapers for removing objects anymore, but relies on server-side GC entirely

soltysh · 2018-05-17T15:31:10Z

Only last commit matters, b/c it waits for #59851 to merge first.

deads2k · 2018-05-17T17:46:24Z

pkg/kubectl/cmd/apply_test.go

-		})
-	}
-}
+// TODO: fix this test


um... Yeah. you can't merge it like this.

deads2k · 2018-05-17T17:46:54Z

/hold

don't want to accidentally merge with tests commented out.

soltysh · 2018-05-24T11:33:10Z

/hold cancel
Rebased and test fixed. @deads2k @juanvallejo ptal

soltysh · 2018-05-25T15:15:07Z

Nit fixed, re-tagging.

fejta-bot · 2018-05-25T18:26:05Z

/retest
This bot automatically retries jobs that failed/flaked on approved PRs (send feedback to fejta).

Review the full test history for this PR.

Silence the bot with an /lgtm cancel comment for consistent failures.

Initializers are alpha, broken and a subject for removal. They don't work well with finalizers and the previous hack present in deployment and replicaset reapers was just hiding this problem.

deads2k · 2018-05-25T20:24:20Z

Provide links to the hacks you showed me. These hacks were hiding more fundamental flaws in the implementation. They didn't work for other clients or other types. It was wrong to add the hacks.

/lgtm

k8s-ci-robot · 2018-05-25T20:24:27Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: deads2k, soltysh

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~hack/OWNERS~~ [deads2k,soltysh]
~~pkg/kubectl/OWNERS~~ [deads2k,soltysh]
~~test/OWNERS~~ [deads2k,soltysh]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

soltysh · 2018-05-25T20:27:14Z

Hack in deployment reaper:

kubernetes/pkg/kubectl/delete.go

Lines 425 to 429 in 8306b0b

    
           if deployment.Initializers != nil { 
        
           	policy := metav1.DeletePropagationBackground 
        
           	deleteOptions := &metav1.DeleteOptions{PropagationPolicy: &policy} 
        
           	return deployments.Delete(name, deleteOptions) 
        
           }

fejta-bot · 2018-05-25T22:38:39Z

/retest
This bot automatically retries jobs that failed/flaked on approved PRs (send feedback to fejta).

Review the full test history for this PR.

Silence the bot with an /lgtm cancel comment for consistent failures.

fejta-bot · 2018-05-26T01:27:34Z

/retest
This bot automatically retries jobs that failed/flaked on approved PRs (send feedback to fejta).

Review the full test history for this PR.

Silence the bot with an /lgtm cancel comment for consistent failures.

fejta-bot · 2018-05-26T05:59:09Z

/retest
This bot automatically retries jobs that failed/flaked on approved PRs (send feedback to fejta).

Review the full test history for this PR.

Silence the bot with an /lgtm cancel comment for consistent failures.

k8s-github-robot · 2018-05-26T07:32:09Z

Automatic merge from submit-queue (batch tested with PRs 63859, 63979). If you want to cherry-pick this change to another branch, please follow the instructions here.

Automatic merge from submit-queue (batch tested with PRs 64300, 64375). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>. Declare kubectl wait flag in a way consistent with other deletion flags **What this PR does / why we need it**: A follow up PR for #64034 and #63979 that makes declaring wait flag consistent with the other flags. **Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*: Fixes #64401 **Special notes for your reviewer**: **Release note**: ```release-note ```

@MaciekPytel

Automatic merge from submit-queue. Wait for the job to be removed **What this PR does / why we need it**: In master we've dropped reapers (#63979) which means the old client does not wait long enough for the resource to be gone (since it's being removed on the server along with its dependents). To fix our e2e ([failure here](https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-new-master-gci-kubectl-skew/7980)) we need to backport parts of the aforementioned PR which is updating tests to wait for the resource to be removed. **Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*: Fixes #64362 /assign @MaciekPytel /assign @deads2k **Release note**: ```release-note NONE ```

Automatic merge from submit-queue (batch tested with PRs 64749, 64797). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>. Handle deleted DaemonSet properly **What this PR does / why we need it**: After kubectl reapers are removed (#63979) and foreground deletion are used, DaemonSet controller may race with garbage collector when it tries to update DaemonSet status of the DaemonSet being deleted. Here's what happened: 1. Someone/something performs a foreground deletion on a DaemonSet 1. DaemonSet finalizer and DeletionTimestamp are both set 1. DaemonSet history objects (ControllerRevisions) and pods are being deleted by garbage collector; meanwhile, DaemonSet controller tries to update DaemonSet status. * Updating DaemonSet status requires constructing DaemonSet history objects, to figure out current revision and which pods do/don't belong to current revision 1. When updating DaemonSet status, DaemonSet controller tries to create a DaemonSet history object that matches current DaemonSet spec 1. Garbage collector then tries to delete that DaemonSet history object. And repeat. Because we can't make DaemonSet pods be deleted before DaemonSet history objects (DaemonSet history objects don't own DaemonSet pods!), we cannot reliably calculate DaemonSet status without history objects anyways. Therefore, we don't update DaemonSet status for DaemonSet being deleted. Note that the reason why the kubectl delete hack works is because it forces DaemonSet pods to be removed before history objects. **Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*: Fixes #64313 **Special notes for your reviewer**: **Release note**: ```release-note NONE ```

k8s-ci-robot added the release-note Denotes a PR that will be considered when it comes time to generate release notes. label May 17, 2018

k8s-ci-robot assigned deads2k and juanvallejo May 17, 2018

k8s-ci-robot added size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. approved Indicates a PR has been approved by an approver from all required OWNERS files. labels May 17, 2018

k8s-ci-robot requested review from dashpole, ericchiang, foxish, janetkuo, msau42, mwielgus, nicksardo, resouer, timothysc and vishh May 17, 2018 15:30

deads2k reviewed May 17, 2018

View reviewed changes

k8s-ci-robot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label May 17, 2018

timothysc removed their request for review May 17, 2018 20:44

This was referenced May 18, 2018

Use DeleteOptions.PropagationPolicy instead of OrphanDependents (deprecated) in kubectl #59851

Merged

kubectl --wait flag #63695

Closed

soltysh force-pushed the drop_reapers branch from 25c7502 to 149cf2a Compare May 18, 2018 12:51

k8s-ci-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label May 18, 2018

soltysh force-pushed the drop_reapers branch from 149cf2a to e84fbfa Compare May 18, 2018 12:57

k8s-ci-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label May 18, 2018

nilebox mentioned this pull request May 19, 2018

add kubectl wait #64034

Merged

soltysh force-pushed the drop_reapers branch from e84fbfa to 0c305fb Compare May 24, 2018 11:32

soltysh force-pushed the drop_reapers branch from 55f6295 to 3888f08 Compare May 25, 2018 15:14

k8s-ci-robot removed the lgtm "Looks good to me", indicates that a PR is ready to be merged. label May 25, 2018

soltysh added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label May 25, 2018

soltysh added 3 commits May 25, 2018 22:18

Remove kubectl reapers

3838726

Handle DaemonSet removal the old way

b8b4c7c

Remove initializer test

2a76535

Initializers are alpha, broken and a subject for removal. They don't work well with finalizers and the previous hack present in deployment and replicaset reapers was just hiding this problem.

soltysh force-pushed the drop_reapers branch from 3888f08 to 2a76535 Compare May 25, 2018 20:18

k8s-ci-robot removed the lgtm "Looks good to me", indicates that a PR is ready to be merged. label May 25, 2018

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label May 25, 2018

k8s-github-robot merged commit 5f578f3 into kubernetes:master May 26, 2018

nilebox mentioned this pull request May 27, 2018

Declare kubectl wait flag in a way consistent with other deletion flags #64375

Merged

wgliang mentioned this pull request May 28, 2018

pull-kubernetes-e2e-gce-device-plugin-gpu test always fail #64384

Closed

soltysh deleted the drop_reapers branch May 28, 2018 12:57

soltysh mentioned this pull request May 29, 2018

Wait for the job to be removed #64439

Merged

nilebox mentioned this pull request Jun 2, 2018

kubectl delete should wait for resource to be deleted before returning #42594

Closed

4 tasks

janetkuo mentioned this pull request Jun 5, 2018

Handle deleted DaemonSet properly #64797

Merged

mortent mentioned this pull request Sep 23, 2018

StatefulSet delete behavior changed in v1.11 #68627

Closed

kopiczko mentioned this pull request Nov 29, 2018

update to Kubernetes 1.12.3 giantswarm/helmclient#66

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Drop reapers #63979

Drop reapers #63979

soltysh commented May 17, 2018

soltysh commented May 17, 2018

deads2k May 17, 2018

soltysh May 18, 2018 •

edited

Loading

deads2k commented May 17, 2018

soltysh commented May 24, 2018

soltysh commented May 25, 2018

fejta-bot commented May 25, 2018

deads2k commented May 25, 2018

k8s-ci-robot commented May 25, 2018

soltysh commented May 25, 2018

fejta-bot commented May 25, 2018

fejta-bot commented May 26, 2018

fejta-bot commented May 26, 2018

k8s-github-robot commented May 26, 2018

Drop reapers #63979

Drop reapers #63979

Conversation

soltysh commented May 17, 2018

soltysh commented May 17, 2018

deads2k May 17, 2018

Choose a reason for hiding this comment

soltysh May 18, 2018 • edited Loading

Choose a reason for hiding this comment

deads2k commented May 17, 2018

soltysh commented May 24, 2018

soltysh commented May 25, 2018

fejta-bot commented May 25, 2018

deads2k commented May 25, 2018

k8s-ci-robot commented May 25, 2018

soltysh commented May 25, 2018

fejta-bot commented May 25, 2018

fejta-bot commented May 26, 2018

fejta-bot commented May 26, 2018

k8s-github-robot commented May 26, 2018

soltysh May 18, 2018 •

edited

Loading