fix bug on kubectl deleting uninitialized resources #51186

dixudx · 2017-08-23T09:50:40Z

What this PR does / why we need it:

Which issue this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close that issue when PR gets merged): fixes #51185

Special notes for your reviewer:
/assign @caesarxuchao @ahmetb

Release note:

fix bug on kubectl deleting uninitialized resources

k8s-ci-robot · 2017-08-23T09:50:47Z

Hi @dixudx. Thanks for your PR.

I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

liggitt · 2017-08-23T14:30:14Z

pkg/kubectl/delete.go

@@ -226,6 +226,17 @@ func (reaper *ReplicaSetReaper) Stop(namespace, name string, timeout time.Durati
 		timeout = Timeout + time.Duration(10*rs.Spec.Replicas)*time.Second
 	}

+	_, err = reaper.updateReplicaSetWithRetries(namespace, name, func(rs *extensions.ReplicaSet) {
+		// set replicaset's initializers to nil


chasing disabling initializers on everything seems very error-prone. @smarterclayton, what would you expect to happen here?

I definitely think that this is a bug in the reaper. Why do we still have the reaper?

A normal user isn't going to be able to perform this operation, so this isn't the correct fix. I think if the object hasn't been initialized yet the reaper should just call delete.

dims · 2017-08-25T12:36:30Z

/ok-to-test

caesarxuchao · 2017-08-25T17:48:20Z

@dixudx could you explain what's the reason of the deletion failure? Is it because kubectl is waiting for the controller's status (like observedGeneration, replicas etc) to update? If that's the case, you can early return in the Scale() call. That way you don't need an extra GET/UPDATE.

The entire problem will be gone when we move kubectl delete to use the server-side garbage collection (tracked in #50340).

dixudx · 2017-08-26T08:15:55Z

Is it because kubectl is waiting for the controller's status (like observedGeneration, replicas etc) to update?

@caesarxuchao Yes, current kubectl will keeps watching the status changes. While for such uninitialized objects with pending initializers, nothing will get changed.

If that's the case, you can early return in the Scale() call.

Yes, this is a nicer way to solve this problem. That means we have to make special pass for Deployment, ReplicaSet, and StatefulSet in the Scale() call, like

if rs.Initializers != nil {
     return nil
}

@liggitt @caesarxuchao WDYT?

caesarxuchao · 2017-08-27T00:13:47Z

@dixudx that looks right to me.

@smarterclayton the removal of the reapers is tracked at: #50340

dixudx · 2017-08-27T00:50:36Z

ping @caesarxuchao @smarterclayton @liggitt

Updated. PTAL. Thanks.

dixudx · 2017-08-27T09:53:02Z

/retest

smarterclayton · 2017-08-28T00:41:44Z

Much simpler

dixudx · 2017-08-28T03:32:03Z

/retest

caesarxuchao · 2017-08-30T22:17:06Z

pkg/kubectl/delete.go

@@ -400,6 +400,9 @@ func (reaper *DeploymentReaper) Stop(namespace, name string, timeout time.Durati
 		d.Spec.RevisionHistoryLimit = &rhl
 		d.Spec.Replicas = 0
 		d.Spec.Paused = true
+		if d.Initializers != nil {
+			d.Initializers = nil


We can't assume kubectl user has the permission to set initializers. Just get the deployment at the beginnig at Stop and delete it immediately if initializers!=nil.

caesarxuchao · 2017-08-30T22:18:50Z

Could you add some tests? Maybe one test case for kubectl delete rs and another one for kubectl delete deployment, since the mechanisms of theirs reapers are different.

dixudx · 2017-08-31T06:48:43Z

@caesarxuchao Add the tests. PTAL.

caesarxuchao · 2017-08-31T07:26:20Z

hack/make-rules/test-cmd-util.sh

+  # Pre-condition: no deployment exists
+  kube::test::get_object_assert deployment "{{range.items}}{{$id_field}}:{{end}}" ''
+  # Create a deployment
+  kubectl create -f hack/testdata/deployment-with-initializer.yaml "${kube_flags[@]}"


#51436 disabled the initializer field if the feature gate is off, perhaps you need to enable the feature gate in this test.

caesarxuchao · 2017-09-01T19:53:55Z

/lgtm

caesarxuchao · 2017-09-01T20:07:42Z

/assign @liggitt @fabianofranz for approval

Thanks.

k8s-ci-robot · 2017-09-01T20:07:43Z

@caesarxuchao: GitHub didn't allow me to assign the following users: for, approval.

Note that only kubernetes members can be assigned.

In response to this:

/assign @liggitt @fabianofranz for approval

Thanks.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

caesarxuchao · 2017-09-02T00:33:21Z

This is a bug fix, so it's ok we don't get approval before the code freeze deadline.

dixudx · 2017-09-02T13:35:57Z

ping @liggitt @fabianofranz for approval. Thanks.

liggitt · 2017-09-02T14:09:27Z

/approve

liggitt · 2017-09-02T15:31:53Z

@fabianofranz @pwittrock for approval and approved-for-milestone tagging if this is required for 1.8

fabianofranz · 2017-09-05T21:00:21Z

/test pull-kubernetes-e2e-kops-aws

fabianofranz · 2017-09-05T21:00:45Z

/approve

k8s-github-robot · 2017-09-05T21:01:41Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: caesarxuchao, dixudx, fabianofranz, liggitt

Associated issue: 51185

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these OWNERS Files:

~~hack/OWNERS~~ [fabianofranz]
~~pkg/kubectl/OWNERS~~ [fabianofranz,liggitt]

You can indicate your approval by writing /approve in a comment
You can cancel your approval by writing /approve cancel in a comment

k8s-github-robot · 2017-09-05T23:35:17Z

/test all [submit-queue is verifying that this PR is safe to merge]

k8s-github-robot · 2017-09-06T00:29:59Z

Automatic merge from submit-queue (batch tested with PRs 51186, 50350, 51751, 51645, 51837)

k8s-ci-robot · 2017-09-06T00:34:23Z

@dixudx: The following test failed, say /retest to rerun them all:

Test name	Commit	Details	Rerun command
pull-kubernetes-e2e-kops-aws	`5e120cf`	link	`/test pull-kubernetes-e2e-kops-aws`

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

k8s-ci-robot assigned ahmetb and caesarxuchao Aug 23, 2017

k8s-ci-robot added needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. labels Aug 23, 2017

k8s-github-robot added size/M Denotes a PR that changes 30-99 lines, ignoring generated files. release-note Denotes a PR that will be considered when it comes time to generate release notes. labels Aug 23, 2017

liggitt reviewed Aug 23, 2017

View reviewed changes

k8s-ci-robot removed the needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. label Aug 25, 2017

dixudx force-pushed the fix_delete_uninitialized_resources branch from 95bddf0 to 493f063 Compare August 27, 2017 00:49

k8s-ci-robot added size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. and removed size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels Aug 27, 2017

caesarxuchao reviewed Aug 30, 2017

View reviewed changes

dixudx force-pushed the fix_delete_uninitialized_resources branch from 493f063 to 7448365 Compare August 31, 2017 06:28

k8s-ci-robot added size/M Denotes a PR that changes 30-99 lines, ignoring generated files. and removed size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. labels Aug 31, 2017

k8s-github-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Aug 31, 2017

dixudx force-pushed the fix_delete_uninitialized_resources branch from 7448365 to 41c8799 Compare August 31, 2017 06:34

k8s-github-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Aug 31, 2017

caesarxuchao reviewed Aug 31, 2017

View reviewed changes

add tests

5e120cf

dixudx force-pushed the fix_delete_uninitialized_resources branch from d5471b5 to 5e120cf Compare September 1, 2017 02:46

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Sep 1, 2017

k8s-ci-robot assigned fabianofranz and liggitt Sep 1, 2017

liggitt added this to the v1.8 milestone Sep 2, 2017

k8s-github-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Sep 5, 2017

fabianofranz added the status/approved-for-milestone label Sep 5, 2017

k8s-github-robot merged commit ee4e4a5 into kubernetes:master Sep 6, 2017

dixudx deleted the fix_delete_uninitialized_resources branch September 6, 2017 01:14

dixudx mentioned this pull request Sep 18, 2017

add document on kubectl's behavior regarding initializers kubernetes/website#5505

Merged

shyamjvs mentioned this pull request Sep 28, 2017

delete pods API call latencies shot up on large cluster tests #51899

Closed

shyamjvs mentioned this pull request Oct 10, 2017

Large kubemark performance tests failing with timeout during ns deletion #53327

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix bug on kubectl deleting uninitialized resources #51186

fix bug on kubectl deleting uninitialized resources #51186

dixudx commented Aug 23, 2017

k8s-ci-robot commented Aug 23, 2017

liggitt Aug 23, 2017

smarterclayton Aug 26, 2017

dims commented Aug 25, 2017

caesarxuchao commented Aug 25, 2017

dixudx commented Aug 26, 2017

caesarxuchao commented Aug 27, 2017

dixudx commented Aug 27, 2017

dixudx commented Aug 27, 2017

smarterclayton commented Aug 28, 2017

dixudx commented Aug 28, 2017

caesarxuchao Aug 30, 2017

caesarxuchao commented Aug 30, 2017

dixudx commented Aug 31, 2017

caesarxuchao Aug 31, 2017

caesarxuchao commented Sep 1, 2017

caesarxuchao commented Sep 1, 2017

k8s-ci-robot commented Sep 1, 2017

caesarxuchao commented Sep 2, 2017

dixudx commented Sep 2, 2017

liggitt commented Sep 2, 2017

liggitt commented Sep 2, 2017

fabianofranz commented Sep 5, 2017

fabianofranz commented Sep 5, 2017

k8s-github-robot commented Sep 5, 2017

k8s-github-robot commented Sep 5, 2017

k8s-github-robot commented Sep 6, 2017

k8s-ci-robot commented Sep 6, 2017 •

edited

fix bug on kubectl deleting uninitialized resources #51186

fix bug on kubectl deleting uninitialized resources #51186

Conversation

dixudx commented Aug 23, 2017

k8s-ci-robot commented Aug 23, 2017

liggitt Aug 23, 2017

Choose a reason for hiding this comment

smarterclayton Aug 26, 2017

Choose a reason for hiding this comment

dims commented Aug 25, 2017

caesarxuchao commented Aug 25, 2017

dixudx commented Aug 26, 2017

caesarxuchao commented Aug 27, 2017

dixudx commented Aug 27, 2017

dixudx commented Aug 27, 2017

smarterclayton commented Aug 28, 2017

dixudx commented Aug 28, 2017

caesarxuchao Aug 30, 2017

Choose a reason for hiding this comment

caesarxuchao commented Aug 30, 2017

dixudx commented Aug 31, 2017

caesarxuchao Aug 31, 2017

Choose a reason for hiding this comment

caesarxuchao commented Sep 1, 2017

caesarxuchao commented Sep 1, 2017

k8s-ci-robot commented Sep 1, 2017

caesarxuchao commented Sep 2, 2017

dixudx commented Sep 2, 2017

liggitt commented Sep 2, 2017

liggitt commented Sep 2, 2017

fabianofranz commented Sep 5, 2017

fabianofranz commented Sep 5, 2017

k8s-github-robot commented Sep 5, 2017

k8s-github-robot commented Sep 5, 2017

k8s-github-robot commented Sep 6, 2017

k8s-ci-robot commented Sep 6, 2017 • edited

k8s-ci-robot commented Sep 6, 2017 •

edited