When --grace-period=0 is provided, wait for deletion #37263

smarterclayton · 2016-11-22T03:24:45Z

The grace-period is automatically set to 1 unless --force is provided, and the client waits until the object is deleted.

This preserves backwards compatibility with 1.4 and earlier. It does not handle scenarios where the object is deleted and a new object is created with the same name because we don't have the initial object loaded (and that's a larger change for 1.5).

Fixes #37117 by relaxing the guarantees provided.

When deleting an object with `--grace-period=0`, the client will begin a graceful deletion and wait until the resource is fully deleted.  To force deletion, use the `--force` flag.

This change is

smarterclayton · 2016-11-22T03:34:14Z

Proposed for 1.5 to avoid backwards compatibility.

k8s-ci-robot · 2016-11-22T03:36:30Z

Jenkins Bazel Build failed for commit 28a4d56. Full PR test history.

The magic incantation to run this job again is @k8s-bot bazel test this. Please help us cut down flakes by linking to an open flake issue when you hit one in your PR.

k8s-ci-robot · 2016-11-22T03:41:02Z

Jenkins GCI GKE smoke e2e failed for commit 32a7c417a81661f05a056334d0305e145d278375. Full PR test history.

The magic incantation to run this job again is @k8s-bot gci gke e2e test this. Please help us cut down flakes by linking to an open flake issue when you hit one in your PR.

smarterclayton · 2016-11-22T05:11:25Z

@k8s-bot bazel test this

fabianofranz · 2016-11-22T14:58:47Z

@smarterclayton so both --grace-period=0 and --now will set gracePeriod to 1, but the difference is that the first waits?

fabianofranz · 2016-11-22T15:04:49Z

pkg/kubectl/cmd/delete.go

+			// into --grace-period=1 and wait until the object is successfully deleted. Users may provide --force
+			// to bypass this wait.
+			wait = true
+			gracePeriod = 1


--grace-period=0 may now be mutually exclusive with --timeout

It's not, timeout is still respected.

smarterclayton · 2016-11-22T15:51:49Z

so both --grace-period=0 and --now will set gracePeriod to 1, but the difference is that the first waits?

To avoid breaking old scripts, --grace-period=0 continues to not return (waits) until the object is gone. --now continues to behave as it did before.

fabianofranz · 2016-11-22T17:12:10Z

LGTM

krousey · 2016-11-22T18:42:29Z

pkg/client/restclient/request.go

@@ -818,10 +818,12 @@ func (r *Request) request(fn func(*http.Request, *http.Response)) error {
 			// connection.
 			defer func() {
 				const maxBodySlurpSize = 2 << 10
-				if resp.ContentLength <= maxBodySlurpSize {
-					io.Copy(ioutil.Discard, &io.LimitedReader{R: resp.Body, N: maxBodySlurpSize})
+				if resp.Body != nil {


This check should not be required.

// The http Client and Transport guarantee that Body is always // non-nil, even on responses without a body or responses with // a zero-length body.

Test cases were passing nil in, and we guard against it in other places. I'll remove both guards and hunt down the bug.

krousey · 2016-11-22T18:43:26Z

pkg/kubectl/cmd/delete_test.go

+				case 1, 2, 3:
+					return &http.Response{StatusCode: 200, Header: defaultHeader(), Body: objBody(codec, &pods.Items[0])}, nil
+				default:
+					return &http.Response{StatusCode: 404, Header: defaultHeader(), Body: nil}, nil


This should provide a non-nil body, even if it is empty.

foxish · 2016-11-22T18:46:58Z

pkg/kubectl/cmd/delete_test.go

 	"k8s.io/kubernetes/pkg/kubectl/resource"
+	"time"


nit: import ordering.

krousey · 2016-11-22T22:57:23Z

cc @roberthbailey

The grace-period is automatically set to 1 unless --force is provided, and the client waits until the object is deleted. This preserves backwards compatibility with 1.4 and earlier. It does not handle scenarios where the object is deleted and a new object is created with the same name.

smarterclayton · 2016-11-23T21:02:55Z

Updated with all comments.

saad-ali · 2016-11-23T21:46:16Z

@k8s-bot test this

dchen1107 · 2016-11-23T21:48:32Z

This pr might need more attention, and I decided not manually merge it for holiday code freeze in favor to merge two large prs: #36673
and #36782

smarterclayton · 2016-11-23T21:50:41Z

Yes, that's fine with me.

eparis · 2016-11-28T21:09:17Z

@k8s-bot gce node e2e test this

eparis · 2016-11-28T21:23:08Z

@k8s-bot node e2e test this

saad-ali · 2016-11-29T02:21:10Z

@foxish can you review. Let's get this merged ASAP, so we can unblock upgrade testing

foxish · 2016-11-29T10:33:10Z

LGTM except for the couple of comments.

foxish · 2016-11-29T11:09:15Z

pkg/kubectl/cmd/replace.go

 	// By default use a reaper to delete all related resources.
 	if cmdutil.GetFlagBool(cmd, "cascade") {
 		glog.Warningf("\"cascade\" is set, kubectl will delete and re-create all resources managed by this resource (e.g. Pods created by a ReplicationController). Consider using \"kubectl rolling-update\" if you want to update a ReplicationController together with its Pods.")
-		err = ReapResult(r, f, out, cmdutil.GetFlagBool(cmd, "cascade"), ignoreNotFound, timeout, cmdutil.GetFlagInt(cmd, "grace-period"), shortOutput, mapper, false)
+		err = ReapResult(r, f, out, cmdutil.GetFlagBool(cmd, "cascade"), ignoreNotFound, timeout, gracePeriod, waitForDeletion, shortOutput, mapper, false)
 	} else {
 		err = DeleteResult(r, out, ignoreNotFound, shortOutput, mapper)


It seems like this should go through ReapResult as well? Currently, we would print out "pod is deleted" prior to its actual deletion, and then eventually hit the timeout condition a few lines below. This is in the case where --cascade=false.

We don't actually even do graceful deletion in this code path (we use the server default) so it's not backwards incompatible. Definitely something we need to fix in the future, but not for 1.5

foxish · 2016-11-29T11:43:11Z

pkg/kubectl/cmd/delete.go

+	copied := *info
+	info = &copied
+	// TODO: refactor Reaper so that we can pass the "wait" option into it, and then check for UID change.
+	return wait.PollImmediate(objectDeletionWaitInterval, timeout, func() (bool, error) {


I think in the timeout case, we should return a better error than "timed out waiting for the condition" because we surface
that error to the user.

the error is going to be "error when stopping pod/foo: timed out waiting for the condition" which is pretty close. I think that's something we could improve in a follow up (there's basically a follow up here which is delete needs a better cleanup)

foxish · 2016-11-29T16:42:17Z

/lgtm

smarterclayton · 2016-11-29T17:28:25Z

Follow up issue opened as #37637

…

On Tue, Nov 29, 2016 at 11:59 AM, Kubernetes Submit Queue < ***@***.***> wrote: @k8s-bot <https://github.com/k8s-bot> test this Tests are more than 96 hours old. Re-running tests. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#37263 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABG_p1MjqpuojUbmyGcoHXo4GfP_ZN4pks5rDFn9gaJpZM4K494f> .

k8s-ci-robot · 2016-11-29T19:29:04Z

Jenkins GCE Node e2e failed for commit 7cdb6b1. Full PR test history.

The magic incantation to run this job again is @k8s-bot node e2e test this. Please help us cut down flakes by linking to an open flake issue when you hit one in your PR.

saad-ali · 2016-11-30T02:47:25Z

@k8s-bot node e2e test this

saad-ali · 2016-11-30T02:47:42Z

@k8s-bot kops aws e2e test this

dims · 2016-11-30T17:29:10Z

@k8s-bot node e2e test this

k8s-github-robot · 2016-11-30T19:15:17Z

Automatic merge from submit-queue

…63-upstream-release-1.5 Automated cherry pick of #37263 upstream release 1.5

k8s-ci-robot added the cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. label Nov 22, 2016

smarterclayton force-pushed the wait_on_immediate branch from 32a7c41 to 28a4d56 Compare November 22, 2016 03:28

k8s-github-robot assigned fabianofranz Nov 22, 2016

k8s-github-robot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. release-note-label-needed labels Nov 22, 2016

smarterclayton mentioned this pull request Nov 22, 2016

kubectl --force deletion is breaking upgrade tests #37117

Closed

smarterclayton added this to the v1.5 milestone Nov 22, 2016

saad-ali assigned foxish and pwittrock Nov 22, 2016

fabianofranz reviewed Nov 22, 2016

View reviewed changes

krousey suggested changes Nov 22, 2016

View reviewed changes

foxish reviewed Nov 22, 2016

View reviewed changes

pkg/kubectl/cmd/delete_test.go

"k8s.io/kubernetes/pkg/kubectl/resource"

"time"

Copy link

Contributor

foxish Nov 22, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: import ordering.

smarterclayton force-pushed the wait_on_immediate branch from 28a4d56 to 7cdb6b1 Compare November 23, 2016 21:02

krousey approved these changes Nov 28, 2016

View reviewed changes

foxish added the cherrypick-candidate label Nov 28, 2016

foxish reviewed Nov 29, 2016

View reviewed changes

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Nov 29, 2016

eparis removed the priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. label Nov 30, 2016

k8s-github-robot merged commit 9ccc291 into kubernetes:master Nov 30, 2016

saad-ali mentioned this pull request Nov 30, 2016

Automated cherry pick of #37263 upstream release 1.5 #37709

Merged

smarterclayton mentioned this pull request Nov 30, 2016

Automated cherry pick of #37263 #37711

Closed

saad-ali added a commit that referenced this pull request Nov 30, 2016

Merge pull request #37709 from saad-ali/automated-cherry-pick-of-#372…

01dd9a7

…63-upstream-release-1.5 Automated cherry pick of #37263 upstream release 1.5

saad-ali added cherry-pick-approved Indicates a cherry-pick PR into a release branch has been approved by the release branch manager. and removed cherrypick-candidate labels Dec 1, 2016

smarterclayton mentioned this pull request Dec 9, 2016

v1.5.0 known issues / FAQ accumulator #37134

Closed

nikhita mentioned this pull request Mar 24, 2017

Improve timeout error message for kubectl delete #43606

Closed

foxish mentioned this pull request Apr 14, 2017

Pod from DaemonSet stuck in Unknown state because of NodeLost #44458

Closed

liggitt mentioned this pull request May 19, 2017

oc delete po --grace-period=0 is not respected openshift/origin#14257

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

When --grace-period=0 is provided, wait for deletion #37263

When --grace-period=0 is provided, wait for deletion #37263

smarterclayton commented Nov 22, 2016 •

edited

smarterclayton commented Nov 22, 2016

k8s-ci-robot commented Nov 22, 2016

k8s-ci-robot commented Nov 22, 2016

smarterclayton commented Nov 22, 2016

fabianofranz commented Nov 22, 2016

fabianofranz Nov 22, 2016

smarterclayton Nov 22, 2016

smarterclayton commented Nov 22, 2016

fabianofranz commented Nov 22, 2016

krousey Nov 22, 2016

smarterclayton Nov 22, 2016

krousey Nov 22, 2016

foxish Nov 22, 2016

krousey commented Nov 22, 2016

smarterclayton commented Nov 23, 2016

saad-ali commented Nov 23, 2016

dchen1107 commented Nov 23, 2016

smarterclayton commented Nov 23, 2016

eparis commented Nov 28, 2016

eparis commented Nov 28, 2016

saad-ali commented Nov 29, 2016

foxish commented Nov 29, 2016 •

edited

foxish Nov 29, 2016 •

edited

smarterclayton Nov 29, 2016

foxish Nov 29, 2016

smarterclayton Nov 29, 2016

foxish commented Nov 29, 2016

smarterclayton commented Nov 29, 2016 via email

k8s-ci-robot commented Nov 29, 2016

saad-ali commented Nov 30, 2016

saad-ali commented Nov 30, 2016

dims commented Nov 30, 2016

k8s-github-robot commented Nov 30, 2016

When --grace-period=0 is provided, wait for deletion #37263

When --grace-period=0 is provided, wait for deletion #37263

Conversation

smarterclayton commented Nov 22, 2016 • edited

smarterclayton commented Nov 22, 2016

k8s-ci-robot commented Nov 22, 2016

k8s-ci-robot commented Nov 22, 2016

smarterclayton commented Nov 22, 2016

fabianofranz commented Nov 22, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

smarterclayton commented Nov 22, 2016

fabianofranz commented Nov 22, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

krousey commented Nov 22, 2016

smarterclayton commented Nov 23, 2016

saad-ali commented Nov 23, 2016

dchen1107 commented Nov 23, 2016

smarterclayton commented Nov 23, 2016

eparis commented Nov 28, 2016

eparis commented Nov 28, 2016

saad-ali commented Nov 29, 2016

foxish commented Nov 29, 2016 • edited

foxish Nov 29, 2016 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

foxish commented Nov 29, 2016

smarterclayton commented Nov 29, 2016 via email

k8s-ci-robot commented Nov 29, 2016

saad-ali commented Nov 30, 2016

saad-ali commented Nov 30, 2016

dims commented Nov 30, 2016

k8s-github-robot commented Nov 30, 2016

smarterclayton commented Nov 22, 2016 •

edited

foxish commented Nov 29, 2016 •

edited

foxish Nov 29, 2016 •

edited