kubectl: wait for all errors and successes on podEviction #64896

rphillips · 2018-06-07T22:15:18Z

What this PR does / why we need it: This fixes kubectl drain to wait until all errors and successes are processed, instead of returning the first error. It also tweaks the behavior of the cleanup to check to see if the pod is already terminating, and if it is to not reissue the pod terminate which leads to an error getting thrown. This fix will allow kubectl drain to complete successfully when a node is draining.

Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):
Fixes #

Special notes for your reviewer:
/cc @sjenning

Release note:

NONE

Reproduction steps

sleep.yml

apiVersion: v1
kind: Pod
metadata:
  name: bash
spec: 
  containers:
  - name: bash
    image: bash
    resources:
      limits:
        cpu: 500m
        memory: 500Mi
    command:
    - bash
    - -c
    - "nothing() { sleep 1; } ; trap nothing 15 ; while true; do echo \"hello\"; sleep 10; done"
  terminationGracePeriodSeconds: 3000
  restartPolicy: Never

$ kubectl create ns testing
$ kubectl create -f sleep.yml
$ kubectl delete ns testing
$ kubectl drain 127.0.0.1 --force

rphillips · 2018-06-08T04:09:04Z

/retest

rphillips · 2018-06-12T15:02:26Z

/assign @apelisse

frobware · 2018-06-14T15:39:27Z

pkg/kubectl/cmd/drain.go

@@ -613,16 +620,21 @@ func (o *DrainOptions) evictPods(pods []corev1.Pod, policyGroupVersion string, g
 	for {
 		select {
 		case err := <-errCh:
-			return err


Do we not now lose the underlying reason? Looking below I see we just do "Drain did not complete...".

Since there can be N different errors, they get glogged and counted. A summary error is returned on line 635. Do you think we should capture all the error messages?

outputting via glog like this rather than actually returning errors means DrainOptions can't easily be used programmatically or composed into larger commands

print to o.ErrOut if you're going to print here, and I would probably accumulate and return the actual errors

sjenning · 2018-06-26T03:11:47Z

/cc @kubernetes/sig-cli-maintainers

This fix will allow kubectl drain to complete successfully when a node is draining.

The particular issue we hit is if the node has a pod in a terminating namespace, the drain will immediately fail with a "can't modify resource in a terminating namespace" error and fail to remove the remaining pods.

With this PR, the drain still fails, but not before trying to remove every pod on the node. This is a better situation in that all pods will be in a terminating state after the first drain.

sjenning · 2018-06-26T03:18:59Z

using wider cc that I just found out about
/cc @kubernetes/sig-cli-pr-reviews

rphillips · 2018-06-26T21:36:41Z

@liggitt I refactored this PR to collect all the errors and make it more reusable.

adohe-zz · 2018-06-27T01:05:12Z

pkg/util/multierror/multierror.go

+// derived from https://github.com/golang/appengine/blob/master/errors.go
+
+// MultiError is returned by batch operations.
+type MultiError []error


we already have AggregateError, what's the difference? seems no need to define multi error here.

I didn't know about Aggregate. I updated the PR to use the Aggregate API. Thanks!

rphillips · 2018-06-27T15:07:33Z

/test pull-kubernetes-integration

rphillips · 2018-06-27T18:36:11Z

/test pull-kubernetes-e2e-gce

rphillips · 2018-06-29T14:12:56Z

@sjenning @liggitt ready for review

sjenning · 2018-07-03T19:11:46Z

thanks!
/lgtm

apelisse · 2018-07-03T19:17:48Z

pkg/kubectl/cmd/drain.go

@@ -571,39 +572,41 @@ func (o *DrainOptions) deleteOrEvictPods(pods []corev1.Pod) error {
 	}
 }

+// evictPods return


apelisse · 2018-07-03T19:22:17Z

pkg/kubectl/cmd/drain.go

 			}
 		case <-globalTimeoutCh:
-			return fmt.Errorf("Drain did not complete within %v", globalTimeout)
+			return utilerrors.NewAggregate(errors)


You're losing the timeout error information here, which means if it times out before the first error is reported, that's going to become a success. Maybe consider just returning timeout error in that case (as it was done before)?

good catch on both. fixed.

apelisse · 2018-07-03T19:40:30Z

pkg/kubectl/cmd/drain.go

 			}
 		case <-globalTimeoutCh:
 			return fmt.Errorf("Drain did not complete within %v", globalTimeout)
 		}
+		if doneCount == numPods {


This condition could have been part of the "for" statement: for doneCount < numPods { ... }.

apelisse · 2018-07-03T19:41:02Z

Thanks for fixing quickly! Feel free to fix the "while" loop (now or later).
/lgtm
/approve

rphillips · 2018-07-03T21:12:40Z

@apelisse I fixed the for loop, this will need one more approval. Thank you!

apelisse · 2018-07-03T22:24:39Z

It'd be great if I could leave all my comments once and for all :-). How hard would it be to add tests?

/lgtm

k8s-ci-robot · 2018-07-03T22:24:44Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: apelisse, rphillips, sjenning

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~pkg/kubectl/OWNERS~~ [apelisse]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

k8s-github-robot · 2018-07-04T01:06:08Z

Automatic merge from submit-queue (batch tested with PRs 65776, 64896). If you want to cherry-pick this change to another branch, please follow the instructions here.

@sjenning

…tion Automatic merge from submit-queue (batch tested with PRs 65776, 64896). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>. kubectl: wait for all errors and successes on podEviction **What this PR does / why we need it**: This fixes `kubectl drain` to wait until all errors and successes are processed, instead of returning the first error. It also tweaks the behavior of the cleanup to check to see if the pod is already terminating, and if it is to not reissue the pod terminate which leads to an error getting thrown. This fix will allow `kubectl drain` to complete successfully when a node is draining. **Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*: Fixes # **Special notes for your reviewer**: /cc @sjenning **Release note**: ```release-note NONE ``` ```yaml apiVersion: v1 kind: Pod metadata: name: bash spec: containers: - name: bash image: bash resources: limits: cpu: 500m memory: 500Mi command: - bash - -c - "nothing() { sleep 1; } ; trap nothing 15 ; while true; do echo \"hello\"; sleep 10; done" terminationGracePeriodSeconds: 3000 restartPolicy: Never ``` ``` $ kubectl create ns testing $ kubectl create -f sleep.yml $ kubectl delete ns testing $ kubectl drain 127.0.0.1 --force ```

k8s-ci-robot added the release-note-none Denotes a PR that doesn't merit a release note. label Jun 7, 2018

k8s-ci-robot requested a review from sjenning June 7, 2018 22:15

k8s-ci-robot added size/S Denotes a PR that changes 10-29 lines, ignoring generated files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. labels Jun 7, 2018

k8s-ci-robot assigned apelisse Jun 12, 2018

frobware reviewed Jun 14, 2018

View reviewed changes

k8s-ci-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jun 24, 2018

rphillips force-pushed the fixes/kubectl_eviction branch from 49a332b to 4bfea39 Compare June 25, 2018 15:10

k8s-ci-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jun 25, 2018

k8s-ci-robot added the sig/cli Categorizes an issue or PR as relevant to SIG CLI. label Jun 26, 2018

rphillips force-pushed the fixes/kubectl_eviction branch from 4bfea39 to c158391 Compare June 26, 2018 21:35

k8s-ci-robot added size/M Denotes a PR that changes 30-99 lines, ignoring generated files. and removed size/S Denotes a PR that changes 10-29 lines, ignoring generated files. labels Jun 26, 2018

rphillips force-pushed the fixes/kubectl_eviction branch from c158391 to 4ff8f32 Compare June 26, 2018 22:29

k8s-ci-robot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. and removed size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels Jun 26, 2018

rphillips force-pushed the fixes/kubectl_eviction branch 2 times, most recently from 823bfb6 to a0efb63 Compare June 26, 2018 22:32

adohe-zz reviewed Jun 27, 2018

View reviewed changes

rphillips force-pushed the fixes/kubectl_eviction branch from a0efb63 to 4fe8896 Compare June 27, 2018 14:29

k8s-ci-robot added size/M Denotes a PR that changes 30-99 lines, ignoring generated files. and removed size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Jun 27, 2018

rphillips force-pushed the fixes/kubectl_eviction branch from 4fe8896 to 792f4dd Compare June 28, 2018 14:28

rphillips force-pushed the fixes/kubectl_eviction branch 2 times, most recently from 45495c9 to 9421527 Compare July 3, 2018 17:57

k8s-ci-robot assigned sjenning Jul 3, 2018

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Jul 3, 2018

apelisse reviewed Jul 3, 2018

View reviewed changes

rphillips force-pushed the fixes/kubectl_eviction branch from 9421527 to d8322d0 Compare July 3, 2018 19:36

k8s-ci-robot removed the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Jul 3, 2018

apelisse reviewed Jul 3, 2018

View reviewed changes

k8s-ci-robot added lgtm "Looks good to me", indicates that a PR is ready to be merged. approved Indicates a PR has been approved by an approver from all required OWNERS files. labels Jul 3, 2018

kubectl: wait for all errors and successes on podEviction

5b4770e

rphillips force-pushed the fixes/kubectl_eviction branch from d8322d0 to 5b4770e Compare July 3, 2018 21:11

k8s-ci-robot removed the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Jul 3, 2018

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Jul 3, 2018

k8s-github-robot merged commit e3fa913 into kubernetes:master Jul 4, 2018

kubectl: wait for all errors and successes on podEviction #64896

kubectl: wait for all errors and successes on podEviction #64896

rphillips commented Jun 7, 2018

rphillips commented Jun 8, 2018

rphillips commented Jun 12, 2018

frobware Jun 14, 2018

rphillips Jun 14, 2018

liggitt Jun 26, 2018

liggitt Jun 26, 2018

sjenning commented Jun 26, 2018 •

edited

sjenning commented Jun 26, 2018

rphillips commented Jun 26, 2018

adohe-zz Jun 27, 2018

rphillips Jun 27, 2018

rphillips commented Jun 27, 2018

rphillips commented Jun 27, 2018

rphillips commented Jun 29, 2018

sjenning commented Jul 3, 2018

apelisse Jul 3, 2018

apelisse Jul 3, 2018

rphillips Jul 3, 2018

apelisse Jul 3, 2018

apelisse commented Jul 3, 2018

rphillips commented Jul 3, 2018

apelisse commented Jul 3, 2018

k8s-ci-robot commented Jul 3, 2018

k8s-github-robot commented Jul 4, 2018

kubectl: wait for all errors and successes on podEviction #64896

kubectl: wait for all errors and successes on podEviction #64896

Conversation

rphillips commented Jun 7, 2018

Reproduction steps

sleep.yml

rphillips commented Jun 8, 2018

rphillips commented Jun 12, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sjenning commented Jun 26, 2018 • edited

sjenning commented Jun 26, 2018

rphillips commented Jun 26, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rphillips commented Jun 27, 2018

rphillips commented Jun 27, 2018

rphillips commented Jun 29, 2018

sjenning commented Jul 3, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

apelisse commented Jul 3, 2018

rphillips commented Jul 3, 2018

apelisse commented Jul 3, 2018

k8s-ci-robot commented Jul 3, 2018

k8s-github-robot commented Jul 4, 2018

sjenning commented Jun 26, 2018 •

edited