kubectl drain does not attempt to wait for any rescheduling

[mikekap]

Is this a BUG REPORT or FEATURE REQUEST? (choose one): FEATURE REQUEST

Kubernetes version (use kubectl version): 1.3.5

Environment:

• Cloud provider or hardware configuration: AWS EC2 (using Autoscaling)
• OS (e.g. from /etc/os-release): Ubuntu Xenial

What happened:
When doing a rolling update of the nodes, I ran into some downtime even though I was carefully draining & restarting machines one at a time. The scenario was:

• Drain first node
• Replace said node with a new node (automatically via ASG)
• Wait for kube to stabilize (i.e. all pods scheduled)
• Drain second node
• Watch as one of my deployments gets fully killed.

kubectl drain didn't notice that the scheduler put an entire replica set on the same node.

What you expected to happen:
I believe this is an artifact of resource scarcity - the scheduler normally attempts to distribute replica sets. Fixing this in the scheduler would perhaps be a lot harder since the scheduler would need to relocate the pod when the new node comes online.

Instead it would be nice if kubectl drain maintained an invariant like "at least one pod in each replica set that has at least 2 pods needs to be alive at any time".

How to reproduce it (as minimally and precisely as possible): Try instructions above with only 2 nodes.

Anything else do we need to know:

[hjacobs]

@mikekap isn't this what Pod Disruption Budget is for?

[xiangpengzhao]

/sig cli

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

Prevent issues from auto-closing with an /lifecycle frozen comment.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or @fejta.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten
/remove-lifecycle stale

[fejta-bot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close