kops rolling-update should do a real rolling-update

[justinsb]

Right now it is a hacky timing-based loop.

Ideally we would wait for the new node to be registered.

[justinsb]

We should also evict nodes before terminating them.

[gigaroby]

What about using a similar strategy to that followed by pod rolling updates in kubernetes?
There could a parameter that controls how many nodes are affected at any given time (let's call it N) and the strategy could be something like:

1. increase the size of the instance group by N
2. wait for the new N nodes to register themselves to the master
3. cordon and drain N old nodes, then delete them from the instance group
4. wait for the new instances to be available on the master
5. repeat from step 3 until there are no more old instances
6. decrease the size of the instance group back by N again

I am not an expert on how ASGs work on EC2 but doing this naively may have some problems, namely that it could be slow to wait for the ASG to realize that N nodes have been deleted and to spawn new ones and also the behavior of the last step may be a bit problematic.

[justinsb]

Note: we should also rolling-update the masters before the nodes, in the case of an version upgrade.

[chrislovecnm]

This is handled much better in my open PR.

[krisnova]

@chrislovecnm - pointer please? Which PR?

[chrislovecnm]

#1134 Drain and validate in rolling update

[mirague]

Is this is expected in 1.5.1?

[chrislovecnm]

@mirague I am working on the PR again today, hopefully, our next release with a feature flag to turn it on.

[mirague]

That's fantastic to hear, thanks @chrislovecnm !

[toidiu]

any update on this?

[Miyurz]

@chrislovecnm any updates here ?

[chrislovecnm]

@Miyurz we have had a feature flag in that allows for drain and validate. It is stable, especially for stateless applications. We need some more TLC for stateless applications.

Use KOPS_FEATURE_FLAGS="+DrainAndValidateRollingUpdate" to use beta code that drains the nodes and validates the cluster. New flags for Drain and Validation operations will be shown when
the environment variable is set.

[chrislovecnm]

Closing is this is stable, and we have #1718