Rolling changes across cluster instances

[mcandre]

Where high availability is desired, it would be helpful for Terraform to provide the option to apply changes to instances on a rolling basis. For example, rather than deleting-creating all nodes at once, delete-create one node at a time, ideally with healthchecks executed before each change.

A more advanced version of this feature could provide a parameter for a minimum number of instances to leave alone during the rolling change. For example, Kafka brokers theoretically need a minimum of 1 node for overall cluster availability, and ZooKeeper needs a minimum of 3 nodes for overall cluster availability. Such a setting would allow operators selecting for expedience of cluster changes vs. selecting for cluster reliability during changes.

In any case, even a dumb one-by-one rolling change flag would be tremendously helpful for operator staff managing large clusters towards five nines reliability.