Create dedicated etcd machines backed by EBS volumes

[jimmycuadra]

It should be configurable (and probably the default set up) to launch a cluster of etcd nodes (probably 3 by default) on separate CoreOS instances and point the Kubernetes API server at them rather than having a single etcd node colocated with the API server. This is a more reliable configuration for production deployments that can tolerate potential etcd failures on a single node and provide better disaster recovery. The etcd data dirs should be located on EBS volumes (not the root device) so the instances can be terminated safely without risk of losing the data.

[vyshane]

Agreed. Ideally:

• The etcd nodes should be in an autoscaling group with a configurable initial number of nodes
• The cloud formation template should set up an internal ELB for etcd
• The etcd nodes should be backed by EBS volumes
• Generation of etcd discovery URL can be left to the user

[jimmycuadra]

Autoscaling of etcd is not a good idea and very unlikely to be supported by an automated tools like this, based on the intentional architectural design of etcd: https://github.com/coreos/etcd/blob/master/Documentation/runtime-reconf-design.md

[aaronlevy]

@jimmycuadra currently the etcd data is backed by an EBS volume (root device), so the data will survive node failure (although will be unavailable until new node is launched).

I will leave this open, however, to track the issue of launching a separate etcd cluster as an option, rather than a single node backed by EBS.

/cc @silas

[AlmogBaku]

any update on this?

[aaronlevy]

The kube-aws tool has been moved to its own top-level directory @ https://github.com/coreos/kube-aws

If this issue still needs to be addressed, please re-open the issue under the new repository.