mumoshu Automatic recovery from permanent failures of etcd3 nodes (#417)
## Features

* Automatic recovery from any number of permanently failed etcd nodes (when `etcd.disasterRecovery.automated` is set to true
  * Currently, it is considered to be a "permanent failure" when health checks to an etcd member is failing longer than the threshold(10 seconds by default)

## Notable changes

* Upgraded etcd to v3
* kube-apiserver uses etcd3 API rather than the former etcd2 API for writing v3 keys when etcd3 is chosen for the etcd cluster
  * Although you can stick with v2 api + v3 etcd(then kube-apiserver writes v2 keys in etcd v3!), the current implementation of etcd snapshot/restore doesn't support etcd v2 data. That's why I made this change
  * etcd3 will be the default storage backend since k8s 1.6 anyways
* Wrote a lengthy `etcdadm` script to automate etcd member health checking, saving snapshots, recovering from up to `N/2` permanently failed nodes(=replace failed members one by one), recovering from more than `N/2` permanently failed nodes(=bootstrapping a brand-new cluster with the latest snapshot)

## Changelog

* Add etcdadm for scripting various etcd3 administration tasks to achieve high availability

* Enable etcd3 paired with the `etcdadm reconfigure` service

* Periodically run etcdadm (save|check) when `etcd.snapshot.automated` or `etcd.disasterRecovery.automated` is set to true, respectively

* Fail-fast etcd-member when `etcdadm reconfigure` fails

* When etcd3 is chosen, use etcd v3 API for communication between etcd and apiserver
k8s state is persisted in etcd3 data rather than etcd2 data in an etcd3 cluster(etcd3 can serve both v2 api and v3 api for accessing v2 data and v3 data respectively)

* Turn off automatic Container Linux updates on Etcd nodes

* Support both v2 and v3 of etcd & Allow switching etcd version (for now)

* Fix a validation for awsNodeLabels

* e2e: Add a convenient sub-command for invoking kube-aws on a specific test cluster

* Start etcd health-checking only after etcd is tried to be started
or we'll end up with health-checking doesn't work when etcd failed to start at all
Latest commit a14ba0f Apr 4, 2017