Join GitHub today
GitHub is home to over 31 million developers working together to host and review code, manage projects, and build software together.Sign up
Adding/removing etcd nodes #43
I just tried adding and removing masters to my cluster (etcd is running on my master nodes). My plan was to replace my existing three (really bad) masters with three new masters.
I added the servers to the Ansible inventory and just ran the playbook. It didn't turn out the way I expected.
After googling around, I found out that adding new nodes to (and removing from) etcd requires you to use
$ etcdctl --cert-file /etc/etcd/pki/etcd.pem --key-file /etc/etcd/pki/etcd-key.pem --ca-file /etc/etcd/pki/ca.pem --endpoints https://127.0.0.1:2379 member add k8s-master-10 https://<ip-address>:2380
Then you get some info back that you should use in your service file:
Note the list of initial cluster nodes and also the initial cluster state.
After doing this manually for each node, I could run the Ansible playbook again to tweak the configuratin files so they are equal on all servers. That worked.
Same goes when removing servers. I had to do:
$ etcdctl --cert-file /etc/etcd/pki/etcd.pem --key-file /etc/etcd/pki/etcd-key.pem --ca-file /etc/etcd/pki/ca.pem --endpoints https://127.0.0.1:2379 member remove <node-id>
... before actually shutting them down and excluding them from the cluster.
This requires a bit of manual work (which is just fine, if it is necessary). Do you have any suggestion on how this can be automated in a better way? Or if not, could/should it be documented within KTRW?