Master nodes bootstrapping script

[briankosw]

Hi @abohmeed, I noticed that you have the file kubernetes/node_startup.tmpl that you aren't including in your Terraform setup right now. I read through the file and it seems that it's supposed to bootstrap the master and worker nodes so that there's a single master node that initializes the Kubernetes cluster (kubeadm init) and teh other master nodes intelligently join the initialized Kubernetes cluster (kubeadm join). Could you explain whether the script works as intended? And why aren't you using the script and instead resorting manual CLI commands? Thanks!

[abohmeed]

Hello @briankosw
Yes, you're correct. The file is used to auto-join master and worker nodes to the cluster. The reason why it is not included in Terraform is that I am using this repo currently to teach how the student can build a Kubernetes cluster manually using Terraform on AWS. Using the node startup script is an advanced stage that I may include in a future lecture/course.
So, yes, you can fork the repo and use the node startup script. It works with no known issues.

[briankosw]

I see. That's great! Could you help me understand how the script ensures that only one of the master nodes run kubeadm init and the rest run kubeadm join? I've written a script that is similar in concept to yours, but I've been unable to convince myself that I can guarantee that only one master node initializes a cluster.

[abohmeed]

Simply, loop over all the master nodes over SSH and execute the init command there. If the server replies, exit the loop. Otherwise, continue looping. Once done, loop on the rest of the master/worker nodes and execute the join command.
On AWS, it's easy to get the master and worker nodes by using tags that Kubernetes already requires to be present.

[briankosw]

I see. I guess the only problem that I can see is when more than one master node runs kubeadm init, which will break the cluster according to the official documentation. What about the following change:

• Designate one master node (e.g. based on the tag as you mentioned above) as the one responsible for initializing the cluster
• The other master nodes will wait until the cluster is initialized as is done in your script
• Whenever a master node drops, it can first check that the cluster is up and join using your method in the script

What do you think about that?

[abohmeed]

I cannot see why more than one master node would run the init command at the same time. As I mentioned, you'll loop on the master nodes and run the command. The first one that reports the command has run successfully causes the loop to exit.
You run the loop sequentially and not in parallel so you try the servers one by one not all at once,

[pg3391]

I created 3 tier k8 cluster using your code @abohmeed , Cluster is good, but I'm no way able to login to bastion host using the private even my public key both, can you share if there is any fix ?