Kubernetes Setup

Multi node Kubernetes cluster setup with kube-prometheus-stack, deployed on GKE or DigitalOcean Kubernetes (DOKS).

Installation

Make sure you have the following software installed on your system:

• Ansible
• Ansible Kubernetes Collection
• Terraform
• kubectl
• Helm
• doctl (only DOKS)
• gcloud SDK (only GKE)

Setup Domain

To set up a domain name, you need to purchase a domain name from a domain name registrar and then set up DNS records for it. This setup assumes that DigitalOcean is used to manage DNS records (both for the GKE and DOKS setup). For this you need to add your domain to your DigitalOcean account and update your domain’s NS records to point to DigitalOcean’s name servers. Later, all necessary A-records are automatically created via ExternalDNS to point your domain to the load balancer.

You may need to manually delete DNS records when switching between DOKS and GKE clusters, as ExternalDNS sometimes does not update records correctly

Create Personal Access Token in DigitalOcean

You need to create a Personal Access Token in DigitalOcean. Terraform (and other tools like ExternalDNS) will use your DigitalOcean Personal Access Token to communicate with the DigitalOcean API and manage resources in your account. Don’t share this key with others, and keep it out of scripts and version control! Export your DigitalOcean Personal Access Token to an environment variable called DO_PAT. This will make using it in subsequent commands easier and keep it separate from your code:

export DO_PAT="YOUR_PERSONAL_ACCESS_TOKEN"

I would recommend adding this line to your shell configuration files to avoid having to do this step again in the future.

Setup GKE

After you've installed the gcloud SDK, initialize it by running the following command to authorize the SDK to access GCP using your user account credentials and add the SDK to your PATH:

gcloud init

Finally, add your account to the Application Default Credentials (ADC). This will allow Terraform to access these credentials to provision resources on GCloud.

gcloud auth application-default login

Setup Terraform

You may also want to enable logging to Standard Output (STDOUT), so you can see what Terraform is trying to do. Do that by running the following command, or again, directly adding it to your shell configuration files.

export TF_LOG=1

Now make a copy of terraform.tfvars.example, rename it to terraform.tfvars and define all variables within.

To initialize Terraform, run the following command once:

terraform init

Usage

Provision resources:

terraform apply -var "do_token=${DO_PAT}" -auto-approve

You can access the cluster directly using kubectl, since Terraform automatically adds the credentials for your cluster to your local kubeconfig.

Destroy resources:

terraform destroy -var "do_token=${DO_PAT}" -auto-approve

Important (only DOKS): Load balancers and block storage will be destroyed through a destroy-time provisioner, using the DigitalOcean API, as they are not directly managed by Terraform and are also not automatically destroyed when the cluster is destroyed (as is the case with GKE). All LB and Block Storage resources in your account will be destroyed! If this is not desired, then deactivate the destroy-time provisioner.

Access monitoring applications

Grafana, Prometheus and the Alertmanager are respectively accessible on the subdomains grafana.*, prometheus.* and alertmanager.* of your domain. It may take a few seconds till all pods are started.