Note

This is a mirrored repo.

󱁢 Terraform Repo for Talos Kubernetes Cluster

Important

This repo is built for my own environment so please review all configurations to verify compatibility!

This repo provides all terraform automation for deploying the Talos Kubernetes cluster:

• Builds three master VMs across three Proxmox nodes.
• Builds three worker VMs across three Proxmox nodes.
• Builds one GPU VM on a fourth Proxmox node.
• Deploys the Talos cluster and stands up Kubernetes.
  • Uses several build extentions for Longhorn (all nodes) and NVIDIA gpu (on the GPU node).
  • Uses Cilium as the CNI.
  • Uses Argo-cd for git-ops.

Tip

This repo is part of my IaC automation series. If you are building this in mind please follow my repo's in the order below.

1.  terraform-iso-get
2.  packer
3.  terraform-core
4.  ansible
5.  you are here terraform-talos
6.  k8s-apps

Build Your Secrets File

Keeping in best practice, this repo does not contain any sensitive information. You will need to create a directory outside of this git repo on a properly encrypted disk/usb to save the secrets file. Below is the template needed for the file which needs to be named tf-secrets.tfvars.

# This is a sensitive file. Do not share!
# All variables for all Terraform files.

pve_host      = "pve1.example.com:8006"
pve_api_token = "username@pam!build"
pve_ssh_user  = "username"
pve_ssh_pass  = "MY-PASSWORD"
sshuser       = "username"
sshpass       = "MY-PASSWORD"
sshkey        = ["ecdsa-sha2-nistp521 MY-SSH-KEY=="]
#base64 encoded: echo -n 'password' | base64
vault_token   = "MY-VAULT-TOKEN=="
#base64 vault-ica.crt
vault_ica    = "MY-PUBLIC-CERT=="

Adjust Variables File

Adjust the variables.tf file to match your environemnt. Make sure you choose an INTERNAL ONLY domain that you own!

Build the Repo

Once the core services are up and running we are now ready to deploy the Kubernetes cluster. In this stage we stand up the cluster and Argo-cd will build out you apps to be ready for use.

1. Go into Terraform directory: cd ~/path-to/terraform-talos
2. Initiate the workspace: terraform init
3. Validate the build: terraform plan -var-file={path/to/secrets/}tf-secrets.tfvars
4. Run the build: terraform apply -var-file={path/to/secrets/}tf-secrets.tfvars -parallelism=2

Verify Installation

Several commands below are useful for ensuring the status of the install.

• Talos status commands:
  • Monitor Talos cluster: TERM=xterm talosctl dashboard --nodes <master-node-ip>
  • Check node disk: talosctl --nodes <node-ip> get disks
  • Check node mounts: talosctl --nodes <node-ip> get mounts
  • Check Longhorn volumes: talosctl --nodes <node-ip> get volumestatus r-longhorn
  • Ensure disk are accepted by longhorn:
    • kubectl -n longhorn-system get pods -l longhorn.io/component=instance-manager
    • kubectl get nodes.longhorn.io -n longhorn-system
    • kubectl -n longhorn-system get nodes.longhorn.io <k8s-node> \ -o jsonpath="{.status.diskStatus['longhorn-v2'].conditions}"
• Cilium status commands:
  • General status: cilium status
  • Detailed status: kubectl -n kube-system exec -it <cilium-pod> -- cilium-dbg status --verbose