k3s cluster backed by Flux (archived due to colo costs)

Highly opinionated template for deploying a single k3s cluster with Ansible and Terraform backed by Flux and SOPS.

The purpose here is to showcase how you can deploy an entire Kubernetes cluster and show it off to the world using the GitOps tool Flux. When completed, your Git repository will be driving the state of your Kubernetes cluster. In addition with the help of the Ansible, Terraform and Flux SOPS integrations you'll be able to commit Age encrypted secrets to your public repo.

Overview

• Introduction
• Prerequisites
• Repository structure
• Lets go!
• Post installation
• Troubleshooting
• What's next
• Thanks

👋 Introduction

The following components will be installed in your k3s cluster by default. Most are only included to get a minimum viable cluster up and running.

• flux - GitOps operator for managing Kubernetes clusters from a Git repository
• metallb - Load balancer for Kubernetes services
• cert-manager - Operator to request SSL certificates and store them as Kubernetes resources
• external-dns - Operator to publish DNS records to Cloudflare (and other providers) based on Kubernetes ingresses
• ingress-nginx - Kubernetes ingress controller used for a HTTP reverse proxy of Kubernetes ingresses
• local-path-provisioner - provision persistent local storage with Kubernetes

Additional applications include hajimari, error-pages, echo-server, system-upgrade-controller, reloader, and kured

For provisioning the following tools will be used:

• Fedora 36 Server - Universal operating system that supports running all kinds of home related workloads in Kubernetes and has a faster release cycle
• Ansible - Provision Fedora Server and install k3s

📝 Prerequisites

Note: This template has not been tested on cloud providers like AWS EC2, Hetzner, Scaleway etc... Those cloud offerings probably have a better way of provsioning a Kubernetes cluster and it's advisable to use those instead of the Ansible playbooks included here. This repository can still be tweaked for the GitOps/Flux portion if there's a cluster working in one those environments.

📚 Reading material

• Organizing Cluster Access Using kubeconfig Files

💻 Systems

• One or more nodes with a fresh install of Fedora Server 36.
  • These nodes can be ARM64/AMD64 bare metal or VMs.
  • An odd number of control plane nodes, greater than or equal to 3 is required if deploying more than one control plane node.
• A Cloudflare account with a domain, this will be managed by Terraform and external-dns. You can register new domains directly thru Cloudflare.
• Some experience in debugging problems and a positive attitude ;)

📍 It is recommended to have 3 master nodes for a highly available control plane.

🔧 Workstation Tools

1. Install the most recent versions of the following CLI tools on your workstation, if you are using Homebrew on MacOS or Linux skip to steps 3 and 4.

   • Required: age, ansible, flux, go-task, ipcalc, jq, kubectl, pre-commit, sops, terraform, yq v4

   • Recommended: direnv, helm, kustomize, prettier, stern, yamllint

2. This guide heavily relies on go-task as a framework for setting things up. It is advised to learn and understand the commands it is running under the hood.

3. Install go-task via Brew

   brew install go-task/tap/go-task

4. Install workstation dependencies via Brew

   task init

⚠️ pre-commit

It is advisable to install pre-commit and the pre-commit hooks that come with this repository. sops-pre-commit will check to make sure you are not committing non-encrypted Kubernetes secrets to your repository.

1. Enable Pre-Commit

   task precommit:init

2. Update Pre-Commit, though it will occasionally make mistakes, so verify its results.

   task precommit:update

📂 Repository structure

The Git repository contains the following directories under cluster and are ordered below by how Flux will apply them.

📁 cluster      # k8s cluster defined as code
├─📁 flux       # flux, gitops operator, loaded before everything
├─📁 crds       # custom resources, loaded before 📁 core and 📁 apps
├─📁 charts     # helm repos, loaded before 📁 core and 📁 apps
├─📁 config     # cluster config, loaded before 📁 core and 📁 apps
├─📁 core       # crucial apps, namespaced dir tree, loaded before 📁 apps
└─📁 apps       # regular apps, namespaced dir tree, loaded last

🤝 Thanks

Big shout out to all the authors and contributors to the projects that we are using in this repository.

Community member @Whazor created this website as a creative way to search Helm Releases across GitHub. You may use it as a means to get ideas on how to configure an applications' Helm values.