Skip to content

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

31 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Infrastructure

Single-node Kubernetes cluster running Talos Linux on GCP, managed with Terraform and Nix.

Philosophy

The goal of this project is confidence. I want to be able to:

  • Take it down anytime to save costs, knowing I can bring it back
  • Redeploy from scratch without fear, because everything is codified
  • Extend over the coming years, adding services without fragile manual steps

Everything is infrastructure as code. No SSH, no manual configuration, no "I forgot how I set that up." If it's not in this repo, it doesn't exist.

What This Is

This repository provisions a minimal, immutable Kubernetes cluster:

  • Talos Linux: Immutable, API-driven OS purpose-built for Kubernetes. No SSH, no shell, no package manager. Managed entirely via talosctl.
  • Single control plane node: One e2-medium (2 vCPU, 4GB RAM) in northamerica-northeast1-a (Montreal).
  • Persistent storage: 20GB SSD attached for stateful workloads (PostgreSQL).
  • Nix flake: Reproducible development environment with all required tools.

Directory Structure

infrastructure/
├── .sops.yaml                   # SOPS config with age public key
├── flake.nix                    # Nix devshell (talosctl, kubectl, terraform, sops, age)
├── flake.lock                   # Locked dependencies (generated by nix)
├── terraform/
│   ├── persistent/              # Long-lived resources (~$4/month)
│   │   ├── main.tf              # Google provider
│   │   ├── variables.tf         # Project ID, region, zone
│   │   ├── storage.tf           # Bucket (cafe-coding-fleet-artifacts) & Data disk (talos-data)
│   │   └── outputs.tf           # Disk name
│   └── compute/                 # Ephemeral resources (~$46/month)
│       ├── main.tf              # Google + Cloudflare + SOPS providers
│       ├── variables.tf         # Machine type, etc.
│       ├── data.tf              # References persistent disk
│       ├── instance.tf          # GCE instance running Talos
│       ├── networking.tf        # Firewall rules (50000, 6443, 80, 443)
│       ├── cloudflare.tf        # DNS A records
│       ├── outputs.tf           # External IP for talosctl/kubectl
│       └── secrets.enc.yaml     # Encrypted Cloudflare credentials
├── talos/
│   ├── .gitkeep                 # Placeholder
│   ├── controlplane.yaml        # Generated - Talos machine config (DO NOT COMMIT)
│   ├── worker.yaml              # Generated - Not used in single-node setup
│   └── talosconfig              # Generated - talosctl client config (DO NOT COMMIT)
├── apps/
│   ├── nginx-ingress/
│   │   └── deploy.yaml          # Ingress controller manifest
│   └── test-app/
│       ├── deployment.yaml      # Test nginx deployment
│       ├── service.yaml         # ClusterIP service
│       └── ingress.yaml         # Ingress for test.justinmcintyre.com
├── scripts/
│   ├── cluster-up.sh            # Start GCP cluster
│   ├── cluster-down.sh          # Stop GCP cluster (exports certs, saves costs)
│   ├── local-cluster.sh         # Start/stop local Docker cluster
│   ├── deploy-apps.sh           # Deploy all applications
│   ├── deploy-ingress.sh        # Deploy nginx ingress controller
│   ├── deploy-cert-manager.sh   # Deploy cert-manager
│   ├── deploy-postgres.sh       # Deploy PostgreSQL
│   ├── deploy-twenty.sh         # Deploy Twenty CRM
│   ├── deploy-test-apps.sh      # Deploy test applications
│   ├── monitor-status.sh        # Monitor cluster health and certs
│   ├── dashboard.sh             # Open Talos dashboard
│   ├── db-connect.sh            # Connect to PostgreSQL
│   ├── disk-usage.sh            # Show disk usage on node
│   ├── list-resources.sh        # List all GCP resources
│   └── internal/                # Called by other scripts
│       ├── bootstrap-gcp.sh     # Called by cluster-up.sh
│       ├── setup-gcp-image.sh   # Called by cluster-up.sh
│       ├── test-local.sh        # Called by local-cluster.sh
│       ├── export-certs.sh      # Called by cluster-down.sh
│       └── require-env.sh       # Ensures scripts run in nix shell
├── docs/
│   ├── kubernetes-basics.md     # Pods, nodes, taints explained for single-node setup
│   ├── gcp-costs.md             # Cost breakdown with pricing sources
│   ├── sops-secrets.md          # SOPS/age key management guide
│   └── cloudflare-setup.md      # API token + zone ID instructions
└── README.md                    # This file

Prerequisites

  1. Nix with flakes enabled

    # If not installed, see https://nixos.org/download
    # Enable flakes in ~/.config/nix/nix.conf:
    experimental-features = nix-command flakes
  2. Docker (for local testing only)

    # Install Docker (distro-specific, see https://docs.docker.com/engine/install/)
    
    # Add yourself to docker group so sudo can access it
    sudo usermod -aG docker $USER
    
    # Log out and back in for group change to take effect

All other tools (talosctl, kubectl, terraform, gcloud, sops, age) are provided by the Nix flake.

Quick Start

Running Commands

All commands can be run via nix run without entering a shell:

# GCP cluster
nix run .#cluster-up
nix run .#cluster-down

# Local Docker cluster
nix run .#local-cluster -- up
nix run .#local-cluster -- down

# Deploy applications
nix run .#deploy-apps

# Utilities
nix run .#dashboard
nix run .#db-connect
nix run .#list-resources

For interactive work, enter the dev shell:

nix develop

This gives you: talosctl, kubectl, terraform, gcloud, sops, age

Authenticate with GCP (first time only)

nix develop -c gcloud auth application-default login

Option A: Local Cluster (Docker)

For testing without GCP costs:

nix run .#local-cluster -- up

# Verify (inside nix develop)
nix develop
talosctl --nodes 10.5.0.2 health
kubectl get nodes

# Tear down
nix run .#local-cluster -- down

Option B: GCP Cluster (Production)

# Create cluster (first run creates persistent disk, bucket, and Talos image)
nix run .#cluster-up

# Verify (inside nix develop)
nix develop
kubectl get nodes
kubectl get pods -A

The bootstrap script:

  • Gets the node IP from terraform output
  • Generates Talos machine config in talos/
  • Applies config to the node
  • Bootstraps etcd
  • Waits for cluster health
  • Merges kubeconfig into ~/.kube/config

Terraform Resources

Terraform is split into two projects for cost savings (see "Shut Down to Save Costs" below).

Persistent (terraform/persistent/) - ~$4/month

Resource Name Purpose
google_compute_disk talos-data 20GB SSD for PostgreSQL data
google_storage_bucket {project}-artifacts General artifacts (future use)

Compute (terraform/compute/) - ~$46/month

Resource Name Purpose
google_compute_instance talos-controlplane The Kubernetes node
google_compute_firewall talos-api Port 50000 - talosctl access
google_compute_firewall kubernetes-api Port 6443 - kubectl access
google_compute_firewall http-https Ports 80/443 - web traffic
cloudflare_record test, test2, crm DNS A records

Firewall Ports

Port Protocol Purpose
50000 TCP Talos API (talosctl commands)
6443 TCP Kubernetes API (kubectl commands)
80 TCP HTTP (for ingress)
443 TCP HTTPS (for ingress)

Common Operations

Redeploy Applications

If you've made changes to app manifests and want to redeploy without full bootstrap:

# Redeploy everything
./scripts/deploy-apps.sh

# Or redeploy individual components
./scripts/deploy-ingress.sh
./scripts/deploy-cert-manager.sh
./scripts/deploy-postgres.sh
./scripts/deploy-twenty.sh
./scripts/deploy-test-apps.sh

All deploy scripts are idempotent - running them on an already-deployed app applies any changes.

Check Cluster Health

# Talos-level health (etcd, kubelet, etc.)
talosctl --nodes <IP> --talosconfig talos/talosconfig health

# Kubernetes-level health
kubectl get nodes
kubectl get pods -A

View Talos Logs

# Kernel messages
talosctl --nodes <IP> --talosconfig talos/talosconfig dmesg

# Service logs
talosctl --nodes <IP> --talosconfig talos/talosconfig logs kubelet
talosctl --nodes <IP> --talosconfig talos/talosconfig logs etcd

Upgrade Talos

# Check current version
talosctl --nodes <IP> --talosconfig talos/talosconfig version

# Upgrade (replace with actual version)
talosctl --nodes <IP> --talosconfig talos/talosconfig upgrade \
  --image ghcr.io/siderolabs/installer:v1.7.0

Shut Down to Save Costs

Terraform is split into persistent/ and compute/ projects specifically for this use case. When not using the cluster, destroy compute resources to pay only for disk storage:

State Monthly Cost
Running 24/7 ~$70 CAD
Running 8hrs/day weekdays ~$20 CAD
Shut down ~$4 CAD (disk only)

Part-time math: Compute is $66 CAD for 730 hrs ($0.09/hr). 8 hrs × 5 days × 4.33 weeks = 173 hrs × $0.09 = ~$16 + $4 disk = ~$20 CAD.

# Destroy VM, firewall, DNS (keeps persistent disk)
./scripts/cluster-down.sh

# Bring it back up
./scripts/cluster-up.sh

Example crontab (up at 8am, down at 6pm weekdays):

0 8 * * 1-5  cd /home/justinm/infrastructure && ./scripts/cluster-up.sh
0 18 * * 1-5 cd /home/justinm/infrastructure && ./scripts/cluster-down.sh

Note: You get a new IP each time. The bootstrap script handles this.

Destroy Everything

cd terraform
terraform destroy

Warning: This deletes the VM AND the persistent disk. Data is lost.

Talos Image Setup

GCP requires VM images to be in your project. The setup-gcp-image.sh script:

  1. Creates a temporary GCS bucket
  2. Downloads the Talos image from factory.talos.dev (~900MB)
  3. Uploads it to the temporary bucket
  4. Creates a GCP Compute Image from the upload
  5. Deletes the temporary bucket

This is called automatically by cluster-up.sh if the image doesn't exist. To upgrade Talos, update the version in the script and delete the old image.

The image uses the default Talos schematic (vanilla, no extensions). For custom images with extensions, generate a schematic ID at https://factory.talos.dev/ and update the script.

Persistent Disk

The data SSD (talos-data, 20GB by default) is attached to the instance at /dev/sdb (device name: data).

It is not formatted or mounted by default. This happens when PostgreSQL is deployed (see milestone 3).

To verify the disk is attached:

talosctl --nodes <IP> --talosconfig talos/talosconfig get disks

Files That Should NOT Be Committed

These contain secrets and are in .gitignore:

  • talos/*.yaml - Machine configs with cluster secrets
  • talos/talosconfig - Client credentials
  • terraform/*.tfstate - May contain sensitive outputs
  • *.agekey - SOPS encryption keys

Troubleshooting

"connection refused" on port 50000

The node hasn't finished booting or config wasn't applied:

# Check if node is reachable
ping <IP>

# Re-apply config
talosctl apply-config --insecure --nodes <IP> --file talos/controlplane.yaml

Bootstrap fails

Usually means etcd didn't start properly:

# Check etcd logs
talosctl --nodes <IP> --talosconfig talos/talosconfig logs etcd

# If stuck, you may need to reset and re-apply config
talosctl --nodes <IP> --talosconfig talos/talosconfig reset --graceful=false
talosctl apply-config --insecure --nodes <IP> --file talos/controlplane.yaml
talosctl bootstrap --nodes <IP> --endpoints <IP> --talosconfig talos/talosconfig

kubectl times out

The Kubernetes API isn't ready yet:

# Check if API server is running
talosctl --nodes <IP> --talosconfig talos/talosconfig services

# Wait for health check to pass
talosctl --nodes <IP> --talosconfig talos/talosconfig health

Local Docker cluster won't start

Ensure Docker is running and you have permission:

# Check Docker is running and accessible
docker info

# If permission denied, add yourself to docker group
sudo usermod -aG docker $USER
# Then log out and back in

# If Docker not found, install it first
# See https://docs.docker.com/engine/install/

Secrets Management (SOPS)

Secrets are encrypted with SOPS using age keys. The encrypted file (terraform/secrets.enc.yaml) is safe to commit.

First-Time Setup

  1. Generate an age key:

    mkdir -p ~/.config/sops/age
    age-keygen -o ~/.config/sops/age/keys.txt
  2. Copy the public key (starts with age1...) and add it to .sops.yaml

  3. Create and encrypt secrets:

    sops terraform/secrets.enc.yaml

    Add your Cloudflare credentials:

    cloudflare_api_token: "your-token"
    cloudflare_zone_id: "your-zone-id"

On a New Machine

Copy your age private key to ~/.config/sops/age/keys.txt. Without this, terraform cannot decrypt secrets.

See docs/sops-secrets.md for the full guide.

Further Reading

Architecture Decisions

  1. Single node: Simplicity over HA. This is a personal/small project.
  2. Talos Linux: Immutable, secure, minimal attack surface. No SSH simplifies security.
  3. Montreal region: Low latency from Eastern Canada, data residency in Canada.
  4. e2-medium: Cheapest viable size (2 vCPU needed for Kubernetes).
  5. Separate data disk: Survives instance recreation, can snapshot independently.
  6. Nix flake: Reproducible tooling, no "works on my machine" issues.

Cost Estimate

~$70/month CAD. See docs/gcp-costs.md for details and pricing sources.

Planned

  1. soft-serve - Git server for code backup

    • SSH-based, minimal overhead
    • Backup in case GitHub is down
    • SQLite storage on persistent disk
  2. Filebrowser - Simple file management

    • Web-based file browser
    • Upload/download files
    • For Hermes agent file access

About

No description, website, or topics provided.

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages