K8s Lab Environment

Multi-cluster Kubernetes laboratory environment running Talos Linux with separate DB and App clusters.

Architecture

DB Cluster (talos cluster)

• Purpose: Database and stateful workloads
• Nodes: 3x control plane nodes
  • talos-0lj-bma (10.0.0.104)
  • talos-6qj-6v8 (10.0.0.103)
  • talos-mf1-tt5 (10.0.0.102)
• Infrastructure:
  • Cilium CNI with L2 announcement
  • Democratic CSI (iSCSI storage provisioning)
  • External Secrets Operator (Azure Key Vault integration)
  • Flux GitOps

App Cluster

• Purpose: Application workloads
• Nodes: 1x control plane node (HA ready)
  • app-cp1 (10.0.0.115)
• Future: VPS worker node for HA
• Infrastructure:
  • Cilium CNI
  • Ready for application deployments

Directory Structure

k8s-lab/
├── app-cluster/              # App cluster Talos configs
│   ├── _out/                 # Generated configs (gitignored)
│   ├── controlplane-network-patch.yaml
│   └── app-kubeconfig        # Kubernetes access (gitignored)
├── db-cluster/               # DB cluster Talos configs
│   └── (machine configs and patches)
├── gitops/                   # Flux GitOps manifests
│   ├── apps/                 # Application definitions
│   ├── infrastructure/       # Infrastructure components
│   └── clusters/             # Cluster-specific configs
├── docs/                     # Documentation
├── archive/                  # Archived/old files (gitignored)
└── backups/                  # Configuration backups (gitignored)

Quick Start

Switch Between Clusters

# Use DB cluster (default)
kubectl config use-context "admin@talos cluster"

# Use App cluster
kubectl config use-context "admin@app-cluster"

# View all contexts
kubectl config get-contexts

Access Clusters

DB Cluster:

kubectl --context="admin@talos cluster" get nodes
talosctl --talosconfig db-cluster/talosconfig-working version

App Cluster:

kubectl --context="admin@app-cluster" get nodes
talosctl --talosconfig app-cluster/_out/talosconfig version

Current Workloads

DB Cluster

• Flux System: GitOps controller managing deployments
• Cilium: CNI with L2 LoadBalancer support
• Democratic CSI: iSCSI storage for persistent volumes
• External Secrets: Azure Key Vault integration
• Linkding: Bookmark manager
• CoreDNS: Cluster DNS
• Test workloads: nginx LoadBalancer test

App Cluster

• Cilium: CNI configured
• CoreDNS: Cluster DNS
• Ready for applications

Management

Flux GitOps

The DB cluster uses Flux for continuous delivery:

# Check Flux status
flux check

# Reconcile manually
flux reconcile kustomization flux-system
flux reconcile kustomization apps

# View deployed resources
kubectl get kustomizations -n flux-system
kubectl get helmreleases -A

Storage (DB Cluster)

Democratic CSI provides iSCSI-based persistent storage:

# List storage classes
kubectl get sc

# List PVCs
kubectl get pvc -A

# List PVs
kubectl get pv

Adding Nodes

App Cluster Worker (Future)

1. Generate worker config
2. Apply to VPS node
3. Join to app cluster for HA

Networking

• DB Cluster: 10.0.0.102-104
• App Cluster: 10.0.0.115
• Gateway: 10.0.0.1
• DNS: 10.0.0.1, 8.8.8.8

Secrets Management

• SOPS encryption with age key
• Azure Key Vault via External Secrets Operator (DB cluster)
• Secrets stored encrypted in git (*.enc.yaml)

Backup Strategy

• Etcd backups via Talos
• Configuration backups in backups/ directory
• GitOps ensures declarative restore capability

Maintenance

Update Talos

# DB cluster
talosctl --talosconfig db-cluster/talosconfig-working \
  --nodes 10.0.0.102,10.0.0.103,10.0.0.104 \
  upgrade --image ghcr.io/siderolabs/installer:v1.11.3

# App cluster
talosctl --talosconfig app-cluster/_out/talosconfig \
  --nodes 10.0.0.115 \
  upgrade --image ghcr.io/siderolabs/installer:v1.11.3

Update Kubernetes

talosctl --talosconfig <config> upgrade-k8s --to 1.34.1

Troubleshooting

Check Node Health

# Talos
talosctl --talosconfig <config> --nodes <ip> health

# Kubernetes
kubectl get nodes
kubectl get pods -A

View Logs

# Talos service logs
talosctl --talosconfig <config> --nodes <ip> logs kubelet

# Kubernetes pod logs
kubectl logs -n <namespace> <pod-name>

Network Issues

# Check Cilium status
cilium status

# View Cilium connectivity
cilium connectivity test

Notes

• Both clusters run Talos Linux v1.11.3 with Kubernetes v1.34.1
• Cilium handles CNI for both clusters
• DB cluster is production-like with 3-node HA
• App cluster is single-node, ready for HA expansion
• Flux manages DB cluster GitOps workflow