Hands-on Kubernetes labs focused on Cloud / DevOps / Platform Engineering, with an emphasis on disaster recovery, AI/ML workloads, autoscaling, GitOps, infrastructure as code, and multi-cloud architecture.
These labs are designed as a practical learning environment rather than a collection of copy-paste tutorials.
The goal is to build the infrastructure, encounter realistic failure modes, measure the results, and understand why the platform behaves the way it does.
Focus: Multi-cluster Kubernetes DR and GitOps-based failover.
Key areas:
- Kubernetes multi-cluster architecture
- Primary / secondary clusters
- ArgoCD
- App of Apps
- ApplicationSets
- Helm
- Terraform / Terragrunt
- application recovery
- data recovery considerations
- RTO measurement
- failure injection
- operational runbooks
The lab ultimately answers:
How quickly and reliably can a Kubernetes workload recover when the primary cluster fails?
Primary measurement: RTO
Detailed requirements:
lab-dr-fundamentals.md
Focus: Running and scaling AI/ML-style workloads on Kubernetes.
Key areas:
- KEDA
- Karpenter
- GPU-backed workloads
- Kubernetes DRA
- workload autoscaling
- node autoscaling
- cold starts
- GitOps
- ArgoCD
- CI/CD
- container image lifecycle
- inference workloads
The lab ultimately answers:
What happens when an AI workload scales from zero, and how do Kubernetes workload and node autoscaling interact?
Primary measurement: cold-start latency
Detailed requirements:
lab-aiml-kubernetes.md
.
├── AGENTS.md
│
├── README.md
├── lab-dr-fundamentals.md
├── lab-aiml-kubernetes.md
├── lab-addendum-kargo-cost-tiers.md
│
├── infra/
│ ├── terragrunt.hcl
│ ├── _modules/
│ ├── local/
│ ├── aws/
│ └── azure/
│
├── lab-01-dr/
└── lab-02-aiml/
| File | Purpose |
|---|---|
AGENTS.md |
Instructions for coding agents working on the repository |
lab-dr-fundamentals.md |
Source of truth for Lab 01 |
lab-aiml-kubernetes.md |
Source of truth for Lab 02 |
lab-addendum-kargo-cost-tiers.md |
Reference for infrastructure tiers and future Kargo work |
infra/ |
Cloud/local infrastructure |
lab-01-dr/ |
Lab 01 implementation |
lab-02-aiml/ |
Lab 02 implementation |
The repository intentionally separates infrastructure provisioning from workload deployment.
Git Repository
│
┌───────────┴───────────┐
│ │
Infrastructure GitOps
Terraform/TG ArgoCD
│ │
▼ ▼
Cluster / Network Kubernetes Workloads
IAM / Node Roles Apps / Operators / Config
Karpenter IAM NodePool / EC2NodeClass
infra/
Responsible for creating infrastructure:
- VPCs
- EKS
- AKS
- IAM
- node roles
- Karpenter prerequisites
- other cloud resources
Managed with:
- Terraform
- Terragrunt
lab-XX/gitops/
Responsible for what runs inside Kubernetes:
- applications
- operators
- Helm releases
- application configuration
- Karpenter
NodePool - Karpenter
EC2NodeClass
Managed with:
- ArgoCD
- Git
This boundary is intentional and should not be collapsed for convenience.
The default environment is Tier 0 / local.
The labs should be possible to explore locally before introducing cloud infrastructure.
Local Kubernetes clusters are bootstrapped with k3d rather than Terraform.
Cloud infrastructure is introduced only when it provides meaningful learning value.
Available cloud targets include:
- AWS / EKS
- Azure / AKS
The infrastructure tier should be selected according to the budget guidance in:
lab-addendum-kargo-cost-tiers.md
Both labs use an ArgoCD App of Apps pattern.
The general model is:
Root Application
│
▼
Child Applications
│
┌─────┼─────┐
▼ ▼ ▼
API Worker Database
This is intentional.
The labs are designed to explore:
- application composition
- multi-cluster deployment
- environment-specific configuration
- ApplicationSets
- Git as the desired-state source
Neither lab should be reduced to a single ArgoCD Application pointing directly at a directory of manifests.
The AI/ML lab also demonstrates a simplified GitOps delivery pipeline:
Developer
│
▼
Push code
│
▼
CI
│
├── Build image
├── Push image
└── Update image tag in Git
│
▼
ArgoCD
│
▼
Kubernetes
│
▼
KEDA
│
▼
Workload scaling
│
▼
Karpenter
│
▼
Node provisioning
The CI pipeline is deliberately minimal.
The purpose is to understand the relationship between:
CI → Git → ArgoCD → Kubernetes → KEDA → Karpenter
rather than to build a sophisticated CI platform.
These are build-and-break labs.
The intended workflow is:
Understand
↓
Scaffold
↓
Implement
↓
Break
↓
Observe
↓
Diagnose
↓
Fix
↓
Measure
↓
Document
The agent should not automatically complete every exercise.
When working through a stage, the implementation should contain enough scaffolding to begin while leaving meaningful decisions and debugging work for the learner.
See AGENTS.md for the detailed agent workflow.
The labs are intentionally biased toward platform-engineering topics that are useful in real cloud environments.
- multi-cluster architecture
- scheduling
- workload scaling
- node scaling
- failure recovery
- cluster operations
- ArgoCD
- App of Apps
- ApplicationSets
- environment configuration
- Git-driven deployment
- Terraform
- Terragrunt
- reusable modules
- dependency graphs
- multi-environment infrastructure
- KEDA
- Karpenter
- GPU workloads
- DRA
- inference workloads
- cold-start behavior
- disaster recovery
- failure injection
- RTO
- recovery procedures
- operational runbooks
Kargo may later be introduced as an extension to Lab 01.
It is not a separate lab.
The goal is not simply:
"The deployment works."
Each lab should produce a technical write-up containing:
- architecture diagram
- implementation overview
- important design decisions
- measured results
- failure scenarios
- debugging process
- fixes
- lessons learned
- limitations
- improvements for a real cloud environment
Each lab should document at least three things that broke, including how they were diagnosed and fixed.
Measure:
Recovery Time Objective (RTO)
Measure:
Cold-start latency
First, read:
AGENTS.md
lab-dr-fundamentals.md
lab-aiml-kubernetes.md
lab-addendum-kargo-cost-tiers.md
Then choose a lab.
Let's start Lab 01.
Let's start Lab 02.
Walk me through Stage 1.
I'm stuck on X.
Show me the answer for X.
I want to add Kargo.
My budget is $X.
The finished repository should demonstrate more than familiarity with Kubernetes tooling.
It should demonstrate the ability to reason about:
- reliability
- scalability
- infrastructure
- GitOps
- cloud architecture
- cost
- failure recovery
- operational trade-offs
The final result should be something that can be discussed as a real Platform Engineering / DevOps portfolio project, backed by actual experiments and measurements rather than only configuration files.