Kata Containers on EKS with Karpenter

Setup guide for running Kata Containers on an existing EKS cluster using Karpenter for on-demand bare metal node provisioning.

Overview

Pod with runtimeClassName: kata-qemu / kata-fc
  -> RuntimeClass nodeSelector targets katacontainers.io/kata-runtime=true
  -> Karpenter provisions bare metal node from kata-metal NodePool
  -> kata-deploy DaemonSet installs Kata binaries + configures containerd
  -> Pod runs inside a lightweight VM (QEMU or Firecracker) with its own kernel
  -> When no Kata pods remain, node scales down after 60s

Prerequisites

• EKS cluster with Karpenter installed
• kubectl configured with cluster access
• helm CLI installed
• Bare metal EC2 instances available in your region (e.g. c6g.metal, c7g.metal, m7g.metal)

Why Bare Metal?

Kata Containers runs workloads inside lightweight VMs using KVM. Standard EC2 instances don't support nested virtualization -- bare metal instances provide direct hardware access required for KVM.

Repo Structure

.
├── karpenter/
│   └── kata.yaml                   # EC2NodeClass + NodePool for bare metal nodes
├── helm/
│   └── kata-deploy-values.yaml     # Helm values for kata-deploy chart
├── tests/
│   ├── test-kata-pod.yaml          # One-shot test pod (QEMU)
│   ├── test-kata-nginx.yaml        # Nginx deployment (QEMU)
│   ├── test-kata-fc-pod.yaml       # One-shot test pod (Firecracker)
│   └── test-kata-fc-nginx.yaml     # Nginx deployment (Firecracker)
├── scripts/
│   ├── deploy.sh                   # Full deployment script
│   └── teardown.sh                 # Full teardown script
└── README.md

Configuration

The deploy script and Karpenter template require two environment variables:

Variable	Description	Example
CLUSTER_NAME	Your EKS cluster name (used for subnet/SG/tag discovery via karpenter.sh/discovery)	my-eks-cluster
NODE_ROLE	IAM role name for Karpenter-managed nodes	KarpenterNodeRole-my-eks-cluster

Quick Start

export CLUSTER_NAME=my-eks-cluster
export NODE_ROLE=KarpenterNodeRole-my-eks-cluster

# Deploy everything
./scripts/deploy.sh

The deploy script will:

1. Validate that CLUSTER_NAME and NODE_ROLE are set
2. Render karpenter/kata.yaml with your values and apply the EC2NodeClass + NodePool
3. Clone the kata-containers Helm chart (sparse checkout, cached in /tmp/kata-repo)
4. Install kata-deploy via Helm
5. Wait for kata-qemu and kata-fc RuntimeClasses to appear

For step-by-step instructions, see Manual Setup below.

Manual Setup

1. Create Karpenter EC2NodeClass and NodePool

The kata.yaml is a template containing both the EC2NodeClass and the kata-metal NodePool. It uses ${CLUSTER_NAME} and ${NODE_ROLE} placeholders that must be substituted before applying.

• CLUSTER_NAME -- your EKS cluster name (used for subnet/SG/tag discovery via karpenter.sh/discovery)
• NODE_ROLE -- the IAM role name for Karpenter-managed nodes (e.g. KarpenterNodeRole-<cluster-name>)

export CLUSTER_NAME=my-eks-cluster
export NODE_ROLE=KarpenterNodeRole-my-eks-cluster
envsubst '${CLUSTER_NAME} ${NODE_ROLE}' < karpenter/kata.yaml | kubectl apply -f -

Key config:

• karpenter.k8s.aws/instance-size: metal -- bare metal only
• karpenter.k8s.aws/instance-category: [c, m, r] -- compute/memory/general families
• Taint kata-runtime=true:NoSchedule -- only Kata workloads schedule here
• Label katacontainers.io/kata-runtime: "true" -- used by kata-deploy DaemonSet and RuntimeClass nodeSelector
• consolidationPolicy: WhenEmpty, consolidateAfter: 60s -- scale to zero when idle

2. Install kata-deploy via Helm

Clone the Kata Containers repo (sparse checkout for the chart only):

git clone --depth 1 --filter=blob:none --sparse \
  https://github.com/kata-containers/kata-containers.git /tmp/kata-repo
cd /tmp/kata-repo
git sparse-checkout set tools/packaging/kata-deploy/helm-chart

Build chart dependencies and install:

cd tools/packaging/kata-deploy/helm-chart/kata-deploy
helm dependency build
helm install kata-deploy . \
  -n kube-system \
  -f /path/to/this-repo/helm/kata-deploy-values.yaml

The values file configures:

• nodeSelector so kata-deploy only runs on bare metal nodes with katacontainers.io/kata-runtime: "true"
• tolerations for the kata-runtime taint
• Both qemu and fc (Firecracker) shims enabled
• Firecracker uses the devmapper snapshotter
• NFD (node-feature-discovery) disabled since node selection is handled by Karpenter

3. Verify

# RuntimeClasses should exist
kubectl get runtimeclass kata-qemu
kubectl get runtimeclass kata-fc

# Deploy a test pod -- this triggers Karpenter to provision a bare metal node
kubectl apply -f tests/test-kata-pod.yaml

# Wait for it to complete (bare metal provisioning takes a few minutes)
kubectl wait --for=jsonpath='{.status.phase}'=Succeeded pod/kata-test --timeout=600s

# Check the kernel -- should differ from host kernel, confirming VM isolation
kubectl logs kata-test
# Expected: 6.x.x (Kata guest kernel, NOT the host kernel)

# Cleanup
kubectl delete pod kata-test

Usage

Add runtimeClassName and the taint toleration to any pod spec:

spec:
  runtimeClassName: kata-qemu   # or kata-fc for Firecracker
  tolerations:
    - key: kata-runtime
      value: "true"
      effect: NoSchedule
  containers:
    - name: my-app
      image: my-image

The RuntimeClass automatically handles nodeSelector for katacontainers.io/kata-runtime: "true" and adds pod overhead (250m CPU, 160Mi memory) for the Kata VM.

Runtime Comparison

	QEMU	Firecracker
RuntimeClass	kata-qemu	kata-fc
Snapshotter	default (overlayfs)	devmapper
Extra setup	None	devmapper via EC2NodeClass userData
Best for	General workloads, broader device support	Lightweight, fast boot, minimal attack surface

Cleanup

Teardown does not require environment variables -- it deletes resources by name.

# Automated
./scripts/teardown.sh

# Or manually:
kubectl delete -f tests/                          # Remove test workloads
helm uninstall kata-deploy -n kube-system          # Remove kata-deploy
kubectl delete nodepool kata-metal                 # Remove NodePool
kubectl delete ec2nodeclass kata                   # Remove EC2NodeClass