kubernetes-warm-images

Kubernetes Warm Images

The goal of this project is to "keep images warm" by pulling images onto Kubernetes nodes.

Use Cases

• "Warm serverless images" - This could be useful to you if you're running serverless workloads on Kubernetes where the overhead for pulling images each time is consequential.
• "Warm critical images" - You may want your nodes to have images cached for critical workloads before they're actually needed.

Architecture

The project will consist of three components.

1. The Controller is responsible for subscribing to relevant object changes in Kubernetes declarative state.
2. The Agent runs on each node and pulls images to the node.
3. The communication medium for the project is nats. NATs is configurable by the operator which means more freedom for those running the project.

Installation

1. Create the namespace

kubectl create ns warm-images

2. Install NATs

I suggest using the Bitnami NATs Helm chart as at time of writing it is maintained, up-to-date and fairly configurable. See more at https://github.com/bitnami/charts/tree/master/bitnami/nats

helm repo add bitnami https://charts.bitnami.com/bitnami
helm repo update
helm install --namespace warm-images wi-nats bitnami/nats

3. Generate your values.yaml file for using with the Helm chart

# Create a values.yaml file
echo "nats:" > values.yaml
echo "  url: \"nats://wi-nats-client:4222\"" >> values.yaml
# Get the username & password from NATs
echo "  username: $(kubectl get cm --namespace warm-images wi-nats -o jsonpath='{.data.*}' | grep -m 1 user | awk '{print $2}')" >> values.yaml 
echo "  password: $(kubectl get cm --namespace warm-images wi-nats -o jsonpath='{.data.*}' | grep -m 1 password | awk '{print $2}')" >> values.yaml
# Allow watching some namespaces & ignoring others
echo "list: \"*\"" >> values.yaml
echo "ignore: \"kube-system\"" >> values.yaml

4. Install Warm Images

# Install 
helm repo add captains-charts https://storage.googleapis.com/captains-charts
helm repo update
helm install --namespace warm-images --values values.yaml wi captains-charts/warm-images

Usage Guide

Changing the namespaces

Modify the values.yaml file for multiple namespaces by separating them with spaces in the field called list

list: "ns1 default ns2"

or watch all namespaces using *:

list: "*"

To ignore some number of namespaces modify the ignore field in the values.yaml. It accepts a space-separated list of namespaces.

Monitoring

TBA

Roadmap v1.0.1

• Remove reg. prefix

Roadmap v2.0.0 - Post Experientia

• Philosophical: Figure out exactly which resources could be watched (DaemonSets, Deployments, etc).
• Philosophical: Figure out how to select.
• Philosophical: Figure out how to skip.
• LabelSelectors?
• Lua for Controller-side custom logic?
• Lua for Agent-side custom logic?
• Config: Exclude images that "contain".
• Tests - Go.
• Tests - Helm.
• Export Prometheus endpoint. Config for Helm.
• Grafana Dashboard.
• Monitoring first pass - logs.

Roadmap - Future 🌟🌟🌟

• Support for ContainerD.
• Clean shutdown - Controller.
• Clean shutdown - Agent.
• Consider support for non-NATs streaming.
• Helm ClusterRole (specific editable one instead of View)
• Test different scenarios (at scale, low availability, etc).
• Push Helm package to online repo as part of GitHub Action.
• K8s documentation (diagram, scaling etc)
• Helm documentation - all values tables & values example files.
• Multi-instance deployments for large clusters.