Skip to content
Horizontal Pod Autoscaler with Nodeless k8s
Branch: master
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Type Name Latest commit message Commit time
Failed to load latest commit information.

Horizontal Pod Autoscaler with Nodeless Kubernetes

Step 1: Create 1-worker Nodeless Kubernetes cluster

Follow instructions in this repo to create a {1 master, 1 Milpa worker} Nodeless Kubernetes cluster.

Grab k8s master and Milpa worker IPs from terraform output:


master_ip =
milpa_worker_ip =

Step 2: Deploy Kubernetes Metrics Server

Log on to Kubernetes master, verify cluster is up, then deploy Kubernetes Metrics Server:

$ ssh -i myechuri-key2.pem ubuntu@
$ kubectl get nodes -o wide
ip-10-0-100-17.ec2.internal    Ready    master   36s   v1.15.0    Ubuntu 16.04.6 LTS   4.4.0-1087-aws   docker://18.9.7
ip-10-0-100-251.ec2.internal   Ready    <none>   12s   v1.15.0   Ubuntu 16.04.6 LTS   4.4.0-1087-aws   kiyot://1.2.0
git clone
cd metrics-server
git checkout bugfix/deploy-1.8+
kubectl apply -f deploy/1.8+/

Step 3: Run resource-consumer to generate load

kubectl run resource-consumer --expose --service-overrides='{ "spec": { "type": "LoadBalancer" } }' --port 8080 --requests='cpu=100m' --overrides='{"apiVersion": "apps/v1", "spec":{ "template":{ "metadata": { "annotations":{"":"kiyot"} }, "spec": { "nodeSelector": {"": "milpa-worker"} } } } }'

Wait for resource-consumer to be in Running state.

$ kubectl get pods

NAME                                 READY   STATUS    RESTARTS   AGE
resource-consumer-6758b8d5dd-5wvdc   1/1     Running   0          78s

Grab external-ip of its load balancer.

export RESOURCE_CONSUMER_ADDRESS="http://$(kubectl get service resource-consumer -ojsonpath='{.status.loadBalancer.ingress[0].hostname}'):8080"

Step 4: Create HPA for resource-consumer

Scale up if CPU utilization goes above 50%. Set min replicas to 1, max to 20.

kubectl autoscale deployment resource-consumer --cpu-percent=50 --min=1 --max=20

Wait for percentage utilization to transition from unknown to a number (takes upto 20 seconds).

ubuntu@ip-10-0-100-234:~$ kubectl get hpa
NAME                REFERENCE                      TARGETS         MINPODS   MAXPODS   REPLICAS   AGE
resource-consumer   Deployment/resource-consumer   <unknown>/50%   1         20        0          11s

ubuntu@ip-10-0-100-234:~$ kubectl get hpa
NAME                REFERENCE                      TARGETS   MINPODS   MAXPODS   REPLICAS   AGE
resource-consumer   Deployment/resource-consumer   1%/50%    1         20        1          63s

Step 5: Generate load over 50% to trigger horizontal pod autoscaling

Please note that we did not have to setup/configure/monitor/manage Cluster Autoscaler because we are running in Nodeless mode! Yay!

Ask resource-consumer to consume 600 millicores for 300 seconds.

curl --data "millicores=600&durationSec=300" $RESOURCE_CONSUMER_ADDRESS/ConsumeCPU

Watch deployments, replicasets, pods, hpa.

watch --differences kubectl get deployments,replicasets,pods,hpa
NAME                                      READY   UP-TO-DATE   AVAILABLE   AGE
deployment.extensions/resource-consumer   12/12   12           12          101m

NAME                                                 DESIRED   CURRENT   READY   AGE
replicaset.extensions/resource-consumer-6758b8d5dd   12        12        12      101m

NAME                                     READY   STATUS    RESTARTS   AGE
pod/resource-consumer-6758b8d5dd-4s6pt   1/1     Running   0          3m30s
pod/resource-consumer-6758b8d5dd-5fl2k   1/1     Running   0          3m45s
pod/resource-consumer-6758b8d5dd-5wvdc   1/1     Running   0          101m
pod/resource-consumer-6758b8d5dd-f6z82   1/1     Running   0          4m46s
pod/resource-consumer-6758b8d5dd-lgjwd   1/1     Running   0          3m45s
pod/resource-consumer-6758b8d5dd-n5c4t   1/1     Running   0          3m45s
pod/resource-consumer-6758b8d5dd-pktl8   1/1     Running   0          3m45s
pod/resource-consumer-6758b8d5dd-qggc2   1/1     Running   0          4m46s
pod/resource-consumer-6758b8d5dd-rjl65   1/1     Running   0          4m46s
pod/resource-consumer-6758b8d5dd-scx8c   1/1     Running   0          3m30s
pod/resource-consumer-6758b8d5dd-sps4m   1/1     Running   0          3m30s
pod/resource-consumer-6758b8d5dd-wbqts   1/1     Running   0          3m30s

NAME                                                    REFERENCE                      TARGETS   MINPODS   MAXPODS   REPLICAS
horizontalpodautoscaler.autoscaling/resource-consumer   Deployment/resource-consumer   11%/50%   1         20        12

Notice the 11 new pod provisioned by HPA in response to the load.

Under the covers, Nodeless k8s manifested just-in-time capacity for these pods. There was no Cluster Autoscaler to configure or manage!


Follow teardown instructions from kubeadm repo.


This tutorial is built on top of Matt Kelly's tutorial. We are grateful to his work in identifying and providing workarounds for bugs in Metrics Server.

You can’t perform that action at this time.