Skip to content

Kubernetes horizontal pod auto scaling with Akka.NET cluster sharding

Notifications You must be signed in to change notification settings

Arkatufus/Akka.Cluster.Sharding-Kubernetes-Scaling

Repository files navigation

Horizontal Pod Auto Scaling With Akka.NET Cluster Sharding

This repository is a clone of Akka.NET shopping cart example, modifying it to use Akka.Hosting and Akka.Discovery.KubernetesApi and designed as a showcase of how Akka.NET cluster pods can be auto scaled.

  • The sharding entity actor in this example has been modified to generate synthetic CPU load when it is initialized.
  • The Kubernetes YAML configuration file was designed to auto scale the cluster from 3 replicas to a maximum of 10 replicas when the metric server detects a spike of average pod CPU consumption over 80% (800m)

How To Use The Example

Setup

  1. Make sure that you have Docker Desktop installed on your computer
  2. Make sure that Docker Desktop is running using WSL Linux containers
  3. Make sure that Docker Desktop Kubernetes feature is enabled
  4. In a Windows PowerShell terminal, navigate to the project directory and execute
    .\install-metrics.cmd
    You should see an output similar to this:
    PS C:\> .\install-metrics.cmd
    serviceaccount/metrics-server created
    clusterrole.rbac.authorization.k8s.io/system:aggregated-metrics-reader created
    clusterrole.rbac.authorization.k8s.io/system:metrics-server created
    rolebinding.rbac.authorization.k8s.io/metrics-server-auth-reader created
    clusterrolebinding.rbac.authorization.k8s.io/metrics-server:system:auth-delegator created
    clusterrolebinding.rbac.authorization.k8s.io/system:metrics-server created
    service/metrics-server created
    deployment.apps/metrics-server created
    apiservice.apiregistration.k8s.io/v1beta1.metrics.k8s.io created
    
  5. Wait a minute or so for Kubernetes metrics-server to spin up.
  6. Confirm that Kubernetes metrics-server is up by executing:
    kubectl top node
    If metrics-server has been installed successfully, you should see an output similar to this:
    PS C:\> kubectl top node
    NAME             CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%
    docker-desktop   172m         2%     3041Mi          52%   
    

Running The Example

In a Windows PowerShell terminal, navigate to the project directory and execute

.\start-k8s.cmd

This will:

  • Compile the project,
  • Build the docker image
  • Create an Akka.NET sharded cluster inside the shopping-cart namespace

Execute kubectl top pod -n shopping-cart a few times until you see that metrics-server has successfully collected metrics from the pods:

PS C:\> kubectl top pod -n shopping-cart
error: metrics not available yet
PS C:\> kubectl top pod -n shopping-cart
error: metrics not available yet
PS C:\> kubectl top pod -n shopping-cart
error: metrics not available yet
PS C:\> kubectl top pod -n shopping-cart
NAME         CPU(cores)   MEMORY(bytes)
backend-0    39m          69Mi
backend-1    58m          75Mi
frontend-0   90m          89Mi

Confirming Auto-scaling

  1. The frontend-0 pod will spawn 30 actors inside the backend stateful set cluster, creating a CPU spike on all 3 backend pods.
  2. Execute
    kubectl top pod -n shopping-cart
    and check that backend pod CPU metrics spikes to above 800m
    PS C:\> kubectl top pod -n shopping-cart
    NAME         CPU(cores)   MEMORY(bytes)
    backend-0    1001m        86Mi
    backend-1    975m         84Mi
    backend-2    962m         83Mi
    frontend-0   63m          99Mi   
    
  3. The pod auto scaler will scale the backend cluster after a few seconds
    PS C:\> kubectl top pod -n shopping-cart
    NAME         CPU(cores)   MEMORY(bytes)
    backend-0    21m          86Mi
    backend-1    17m          87Mi
    backend-2    18m          86Mi
    backend-3    18m          87Mi
    backend-4    17m          83Mi
    backend-5    16m          88Mi
    backend-6    17m          82Mi
    backend-7    16m          91Mi
    frontend-0   35m          99Mi
  4. Shut down the frontend-0 pod by scaling the frontend stateful set to 0. Execute:
    kubectl scale --replicas=0 statefulset/frontend -n shopping-cart
  5. Confirm that the frontend-0 pod has been shut down by executing kubectl top pod -n shopping-cart
    PS C:\> kubectl top pod -n shopping-cart
    NAME        CPU(cores)   MEMORY(bytes)
    backend-0   21m          87Mi
    backend-1   21m          88Mi
    backend-2   17m          84Mi
    backend-3   20m          88Mi
    backend-4   20m          84Mi
    backend-5   26m          89Mi
    backend-6   19m          83Mi
    backend-7   17m          92Mi
    Note that frontend-0 pod is now missing from the list
  6. Wait 5-10 minutes and check the cluster status again by executing kubectl top pod -n shopping-cart
    PS C:\> kubectl top pod -n shopping-cart
    NAME        CPU(cores)   MEMORY(bytes)
    backend-0   21m          87Mi
    backend-1   21m          88Mi
    backend-2   17m          84Mi
    Note that the cluster has been automatically scaled down

Stopping The Example

To stop the Kubernetes cluster, execute:

.\destroy-k8s.cmd

About

Kubernetes horizontal pod auto scaling with Akka.NET cluster sharding

Topics

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published