Using the Cluster Autoscaler with Agones

[markmandel]

Problem

• It is desirable to autoscale the Kubernetes cluster to account for increases in player count, and need for more game servers, in a way that will work across cloud
• There currently exists a generic cluster autoscaler project -- but it is mostly targeted at stateless workloads

Design

The overall design to use the open source cluster autoscaler revolves around using the cluster autoscaler
to remove empty nodes as they appear, but specifically disallows the autoscaler from attempting to evict GameServer Pods in an attempt to move them to a new Node.

This gives us the following benefits:

• Implementations for multiple cloud providers are already written and community tested
• Load testing (on GKE) has been done for us
• Existing SLOs already exist for the autoscaler
• Removes any possibility for the autoscaler to cause race conditions when allocating GameServers.

Since the autoscaler can be implement with Agones GameServers -- this essentially means that scaling and autoscaling can be essentially managed at a Fleet level.
If you want a bigger cluster, increase the size of your Fleet, and your cluster will adjust. If you want a smaller cluster, shrink your fleet, and the cluster will adjust.

Implementation

Write a scheduler that bin packs all our pods into as tight a cluster as possible

A custom scheduler will be built that will prioritise the scheduling of GameServer Pods onto Nodes that have the most GameServer pods.
This will ease scaling down, as it will mean the game servers aren't spread out of many Nodes, and there is wasted resource space.

(Unless there is a way to do this with the default scheduler, but I've not found one so far -- best I could find was PreferredDuringSchedulingIgnoredDuringExecution on HostName)

Prioritise Allocating GameServers from Nodes that already have Allocated GameServers

To also make it easier to scale down, we essentially want to bin-pack as many allocated game servers on a single node as much as possible.

To that end, the allocate() function will order the GameServers it is considering by the number of other GameServers that exist on the same node as it.

This ensures that we don't end up with (as much as possible) a "swiss cheese" problem, with Allocated game servers spread out across the cluster. Bin packing Allocated GameServers
makes it much easier Fleets to scale down in a way that will leave empty nodes for the autoscaler to delete.

When Fleets get shrunk, prioritise removal from Nodes with the least number of GameServer Pods

Again, to make it easier to create empty nodes when scaling down Fleets, prioritise removing un-allocated GameServer Pods from Nodes with the least number
of GameServers Pods currently on them.

Mark All GameServer Pods as not "safe-to-evict"

If a Pod has the annotation cluster-autoscaler.kubernetes.io/safe-to-evict: false, then the Node that the Pod is on cannot be removed.

Therefore, all GameServer Pods should have the annotation cluster-autoscaler.kubernetes.io/safe-to-evict: false, so the autoscaler will not attempt to evict them.

Since we are bin packing through our custom scheduler, we won't actually have a need to move existing GameServer pods when nodes shrink, as we will only be leaving behind empty Nodes.

Mark the Agones controller as safe-to-evict: false

The Agones controller should have the annotation cluster-autoscaler.kubernetes.io/safe-to-evict: false, to ensure the autoscaler
doesn't try and move around the controller.

Documentation

Documentation of the above needs to be implemented, but also pointing to how to setup the autoscaler on different cloud providers, etc.

Research

• https://github.com/kubernetes/autoscaler/tree/master/cluster-autoscaler
• https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/FAQ.md
• https://cloud.google.com/kubernetes-engine/docs/how-to/cluster-autoscaler
• https://cloud.google.com/kubernetes-engine/docs/concepts/cluster-autoscaler
• https://kubernetes.io/docs/tasks/administer-cluster/configure-multiple-schedulers/
• https://thenewstack.io/implementing-advanced-scheduling-techniques-with-kubernetes/
• https://kubernetes.io/blog/2017/03/advanced-scheduling-in-kubernetes/

History

[markmandel]

Updated with exact details on the Taint being applied to the Node. Worked it out by digging through the code.

[GabeBigBoxVR]

This is great, I think one potential issue is when to scale up. For our servers, we use stateful Unity game servers and having an option where we can always leave one empty server running would be desired. I'm wondering what's the best way for demand to be fed into the autoscaler.

Regarding scale down, the defaults of 10 minutes are fine for me, given that it will ensure that spin up times are reasonably fast.

[markmandel]

@GabeBigBoxVR you would essentially control scale up/down by scaling up and down your Fleets - you cluster would then adjust to the space the Fleet was taking up accordingly. It's actually quite a nice model, as you only need to influence one part of your config - the rest happens automatically.

You can see some work on a Fleet autoscaler in #340

[markmandel]

Updated with some more thoughts on Fleet scale down - see history for changes.

[markmandel]

Updated with what I think this is the working design - much simpler details, and much less room for error.

See history for changes if you want to see previous versions.

The only leftover question I have is about how the scheduler will work, but I think that's a solvable problem, and just requires some research.