Autoscaling

You can tell Fenzo how, and under what conditions, it should scale the number of available hosts up or down. This page explains these autoscaling options.

Contents:

1. Introduction
2. Shortfall Evaluation
3. Custom Autoscale Rules
   1. Creating an Autoscale Rule
   2. Defining Your Autoscale Groups
   3. Adding Rules to Your Task Scheduler at Build Time
   4. Adding or Changing Your Autoscale Rules at Run Time
4. Additional Autoscaler Build-Time Options
5. The AutoScaler Callback

Introduction

Fenzo regulates autoscaling in two ways:

1. by defining a set of autoscaling groups and then applying a set of rules, each specific to one such group, that specify the minimum and maximum number of idle hosts to allow and the cooldown period between autoscaling actions in which other such actions are forbidden
2. by also applying Fenzo’s resource shortfall evaluation to estimate resource needs based on the current task workload, and to trigger a scale-up action when Fenzo believes resources are wanting

Shortfall Evaluation

It is difficult to anticipate a resource shortfall so that you can ensure there will be enough hosts up and running to be able to accept all of the incoming tasks. Fenzo implements a shortfall evaluator that autoscales a cluster up to respond to new tasks.

You may optionally disable shortfall evaluation by using the .disableShortfallEvaluation() builder method when you build your task scheduler (see Building Your Scheduler). If you do not disable it, it will operate independently from your custom autoscale rules and may initiate autoscale-up actions even within the cooldown period of those rules.

Custom Autoscale Rules

You can set the autoscale rules for your task scheduler when you build it (see Building Your Scheduler), or afterwards, by means of task scheduler methods. You can also modify the autoscale rules in place on a task scheduler by means of those methods. (However, the only way to disable shortfall evaluation is in the task scheduler builder; you cannot disable shortfall evaluation in an already-established task scheduler.)

Creating an Autoscale Rule

You define an autoscale rule by means of an object that implements the AutoScaleRule interface. This interface consists of the following five methods:

method	returns	description
getCoolDownSecs()	long	Return the number of seconds to wait after a scale action is triggered in a group before allowing another scale action to trigger in the same group.
getMaxIdleHostsToKeep()	int	Return the maximum number of hosts in this autoscale group that Fenzo is to keep in idle readiness before it begins to scale down.
getMinIdleHostsToKeep()	int	Return the minimum number of hosts in this autoscale group that Fenzo is to keep in idle readiness before it begins to scale up.
getRuleName()	String	Return the value of the host attribute that defines this autoscale group, the attribute you set when you built your task scheduler by means of its builder’s withAutoScaleByAttributeName() method (see below). This name will be available to your autoscale callback by means of its AutoScaleAction-implementing parameter.
idleMachineTooSmall(lease)	boolean	This method is passed a VirtualMachineLease object that describes an unused host. Return true if that host has sufficient resources to be considered an idle but potentially-useful host; return false if that host will not be sufficient for the tasks assigned to this autoscale group and therefore should not be considered as one of the idle hosts.

Defining Your Autoscale Groups

Typically, you want to autoscale different sorts of hosts in different ways. To do this, you divide hosts into groups and apply a different autoscale rule to each group. To define a group, you must identify (or create) a host attribute that uniquely defines which hosts belong to which group, so that every host in a particular group has the same value for that attribute, and no host in any other group has that value for that attribute. That is to say, every host must belong to a single group.

You always define your autoscale groups when you build your task scheduler, by means of its builder methods, like this:

myScheduler = new TaskScheduler.Builder()
                               .withAutoScaleByAttributeName(someAttributeName)
                               ⋮
                               .build();

Adding Rules to Your Task Scheduler at Build Time

To build a task scheduler with custom autoscale rules, call its withAutoScaleRule() builder method. Pass this method an object that implements the AutoScaleRule interface that describes an autoscale rule. The Fenzo autoscaler will apply this rule to the hosts in the group to which it applies. You define which rule applies to which group of hosts by means of the rule’s getRuleName() method, which returns the value of the host attribute you defined with .withAutoScaleByAttributeName() for all hosts in the group the rule applies to.

You add this method in the builder chain as follows:

myScheduler = new TaskScheduler.Builder()
                               .withAutoScaleByAttributeName(someAttributeName)
                               .withAutoScaleRule(ruleForAutoscaleGroup1)
                               .withAutoScaleRule(ruleForAutoscaleGroup2)
                               ⋮
                               .build();

Adding or Changing Your Autoscale Rules at Run Time

The following TaskScheduler methods allow you to add or change the autoscale rules in force on an already-built task scheduler:

method	description
getAutoScaleRules()	This method returns a Collection of all of the AutoScaleRule objects that represent the currently-active autoscale rules in Fenzo.
addOrReplaceAutoScaleRule()	Add a rule to a particular autoscale group. This method will overwrite any rule that previously applied to that group.
removeAutoScaleRule()	Remove the rule that applies to a particular autoscale group.
setAutoscalerCallback()	Change the callback routine you established at build time, or add a callback routine if you did not do so at build time. Fenzo calls this routine each time it performs a scale up or scale down action, and indicates to the callback what rule prompted the action.

Additional Autoscaler Build-Time Options

You can modify Fenzo’s autoscaling process with some additional build-time methods:

myScheduler = new TaskScheduler.Builder()
                               .withAutoScaleByAttributeName(someAttributeName)
                               .withAutoScaleDownBalancedByAttributeName(someAttributeName) // optional
                               .withAutoScalerMapHostnameAttributeName(myMappingFunction)   // optional
                               .withAutoscalerCallback(myAutoscaleFunction)
                               ⋮
                               .build();

The following table explains these methods:

method	description
.withAutoScaleDownBalancedByAttributeName()	Call this method to tell the autoscaler to try to maintain a balance of host varieties when it scales down a cluster. Pass the method a host attribute, and the autoscaler will attempt to scale down in such a way as to maintain a similar number of hosts with each value for that attribute.
.withAutoScalerMapHostnameAttributeName()	In some circumstances (for instance with Amazon Web Services), the host name is not the correct identifier for the host in the context of an autoscaling action (for instance, in AWS, you need the EC2 instance identifier). If this is the case for your system, you need to implement a function that maps the host name to the identifier for the host in an autoscaling context so that Fenzo can perform autoscaling properly. You provide this function to the task manager by means of this builder method.
.withAutoScalerCallback()	The callback method you pass to this function receives notice that it should perform an autoscale function, either UP or DOWN, and then scales the cluster accordingly.

The AutoScaler Callback

Your AutoScaler callback scales the cluster based on directives from the task scheduler. You can register this callback with the task scheduler when you build the task scheduler, by means of the withAutoScalerCallback() builder method, or after you have build the task scheduler, by calling its setAutoScalerCallback() method.

Your AutoScaler callback method takes as its parameter an object that implements the AutoScaleAction interface. This object indicates whether your method is to scale the cluster up or down, and contains some additional information that may be helpful to it in either case.

The object that is passed to your AutoScaler callback will be either an ScaleUpAction object or a ScaleDownObject object, which represent, naturally enough, indications that your callback should scale the cluster up or down, respectively. Your callback can determine which of these objects it has by calling the object’s getType() method, which returns either Up or Down.

These objects have the following methods that can help your callback routine determine how best to proceed, or can provide you with useful information about how and why your callback is being invoked:

action	method	purpose
both	getRuleName()	get the name of the autoscale rule that triggered this action
Up	getScaleUpCount()	get the number of hosts your callback should add to the cluster in order to scale up appropriately
Down	getHosts()	get a Collection of host names representing those hosts your callback should remove from the cluster as it scales down