Horizontal Pod Autoscaler - Busy Runner Scheme #223
Conversation
|
#213 to cross link another active hpa PR |
|
Hello @ZacharyBenamram, I have a small question and I appreciate it if you can help me to understand more about this. Assume I have 3 runners (running in separate EKS clusters) that are added to a specific Github repository, How does the controller know which runner deployment to scale up/down? I think it supposed to be based on the runner labels maybe? but I guess it will always scale all the 3 runners (due to a workflow that is in the queue pending). |
|
@ahmad-hamade this autoscaling scheme will scale the runners that your runner-deployment creates on your behalf, or the runners you've created Runner resources for. If you are running 3 runners all in separate EKS clusters - i am also assuming you are also running 3 separate controllers. If that is the case, each horizontal runner autoscaler that you have defined in each cluster will be responsible for scaling that clusters runners - which is a benefit over the previous implementation, where a single horizontal runner autoscaler is scaling across all runners, since it cannot distinguish pods running in separate clusters. |
|
Thank you @ZacharyBenamram 👍 |
| var desiredReplicas int | ||
| fractionBusy := float64(numRunnersBusy) / float64(numRunners) | ||
| if fractionBusy >= scaleUpThreshold { | ||
| scaleUpReplicas := int(float64(numRunners)*scaleUpFactor + 0.5) |
There was a problem hiding this comment.
| scaleUpReplicas := int(float64(numRunners)*scaleUpFactor + 0.5) | |
| scaleUpReplicas := int(float64(numRunners)*scaleUpFactor) |
Can we remove the magic number 0.5 out of this calculation? I think the defaultScaleUpFactor of 1.3 is enough, and you can optionally add that to the default value so that it becomes defaultScaleUpFactor = 1.8.
There was a problem hiding this comment.
@ZacharyBenamram Awesome work! Thank you so much for your contribution
I had a comment regarding the defaultScaleUpFactor so it would be great if you could address that in comment or another pull request.
Providing an alternative to the current hpa scheme. The TotalNumberOfQueuedAndInProgressWorkflowRuns scheme requires that the hpa be aware of all the repositories that are currently managed by actions - this doesnt scale well in larger organizations where multiple teams are enrolling into actions - they must be aware that the hpa spec be updated.
The PercentageRunnersBusy scheme is a simple autoscaling scheme that retrieves all action runners from github, finds the subset of runners that exist in the hpa namespace by comparing it against the runner resources, and then evaluates the scale up / scale down condition.
This scheme also addresses the issue where an organization has various environments + multiple hpas - where each hpa should be responsible for scaling only the runners the shared namespace.