Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

FleetAutoScaler scalingLimited is reporting the incorrect state #2828

Closed
Dinhh1 opened this issue Nov 28, 2022 · 5 comments · Fixed by #3448
Closed

FleetAutoScaler scalingLimited is reporting the incorrect state #2828

Dinhh1 opened this issue Nov 28, 2022 · 5 comments · Fixed by #3448
Labels
area/operations Installation, updating, metrics etc good first issue These are great first issues. If you are looking for a place to start, start here! help wanted We would love help on these issues. Please come help us! kind/bug These are bugs.

Comments

@Dinhh1
Copy link
Contributor

Dinhh1 commented Nov 28, 2022

What happened:
It seems as though the scalingLimited attribute of an autoscaler is incorrectly reporting it's state.

What you expected to happen:
I expect scalingLimited = false when the number of gameservers < the maxReplicas

How to reproduce it (as minimally and precisely as possible):

  1. Configure an autoscaler with: max - 50, min - 10, buffer - 5
  2. Allocate 1 or 2 GameServers
  3. Observer how scalingLimited is set to true even though we're below our max replica threshold
    image (3)

Anything else we need to know?:

Environment:

@Dinhh1 Dinhh1 added the kind/bug These are bugs. label Nov 28, 2022
@markmandel markmandel added area/operations Installation, updating, metrics etc help wanted We would love help on these issues. Please come help us! good first issue These are great first issues. If you are looking for a place to start, start here! labels Nov 28, 2022
@oniku-2929
Copy link
Contributor

I also met the same situation in my EKS and Minikube environment.
Then I did a little research and test about it.

In conclusion, I guess it's expected behavior.
Because this flag is exposed as a "scale-in & scale-out" Limited flag.
I misunderstood it's a "scale-out" Limited flag.

In pkg/fleetautoscalers/fleetautoscalers.go func applyBufferPolicy

func applyBufferPolicy(b *autoscalingv1.BufferPolicy, f *agonesv1.Fleet) (int32, bool, error) {
var replicas int32
if b.BufferSize.Type == intstr.Int {
replicas = f.Status.AllocatedReplicas + int32(b.BufferSize.IntValue())
} else {
// the percentage value is a little more complex, as we can't apply
// the desired percentage to any current value, but to the future one
// Example: we have 8 allocated replicas, 10 total replicas and bufferSize set to 30%
// 30% means that we must have 30% ready instances in the fleet
// Right now there are 20%, so we must increase the fleet until we reach 30%
// To compute the new size, we start from the other end: if ready must be 30%
// it means that allocated must be 70% and adjust the fleet size to make that true.
bufferPercent, err := intstr.GetValueFromIntOrPercent(&b.BufferSize, 100, true)
if err != nil {
return 0, false, err
}
// use Math.Ceil to round the result up
replicas = int32(math.Ceil(float64(f.Status.AllocatedReplicas*100) / float64(100-bufferPercent)))
}
limited := false
if replicas < b.MinReplicas {
replicas = b.MinReplicas
limited = true
}
if replicas > b.MaxReplicas {
replicas = b.MaxReplicas
limited = true
}
return replicas, limited, nil
}

This limited flag is exposed as a Prometheus metric "agones_fleet_autoscalers_limited" and then we can see it as a "Scaling Limited" flag on Grafana.

In the above situation,

1.Configure an autoscaler with: max - 50, min - 10, buffer - 5
2.Allocate 1 or 2 GameServers

Initially, the replicas count is 0(no allocation) + 5 = 5.
then replicas fix to 10(MinReplicas) and will be limited for avoiding scale-in.

still, the same adjustment happens after 1 or 2 GameServers allocations.
So "Scaling Limited" is True.

on "Metrics" Page.
https://agones.dev/site/docs/guides/metrics/

agones_fleet_autoscalers_limited The fleet autoscaler is capped (1)

I think it would be helpful to add "scale-in or scale-out is limited based on the fleet autoscaler's setting" in the description for avoiding such a misunderstanding.

I couldn't see the Slack Thread that is mentioned above.
So I don't understand any details about it on Slack.

@atgane
Copy link
Contributor

atgane commented Oct 12, 2023

Hi, may I handle this issue? I'll create PR soon!

@markmandel
Copy link
Member

Reading above, I'm not 100% sure if this is actually an issue 🤔

That being said, please investigate and let us know if you can confirm either way.

@markmandel
Copy link
Member

Or to be clearer - are you looking to update the code or the docs? (Sounds like the docs do need an update!)

@atgane
Copy link
Contributor

atgane commented Oct 19, 2023

I will update docs!

gongmax added a commit that referenced this issue Oct 25, 2023
* update fleet autoscaling limited signification(#2828)

* fix

* fix

---------

Co-authored-by: Mengye (Max) Gong <8364575+gongmax@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/operations Installation, updating, metrics etc good first issue These are great first issues. If you are looking for a place to start, start here! help wanted We would love help on these issues. Please come help us! kind/bug These are bugs.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants