Should --scale-down-delay-after-add be per-nodepool?

[therc]

I just saw some nodes getting removed in our cluster after a whole 24m, rather than the expected default of 10m. This was because of two different scale up events that occurred, one before and one after the 10m mark. The autoscaler didn't reuse the empty nodes because, both times, they had no GPUs. This is on AWS, but it should apply everywhere.

So... would it make sense to make the cooldown apply only to each autoscaling group or instance type? I should be able to take down CPU-only instances even if I spawn a new GPU instance every five minutes.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

[fejta-bot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

[k8s-ci-robot]

@fejta-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[ajcann]

@therc Did you ever get traction on this or find a workaround / different understanding?

[rrangith]

/reopen
/remove-lifecycle rotten

[k8s-ci-robot]

@rrangith: You can't reopen an issue/PR unless you authored it or you are a collaborator.

In response to this:

/reopen
/remove-lifecycle rotten

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[gjtempleton]

/reopen

[k8s-ci-robot]

@gjtempleton: Reopened this issue.

In response to this:

/reopen

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[elmiko]

we had a good discussion about this during today's sig meeting, i am curious if we want to preserve the default behavior in any way?

i'm not sure that users would want the original behavior, but i would be concerned about breaking people who did expect it.

[vadasambar]

I wonder if the reason for blocking scale down if scale up occurs is because of the following assumption:

• If the cluster is already scaling up, there shouldn't be a need for scale down

It seems like we want to break the above assumption and apply it on per node or nodegroups or a group of nodegroups basis instead of applying it to the cluster as a whole.

[rrangith]

we had a good discussion about this during today's sig meeting, i am curious if we want to preserve the default behavior in any way?

i'm not sure that users would want the original behavior, but i would be concerned about breaking people who did expect it.

If we want to preserve default behaviour, we could make it configurable. For example per nodepool cooldown could be the default, and users could opt out to have the old behaviour.

Or we could stick with per nodepool cooldown being the only option, and users could adjust scale-down-unneeded-time if they want their unneeded nodes to stay longer.

I wonder if the reason for blocking scale down if scale up occurs is because of the following assumption:

• If the cluster is already scaling up, there shouldn't be a need for scale down

It seems like we want to break the above assumption and apply it on per node or nodegroups or a group of nodegroups basis instead of applying it to the cluster as a whole.

Correct. Like described by the original issue reporter, if one nodepool is for GPU based instance types and keeps getting scaled up every 5 minutes, the nodepool for CPU-ony instance types should be able to be scaled down. Currently in this situation, the CPU-only instances would stick around indefinitely until the GPU nodepool is down scaling up.

A more concrete usecase where this feature would be useful: I have nodepool A with instanceType A, and nodepool B with instanceType B. At first nodepool A has N instances, but we want to migrate all pods to nodepool B which currently has 0 instances. So we cordon all nodes from nodepool A and start migrating the pods 1 by 1 to nodepool B. While this is going on, nodepool B is continually scaling up as each pod migrates 1 by 1. So scaledowns will continually be blocked and will lead to an excess of nodes. If per-nodepool scaledown cooldown is implemented, nodepool A would still be able to scaledown since it has no scaleups.

[qianlei90]

some discussion here：#3789 (comment)

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[vadasambar]

/remove-lifecycle stale

[vadasambar]

/assign vadasambar