Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use the AutoScalingGroup for getting & scaling the nodegroups #3543

Merged
merged 4 commits into from Apr 9, 2021

Conversation

aclevername
Copy link
Contributor

@aclevername aclevername commented Apr 8, 2021

Description

Update the get nodegroup logic to fetch it from the ASG/EKS nodegroup instead of from the cloudformation as that can be out of date. Scale now modifys the ASG/EKS nodegroup directly as the cloudformation can drift away from the actual values due to cluster-autoscaler, simpler to just acknlowedge its not a true representation of the state and not use it rather than occasionally modifying it to be up to date.

Example

Get

For example if I now create two nodegroups:

eksctl get nodegroups --cluster jk
2021-04-08 16:36:00 [ℹ]  eksctl version 0.45.0-dev+12f62a86.2021-04-08T16:35:29Z
2021-04-08 16:36:00 [ℹ]  using region us-west-2
CLUSTER NODEGROUP       STATUS          CREATED                 MIN SIZE        MAX SIZE        DESIRED CAPACITY        INSTANCE TYPE   IMAGE ID                ASG NAME
jk      mng1            CREATE_COMPLETE 2021-04-08T15:23:07Z    1               13              3                       t2.medium       AL2_x86_64              eks-c8bc590a-33a0-e453-4cf6-0c4f1b81450c
jk      ng1             CREATE_COMPLETE 2021-04-08T15:22:05Z    1               13              3                       t2.small        ami-0cdd9b2ea797fd5db   eksctl-jk-nodegroup-ng1-NodeGroup-1CVH7TM1UIJGN

If I manually modify the desired to 2 and maxSize to 3 for both Get now returns:

eksctl get nodegroups --cluster jk
2021-04-08 16:42:23 [ℹ]  eksctl version 0.45.0-dev+12f62a86.2021-04-08T16:35:29Z
2021-04-08 16:42:23 [ℹ]  using region us-west-2
CLUSTER NODEGROUP       STATUS          CREATED                 MIN SIZE        MAX SIZE        DESIRED CAPACITY        INSTANCE TYPE   IMAGE ID                ASG NAME
jk      mng1            CREATE_COMPLETE 2021-04-08T15:23:07Z    1               3               2                       t2.medium       AL2_x86_64              eks-c8bc590a-33a0-e453-4cf6-0c4f1b81450c
jk      ng1             CREATE_COMPLETE 2021-04-08T15:22:05Z    1               3               2                       t2.small        ami-0cdd9b2ea797fd5db   eksctl-jk-nodegroup-ng1-NodeGroup-1CVH7TM1UIJGN

Where as before it would of returned:

eksctl get nodegroups --cluster jk
2021-04-08 16:42:44 [ℹ]  eksctl version 0.44.0
2021-04-08 16:42:44 [ℹ]  using region us-west-2
CLUSTER NODEGROUP       STATUS          CREATED                 MIN SIZE        MAX SIZE        DESIRED CAPACITY        INSTANCE TYPE   IMAGE ID                ASG NAME
jk      mng1            CREATE_COMPLETE 2021-04-08T15:22:05Z    1               13              3                       t2.medium                               eks-c8bc590a-33a0-e453-4cf6-0c4f1b81450c
jk      ng1             CREATE_COMPLETE 2021-04-08T15:22:05Z    1               13              3                       t2.small        ami-0cdd9b2ea797fd5db   eksctl-jk-nodegroup-ng1-NodeGroup-1CVH7TM1UIJGN

Scale

Before if you attempted to scale just the min or max, the desired value would go back to what was written in cloudformation, overriding any manual changes. This no longer happens, for example:

eksctl scale nodegroup --cluster jk --name ng1 --nodes-max 5
2021-04-08 16:44:15 [ℹ]  scaling nodegroup "ng1" in cluster jk
2021-04-08 16:44:18 [ℹ]  nodegroup successfully scaled

eksctl scale nodegroup --cluster jk --name mng1 --nodes-max 5
2021-04-08 16:44:21 [ℹ]  scaling nodegroup "mng1" in cluster jk
2021-04-08 16:44:24 [ℹ]  waiting for scaling of nodegroup "mng1" to complete
2021-04-08 16:44:41 [ℹ]  waiting for scaling of nodegroup "mng1" to complete
2021-04-08 16:44:41 [ℹ]  nodegroup successfully scaled

eksctl get nodegroups --cluster jk
2021-04-08 16:44:51 [ℹ]  eksctl version 0.45.0-dev+12f62a86.2021-04-08T16:35:29Z
2021-04-08 16:44:51 [ℹ]  using region us-west-2
CLUSTER NODEGROUP       STATUS          CREATED                 MIN SIZE        MAX SIZE        DESIRED CAPACITY        INSTANCE TYPE   IMAGE ID                ASG NAME
jk      mng1            CREATE_COMPLETE 2021-04-08T15:23:07Z    1               5               2                       t2.medium       AL2_x86_64              eks-c8bc590a-33a0-e453-4cf6-0c4f1b81450c
jk      ng1             CREATE_COMPLETE 2021-04-08T15:22:05Z    1               5               2                       t2.small        ami-0cdd9b2ea797fd5db   eksctl-jk-nodegroup-ng1-NodeGroup-1CVH7TM1UIJGN

However if I attempt the same set of commands with the previous version of eksctl it would of reverted the min and desired back to the original for unmanaged nodegroups:

eksctl scale nodegroup --cluster jk --name ng1 --nodes-max 5
2021-04-08 16:46:52 [ℹ]  scaling nodegroup "ng1" in cluster jk
2021-04-08 16:46:54 [ℹ]  scaling nodegroup, max size from 13 to 5
2021-04-08 16:46:55 [ℹ]  waiting for CloudFormation changeset "eksctl-scale-nodegroup-1617896814" for stack "eksctl-jk-nodegroup-ng1"
2021-04-08 16:47:11 [ℹ]  waiting for CloudFormation changeset "eksctl-scale-nodegroup-1617896814" for stack "eksctl-jk-nodegroup-ng1"
2021-04-08 16:47:12 [ℹ]  waiting for CloudFormation stack "eksctl-jk-nodegroup-ng1"
2021-04-08 16:47:31 [ℹ]  waiting for CloudFormation stack "eksctl-jk-nodegroup-ng1"
2021-04-08 16:47:51 [ℹ]  waiting for CloudFormation stack "eksctl-jk-nodegroup-ng1"
2021-04-08 16:48:10 [ℹ]  waiting for CloudFormation stack "eksctl-jk-nodegroup-ng1"
2021-04-08 16:48:27 [ℹ]  waiting for CloudFormation stack "eksctl-jk-nodegroup-ng1"
2021-04-08 16:48:46 [ℹ]  waiting for CloudFormation stack "eksctl-jk-nodegroup-ng1"
jake@jake-weave [16:48:46] [~/weave/eksctl] [main *]

eksctl scale nodegroup --cluster jk --name mng1 --nodes-max 5
2021-04-08 16:50:39 [ℹ]  scaling nodegroup "mng1" in cluster jk
2021-04-08 16:50:40 [ℹ]  scaling nodegroup, max size from 13 to 5
2021-04-08 16:50:41 [ℹ]  waiting for CloudFormation changeset "eksctl-scale-nodegroup-1617897040" for stack "eksctl-jk-nodegroup-mng1"
2021-04-08 16:50:57 [ℹ]  waiting for CloudFormation changeset "eksctl-scale-nodegroup-1617897040" for stack "eksctl-jk-nodegroup-mng1"
2021-04-08 16:50:59 [ℹ]  waiting for CloudFormation stack "eksctl-jk-nodegroup-mng1"
2021-04-08 16:51:17 [ℹ]  waiting for CloudFormation stack "eksctl-jk-nodegroup-mng1"
2021-04-08 16:51:37 [ℹ]  waiting for CloudFormation stack "eksctl-jk-nodegroup-mng1"
2021-04-08 16:51:57 [ℹ]  waiting for CloudFormation stack "eksctl-jk-nodegroup-mng1"
2021-04-08 16:52:13 [ℹ]  waiting for CloudFormation stack "eksctl-jk-nodegroup-mng1"

eksctl get nodegroups --cluster jk
2021-04-08 16:54:28 [ℹ]  eksctl version 0.45.0-dev+12f62a86.2021-04-08T16:53:38Z
2021-04-08 16:54:28 [ℹ]  using region us-west-2
CLUSTER NODEGROUP       STATUS          CREATED                 MIN SIZE        MAX SIZE        DESIRED CAPACITY        INSTANCE TYPE   IMAGE ID                ASG NAME
jk      mng1            ACTIVE 2021-04-08T15:23:07Z    1               5               2                       t2.medium       AL2_x86_64              eks-c8bc590a-33a0-e453-4cf6-0c4f1b81450c
jk      ng1             UPDATE_COMPLETE 2021-04-08T15:22:05Z    1               5               3                       t2.small        ami-0cdd9b2ea797fd5db   eksctl-jk-nodegroup-ng1-NodeGroup-1CVH7TM1UIJGN

Checklist

  • Added tests that cover your change (if possible)
  • Added/modified documentation as required (such as the README.md, or the userdocs directory)
  • Manually tested
  • Made sure the title of the PR is a good description that can go into the release notes
  • (Core team) Added labels for change area (e.g. area/nodegroup) and kind (e.g. kind/improvement)

BONUS POINTS checklist: complete for good vibes and maybe prizes?! 🤯

  • Backfilled missing tests for code in same general area 🎉
  • Refactored something and made the world a better place 🌟

@aclevername aclevername force-pushed the asg-scale branch 2 times, most recently from e1d00d0 to f561b0b Compare April 8, 2021 15:30
@aclevername aclevername changed the title use autoscaling group/EKS nodegroup summary as source of truth for nodegroup scaling config Use the AutoScalingGroup for getting & scaling the nodegroups Apr 8, 2021
@aclevername aclevername marked this pull request as ready for review April 8, 2021 16:15
@aclevername aclevername force-pushed the asg-scale branch 2 times, most recently from 58916cb to c8e8b8f Compare April 9, 2021 08:34
Comment on lines 205 to 209
if name == "" {
summaries = append(summaries, summary)
} else if summary.Name == name {
summaries = append(summaries, summary)
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what's going on here?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm just preserving the existing weird logic, not me! its some hack for searching for an individual nodegroup vs all.

@aclevername aclevername enabled auto-merge (squash) April 9, 2021 12:53
pkg/cfn/manager/nodegroup.go Outdated Show resolved Hide resolved
Co-authored-by: Chetan Patwal <cPu1@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Allow modifying the minSize and maxSize value of a NodeGroup without changing desiredCapacity
3 participants