Multiple instance types for managed machine pools #4358

arjunrn · 2023-06-26T06:50:41Z

What type of PR is this?

/kind feature

What this PR does / why we need it:

EKS node groups support multiple instance types. The order of the instance types and capacity type together determine which instances are created by AWS when the node group is scaled up. Also added validation so that both fields instanceType and instanceTypes cannot be specified together. . Added this as a new field to the AWSManagedMachinePool in in version v1beta2.

Fixes #3585

Special notes for your reviewer:
This new field does not exist in the older v1beta1 type of AWSManagedMachinePool. I'm not sure if it is even needed because the older single instance type field is still present. However code-gen complains that this field is not properly converted. Note that v1beta2 is the stored version so there's no chance for a loss of data.

Checklist:

squashed commits
includes documentation
adds unit tests
adds or updates e2e tests

Release note:

NONE

k8s-ci-robot · 2023-06-26T06:50:49Z

Welcome @arjunrn!

It looks like this is your first PR to kubernetes-sigs/cluster-api-provider-aws 🎉. Please refer to our pull request process documentation to help your PR have a smooth ride to approval.

You will be prompted by a bot to use commands during the review process. Do not be afraid to follow the prompts! It is okay to experiment. Here is the bot commands documentation.

You can also check if kubernetes-sigs/cluster-api-provider-aws has its own contribution guidelines.

You may want to refer to our testing guide if you run into trouble with your tests not passing.

If you are having difficulty getting your pull request seen, please follow the recommended escalation practices. Also, for tips and tricks in the contribution process you may want to read the Kubernetes contributor cheat sheet. We want to make sure your contribution gets all the attention it needs!

Thank you, and welcome to Kubernetes. 😃

k8s-ci-robot · 2023-06-26T06:50:50Z

Hi @arjunrn. Thanks for your PR.

I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

richardcase · 2023-06-28T07:56:40Z

/ok-to-test

richardcase

Thanks for this. I have added InstanceType to the issue where we are tracking changes for a future v1beta3 API version. so we can drop the field all together in the future.

richardcase · 2023-06-28T07:59:11Z

exp/api/v1beta2/awsmanagedmachinepool_types.go

@@ -120,6 +120,10 @@ type AWSManagedMachinePoolSpec struct {
 	// +optional
 	InstanceType *string `json:"instanceType,omitempty"`

+	// InstanceTypes specifies the AWS instance types


We should probably add a note to InstanceType to say the filed is deprecated and that InstanceTypes should be used instead.

When we bump the API version we can then remove the old field

Added the note.

richardcase · 2023-06-28T08:01:55Z

exp/api/v1beta2/awsmanagedmachinepool_webhook.go

@@ -123,6 +123,10 @@ func (r *AWSManagedMachinePool) validateLaunchTemplate() field.ErrorList {
 		return allErrs
 	}

+	if r.Spec.InstanceTypes != nil {


We can also log in the Create about the deprecation of the InstanceType field

Added log line.

enxebre · 2023-06-28T08:30:15Z

exp/api/v1beta2/awsmanagedmachinepool_types.go

@@ -120,6 +120,10 @@ type AWSManagedMachinePoolSpec struct {
 	// +optional
 	InstanceType *string `json:"instanceType,omitempty"`

+	// InstanceTypes specifies the AWS instance types
+	// +optional


Can we elaborate in the comment what happens when you specify multiple instances types?
What's the expectation, allocation strategy...?

The expectation is any of these instance types could be present in the node group. The allocation strategy is completely managed by AWS. Is there anything you think I should mention in the doc string?

Right now by reading the API I'm not sure what intent the field is expressing. I'd add something like

The allocation strategy to provision On-Demand capacity is set to prioritized. Managed node groups use the order of instance types passed in the API to determine which instance type to use first when fulfilling On-Demand capacity.

From https://docs.aws.amazon.com/eks/latest/userguide/managed-node-groups.html#managed-node-group-capacity-types

Or at minimum I'd clarify something like

Ordering of the passed values means priority. See https://docs.aws.amazon.com/eks/latest/userguide/managed-node-groups.html#managed-node-group-capacity-types for more info

Just a suggestion feel free to proceed otherwise.

I agree that would be a good idea to expand the comment. I quite like the suggestion fro @enxebre 👍

Ordering of the passed values means priority. See https://docs.aws.amazon.com/eks/latest/userguide/managed-node-groups.html#managed-node-group-capacity-types for more info

Updated the doc string. PTAL.

enxebre · 2023-07-05T07:23:54Z

pkg/cloud/services/eks/nodegroup.go

@@ -227,6 +227,10 @@ func (s *NodegroupService) createNodegroup() (*eks.Nodegroup, error) {
 	if managedPool.InstanceType != nil {
 		input.InstanceTypes = []*string{managedPool.InstanceType}
 	}
+	for _, instanceType := range managedPool.InstanceTypes {


may be clarify in the API docs as well that setting both InstanceType and InstanceTypes will append the latter to the former? That UX might come us a surprise. Or maybe event prevent both from being set in validation

That's already the case through validation. The code looks this way but the webhooks would prevent both of these from being set.

https://github.com/kubernetes-sigs/cluster-api-provider-aws/blob/a16dd53553cfc688f2ce29e7ab08181572181d03/exp/api/v1beta2/awsmanagedmachinepool_webhook.go#L231-L237

enxebre · 2023-07-05T07:24:57Z

Just one more comment #4358 (comment).
Otherwise lgtm after squashing commits.

enxebre · 2023-07-06T09:44:41Z

nit: commit is missing imperative verb i.e. Add multiple instances...
Also to clarify the commit desc

EKS node groups support multiple instance types

when the capacity type is spot.

This comes in more handy for spot but is also valid for on demand, correct?
/lgtm

EKS node groups support [multiple instance types ](https://docs.aws.amazon.com/eks/latest/APIReference/API_CreateNodegroup.html#API_CreateNodegroup_RequestSyntax). The order of the instance types and capacity type together determine which instances are created by AWS when the node group is scaled up. Also added validation so that both fields `instanceType` and `instanceTypes` cannot be specified together.

k8s-ci-robot · 2023-07-06T10:18:59Z

New changes are detected. LGTM label has been removed.

k8s-ci-robot · 2023-07-06T10:19:10Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please ask for approval from enxebre. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

arjunrn · 2023-07-06T12:43:47Z

@enxebre @richardcase Please take another look and let me know if you need anything else.

richardcase · 2023-08-29T09:17:12Z

@arjunrn - did you mean to close this?

idanshaby · 2023-10-31T09:25:49Z

@richardcase it looks like there was a bit of miscommunication here.
I think that this PR was ready from @arjunrn 's side, and that he was waiting for you to review it.
This is a fundamental functionality that's required for people to use CAPA in production.
Can anyone please try to push it? 🙏

arjunrn · 2023-10-31T21:17:20Z

@idanshaby Please feel free to use the changes in my branch to create a new PR. We've discontinued the use of CAPI/CAPA and I'm not working on this feature any more.

k8s-ci-robot added needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels Jun 26, 2023

k8s-ci-robot requested review from AverageMarcus and shivi28 June 26, 2023 06:51

arjunrn changed the title ~~WIP: Multiple instance types to managed machine pools~~ WIP: Multiple instance types for managed machine pools Jun 26, 2023

arjunrn changed the title ~~WIP: Multiple instance types for managed machine pools~~ Multiple instance types for managed machine pools Jun 26, 2023

k8s-ci-robot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jun 26, 2023

k8s-ci-robot added ok-to-test Indicates a non-member PR verified by an org member that is safe to test. and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Jun 28, 2023

richardcase mentioned this pull request Jun 28, 2023

v1beta3 API Changes #3853

Open

2 tasks

richardcase reviewed Jun 28, 2023

View reviewed changes

enxebre reviewed Jun 28, 2023

View reviewed changes

k8s-ci-robot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. and removed size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels Jun 29, 2023

arjunrn force-pushed the multi-instance-eks-node-groups branch from c5bdab4 to e7f8f41 Compare June 29, 2023 12:09

enxebre reviewed Jul 5, 2023

View reviewed changes

arjunrn force-pushed the multi-instance-eks-node-groups branch from a16dd53 to c151661 Compare July 5, 2023 07:37

arjunrn requested review from enxebre and richardcase July 6, 2023 09:09

k8s-ci-robot assigned enxebre Jul 6, 2023

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Jul 6, 2023

arjunrn force-pushed the multi-instance-eks-node-groups branch from c151661 to a5a5187 Compare July 6, 2023 10:18

k8s-ci-robot removed the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Jul 6, 2023

arjunrn closed this Aug 29, 2023

arjunrn deleted the multi-instance-eks-node-groups branch October 31, 2023 09:26

arjunrn restored the multi-instance-eks-node-groups branch October 31, 2023 11:07

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Multiple instance types for managed machine pools #4358

Multiple instance types for managed machine pools #4358

arjunrn commented Jun 26, 2023 •

edited

Loading

k8s-ci-robot commented Jun 26, 2023

k8s-ci-robot commented Jun 26, 2023

richardcase commented Jun 28, 2023

richardcase left a comment

richardcase Jun 28, 2023

arjunrn Jun 29, 2023

richardcase Jun 28, 2023

arjunrn Jun 29, 2023

enxebre Jun 28, 2023

arjunrn Jun 29, 2023

enxebre Jun 29, 2023 •

edited

Loading

richardcase Jun 30, 2023

arjunrn Jun 30, 2023

enxebre Jul 5, 2023 •

edited

Loading

arjunrn Jul 5, 2023

arjunrn Jul 5, 2023

enxebre commented Jul 5, 2023

enxebre commented Jul 6, 2023

k8s-ci-robot commented Jul 6, 2023

k8s-ci-robot commented Jul 6, 2023

arjunrn commented Jul 6, 2023

richardcase commented Aug 29, 2023

idanshaby commented Oct 31, 2023

arjunrn commented Oct 31, 2023

Multiple instance types for managed machine pools #4358

Multiple instance types for managed machine pools #4358

Conversation

arjunrn commented Jun 26, 2023 • edited Loading

k8s-ci-robot commented Jun 26, 2023

k8s-ci-robot commented Jun 26, 2023

richardcase commented Jun 28, 2023

richardcase left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

enxebre Jun 29, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

enxebre Jul 5, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

enxebre commented Jul 5, 2023

enxebre commented Jul 6, 2023

k8s-ci-robot commented Jul 6, 2023

k8s-ci-robot commented Jul 6, 2023

arjunrn commented Jul 6, 2023

richardcase commented Aug 29, 2023

idanshaby commented Oct 31, 2023

arjunrn commented Oct 31, 2023

arjunrn commented Jun 26, 2023 •

edited

Loading

enxebre Jun 29, 2023 •

edited

Loading

enxebre Jul 5, 2023 •

edited

Loading