Switching an existing ASG-based instance group to use instanceRequirements rather than instances fails

[danports]

/kind bug

1. What kops version are you running? The command kops version, will display
this information.
1.26.2

2. What Kubernetes version are you running? kubectl version will print the
version if a cluster is running or provide the Kubernetes version specified as
a kops flag.
1.26.3

3. What cloud provider are you using?
AWS

4. What commands did you run? What is the simplest way to reproduce this issue?
Deploy a cluster with an instance group that specifies some instance types under spec.mixedInstancesPolicy.instances. Then update the instance group manifest and replace that section with an instanceRequirements section, e.g.:

    instanceRequirements:
      cpu:
        min: "2"
        max: "4"
      memory:
        min: "4G"
        max: "8G"

5. What happened after the commands executed?
When running kops update cluster with the updated instance group manifest, it gets stuck in a failure loop:

W0406 18:22:13.945102    2600 autoscalinggroup.go:662] cannot apply changes to AutoScalingGroup: *awstasks.AutoscalingGroup {"Name":null,"Lifecycle":"","Granularity":null,"InstanceProtection":null,"LaunchTemplate":null,"LoadBalancers":null,"MaxInstanceLifetime":null,"MaxSize":null,"Metrics":null,"MinSize":null,"MixedInstanceOverrides":null,"InstanceRequirements":null,"MixedOnDemandAllocationStrategy":"lowest-price","MixedOnDemandBase":null,"MixedOnDemandAboveBase":null,"MixedSpotAllocationStrategy":null,"MixedSpotInstancePools":null,"MixedSpotMaxPrice":null,"Subnets":null,"SuspendProcesses":null,"Tags":null,"TargetGroups":null,"CapacityRebalance":null,"WarmPool":null}
W0406 18:22:14.198944    2600 executor.go:139] error running task "AutoscalingGroup/somegroup" (8m56s remaining to succeed): error updating AutoscalingGroup: ValidationError: The specified OnDemandAllocationStrategy is not valid. The prioritized allocation strategy is not compatible with instance requirements. Valid options are [lowest-price].

FWIW, it doesn't matter whether onDemandAllocationStrategy: lowest-price is included in the manifest - the error is the same.

6. What did you expect to happen?
The existing ASG should have been updated with the instance requirements specified.

7. Please provide your cluster manifest. Execute
kops get --name my.example.com -o yaml to display your cluster manifest.
You may want to remove your cluster name and other sensitive information.

See snippet above.

8. Please run the commands with most verbose logging by adding the -v 10 flag.
Paste the logs into this report, or in a gist and provide the gist link here.

See above.

9. Anything else do we need to know?

Creating a new instance group with instanceRequirements specified works fine. Switching an existing instance group to instanceRequirements also works fine if you delete the existing ASG before running kops update cluster.

[vashishkov]

Can confirm, have same issue

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[danports]

/remove-lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle rotten
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

[srolija]

We seem to have a similar issue. So just to check, what did you have as MixedOnDemandAllocationStrategy prior to apply? Was it the default prioritized?

In our case we created the group initially with MixedOnDemandAllocationStrategy=prioritized and then wanted to change it to MixedOnDemandAllocationStrategy=lowest-price but it seems like the reconciliation algorithm doesn't check for that.

There is a check for the spot instances here:

kops/upup/pkg/fi/cloudup/awstasks/autoscalinggroup.go

Lines 536 to 539 in db26ad5

	if changes.MixedSpotAllocationStrategy != nil {
	setup(request).InstancesDistribution.SpotAllocationStrategy = e.MixedSpotAllocationStrategy
	changes.MixedSpotAllocationStrategy = nil
	}

However, there is no such check for the MixedOnDemandAllocationStrategy which seems like the root cause why it is not detected.

Same seems to be the case here:

kops/upup/pkg/fi/cloudup/awstasks/autoscalinggroup.go

Lines 431 to 435 in db26ad5

	OnDemandPercentageAboveBaseCapacity: e.MixedOnDemandAboveBase,
	OnDemandBaseCapacity: e.MixedOnDemandBase,
	SpotAllocationStrategy: e.MixedSpotAllocationStrategy,
	SpotInstancePools: e.MixedSpotInstancePools,
	SpotMaxPrice: e.MixedSpotMaxPrice,

And here:

kops/upup/pkg/fi/cloudup/awstasks/autoscalinggroup.go

Lines 743 to 752 in db26ad5

	// @check if any of the mixed instance policies settings are toggled
	if e.MixedOnDemandAboveBase != nil {
	return true
	}
	if e.MixedOnDemandBase != nil {
	return true
	}
	if e.MixedSpotAllocationStrategy != nil {
	return true
	}

[srolija]

/remove-lifecycle rotten