The same thing happened for one of our EKS clusters. here's the full output for a loop:

I1002 14:50:40.757207       1 reflector.go:790] k8s.io/client-go/informers/factory.go:150: Watch close - *v1.ReplicaSet total 112 items received
I1002 14:50:42.157028       1 reflector.go:790] k8s.io/client-go/informers/factory.go:150: Watch close - *v1.PersistentVolumeClaim total 8 items received
I1002 14:51:09.451280       1 static_autoscaler.go:287] Starting main loop
I1002 14:51:09.451449       1 auto_scaling_groups.go:393] Regenerating instance to ASG map for ASG names: []
I1002 14:51:09.451463       1 auto_scaling_groups.go:400] Regenerating instance to ASG map for ASG tags: map[k8s.io/cluster-autoscaler/enabled: k8s.io/cluster-autoscaler/public:]
I1002 14:51:09.586582       1 auto_scaling_groups.go:142] Updating ASG eks-x86-m7i-flex-xlarge-f6c4dd96-da24-ca3f-2824-ffbc8b741746
I1002 14:51:09.586626       1 aws_wrapper.go:703] 0 launch configurations to query
I1002 14:51:09.586632       1 aws_wrapper.go:704] 0 launch templates to query
I1002 14:51:09.586638       1 aws_wrapper.go:724] Successfully queried 0 launch configurations
I1002 14:51:09.586645       1 aws_wrapper.go:735] Successfully queried 0 launch templates
I1002 14:51:09.586651       1 aws_wrapper.go:746] Successfully queried instance requirements for 0 ASGs
I1002 14:51:09.586659       1 aws_manager.go:129] Refreshed ASG list, next refresh after 2023-10-02 14:52:09.586657471 +0000 UTC m=+375178.448258144
I1002 14:51:09.588859       1 aws_manager.go:185] Found multiple availability zones for ASG "eks-x86-m7i-flex-xlarge-f6c4dd96-da24-ca3f-2824-ffbc8b741746"; using us-east-1a for failure-domain.beta.kubernetes.io/zone label
I1002 14:51:09.589066       1 static_autoscaler.go:709] Decreasing size of eks-x86-m7i-flex-xlarge-f6c4dd96-da24-ca3f-2824-ffbc8b741746, expected=12 current=11 delta=-1
E1002 14:51:09.589091       1 static_autoscaler.go:439] Failed to fix node group sizes: failed to decrease eks-x86-m7i-flex-xlarge-f6c4dd96-da24-ca3f-2824-ffbc8b741746: attempt to delete existing nodes targetSize:12 delta:-1 existingNodes: 12

There's no obvious actions taken on our side that caused the problem to appear.

When this has happened to us, it seems to have helped if I changed the number of instances in the node group manually, which then would get the cluster autoscaler to be able to correct the size again.
It does however seem likely to reoccur again a couple of hours later.

My hypothesis so far is that this might be related to the cluster using the m7i-flex family of instances which aren't necessarily available in all availability zones. That can cause the node group to be set to for instance have 5 instances in it, but there will only be 4 created by AWS because the node group is configured to make use of subnets/AZs where it can't create instances.

I've now replaced the node group with a new one that only has subnets/AZs enabled that has this family of instances available and hope that might help.

@com6056 Any chance you might be in a similar situation?

We aren't using those instance types, so I don't think that is what is causing it (at least for us). We have ASGs with a mixed instance type policy and there should usually be instances available (and if not, it should just fail and fallback to a different node group).

Same issue, I use AWS as provider and kubernetes 1.28:
attempt to delete existing nodes targetSize:13 delta:-1 existingNodes: 13
From what I noticed in function DecreaseTargetSize it doesn't receive correct actual group size with this line of code:
nodes, err := ng.awsManager.GetAsgNodes(ng.asg.AwsRef)

After downgrade of cluster-autoscaler from1.28 to 1.27.3 it started to work ok.

We are seeing this issue in our environments too and we are doing nothing special with instance types, at one point after a few reboots of autoscaler it worked for a little while then stopped working again.

Currently downgrading as suggested by @mykhailogorsky

@x13n could this be caused by #5976 enabling parallel drain by default? I guess I could try 1.28 again with -parallel-drain=false to confirm 👀

Nope still hitting it, even with --parallel-drain="false":

cluster-autoscaler-aws-6d95d65-hbgdb cluster-autoscaler I1115 19:34:18.815122       1 static_autoscaler.go:709] Decreasing size of build-32-128, expected=5 current=1 delta=-4
cluster-autoscaler-aws-6d95d65-hbgdb cluster-autoscaler E1115 19:34:18.815135       1 static_autoscaler.go:439] Failed to fix node group sizes: failed to decrease build-32-128: attempt to delete existing nodes targetSize:5 delta:-4 existingNodes: 5

We have seen no issues since our downgrade to v1.27.3 and things have been operating as we expect, seems like it is an issue introduced in 1.18.0.

We are also seeing this in v1.28.0

This is unrelated to scale down logic - this is called only to reduce target size on a node group without actually deleting any nodes. The error suggests that it would actually require deleting existing VMs to reduce the node group size. Not sure why this started in 1.28. As a data point, this may be AWS specific - I haven't seen this error on GKE.

We seen the same issue with 1.28.1. But only when deleting a node manually from from the cluster.
After some time the autoscaler then tries to change the number of nodes in the node group and then fails.

Got the same issue today. I found out that we had more instances in our ASG than there were Nodes in Kubernetes. After finding the instance in EC2 which had not joined the cluster as Node and terminating the instance the cluster-autoscaler started to recover again.

Seeing the same issue here. Terminating the instance that failed to join also fixes it.

Same issue on 1.28.2 on AWS.

Terminated the instance that was running but not part of the cluster. (ASG seems to have created a new node then, which did join the cluster)

cluster-autoscaler was restarting with failed health checks (apparently getting 500s) and had the log messages as above.

The culprit is the implementation of aws provider：

func (ng *AwsNodeGroup) DecreaseTargetSize(delta int) error {
	if delta >= 0 {
		return fmt.Errorf("size decrease size must be negative")
	}

	size := ng.asg.curSize
	nodes, err := ng.awsManager.GetAsgNodes(ng.asg.AwsRef)
	if err != nil {
		return err
	}
	if int(size)+delta < len(nodes) {
		return fmt.Errorf("attempt to delete existing nodes targetSize:%d delta:%d existingNodes: %d",
			size, delta, len(nodes))
	}
	return ng.awsManager.SetAsgSize(ng.asg, size+delta)
}

which size is almost always equal to len(nodes) because aws nodes are composed of active nodes and fake nodes (place-holder), they are always equal to the target size.

related issue: #5829

The culprit is the implementation of aws provider：

func (ng *AwsNodeGroup) DecreaseTargetSize(delta int) error {
	if delta >= 0 {
		return fmt.Errorf("size decrease size must be negative")
	}

	size := ng.asg.curSize
	nodes, err := ng.awsManager.GetAsgNodes(ng.asg.AwsRef)
	if err != nil {
		return err
	}
	if int(size)+delta < len(nodes) {
		return fmt.Errorf("attempt to delete existing nodes targetSize:%d delta:%d existingNodes: %d",
			size, delta, len(nodes))
	}
	return ng.awsManager.SetAsgSize(ng.asg, size+delta)
}

which size is almost always equal to len(nodes) because aws nodes are composed of active nodes and fake nodes (place-holder), they are always equal to the target size.

Maybe we can filter out active running nodes, ignore these stale place-holder fake nodes which status == placeholderUnfulfillableStatus, and make sure asg target size could converge to active nodes

for example:

func (ng *AwsNodeGroup) DecreaseTargetSize(delta int) error {
	if delta >= 0 {
		return fmt.Errorf("size decrease size must be negative")
	}

	size := ng.asg.curSize
	nodes, err := ng.awsManager.GetAsgNodes(ng.asg.AwsRef)
	if err != nil {
		return err
	}

	// filter out active nodes, ignore these stale place-holder fake nodes which status == placeholderUnfulfillableStatus
	// make sure asg target size could converge to active nodes
	filteredNodes := make([]AwsInstanceRef, 0)
	for i := range nodes {
		node := nodes[i]
		instanceStatus, err := ng.awsManager.GetInstanceStatus(node)
		if err != nil {
			klog.V(4).Infof("Could not get instance status, continuing anyways: %v", err)
		} else if instanceStatus != nil && *instanceStatus == placeholderUnfulfillableStatus {
			continue
		}
		filteredNodes = append(filteredNodes, node)
	}
	nodes = filteredNodes

	if int(size)+delta < len(nodes) {
		return fmt.Errorf("attempt to delete existing nodes targetSize:%d delta:%d existingNodes: %d",
			size, delta, len(nodes))
	}
	return ng.awsManager.SetAsgSize(ng.asg, size+delta)
}

FWIW, we're stuck with this bug as well; downgrading to 1.27.3 isn't a great option for us:

uses the unknown EC2 instance type "m7i.48xlarge"

Hello, I had exactly the same problem running in 1.29 k8s and 1.29.0 image version for cluster-autoscaler. One node was in problem and all our system wasn't scalated due to this reason during lot of hours.

Seeing this as well with cluster autoscaler@1.28.2 on EKS@1.28. If the bootstrap script (/etc/eks/bootstrap.sh) fails, the node will just sit in the ASG and all CA scaling up/down will be stopped.

I observed a similar issue. The initial scale-up timeout caused the error to start appearing. The instance was eventually scaled up, but it wasn't registered as a Kubernetes node. When I deleted the node manually, issue has been resolved. I believe it was related to the low availability of a particular instance type (g4dn.4xlarge). EKS 1.28 with 1.28.2 cluster-autoscaler.

I0313 12:34:34.470823       1 executor.go:147] Scale-up: setting group eks-mli_inference_g4dn4xl-02c6fc6f-788b-c2ae-e9d5-647dfa039598 size to 7
I0313 12:34:34.470926       1 auto_scaling_groups.go:265] Setting asg eks-mli_inference_g4dn4xl-02c6fc6f-788b-c2ae-e9d5-647dfa039598 size to 7
...
W0313 12:49:40.563382       1 clusterstate.go:266] Scale-up timed out for node group eks-mli_inference_g4dn4xl-02c6fc6f-788b-c2ae-e9d5-647dfa039598 after 15m5.500491908s
W0313 12:49:40.584456       1 clusterstate.go:297] Disabling scale-up for node group eks-mli_inference_g4dn4xl-02c6fc6f-788b-c2ae-e9d5-647dfa039598 until 2024-03-13 12:54:40.13918732 +0000 UTC m=+2030.419314682; errorClass=Other; errorCode=timeout
W0313 12:50:01.422358       1 orchestrator.go:582] Node group eks-mli_inference_g4dn4xl-02c6fc6f-788b-c2ae-e9d5-647dfa039598 is not ready for scaleup - backoff
W0313 12:50:22.047950       1 orchestrator.go:582] Node group eks-mli_inference_g4dn4xl-02c6fc6f-788b-c2ae-e9d5-647dfa039598 is not ready for scaleup - backoff
I0313 12:50:32.503066       1 static_autoscaler.go:709] Decreasing size of eks-mli_inference_g4dn4xl-02c6fc6f-788b-c2ae-e9d5-647dfa039598, expected=7 current=6 delta=-1
E0313 12:50:32.503102       1 static_autoscaler.go:439] Failed to fix node group sizes: failed to decrease eks-mli_inference_g4dn4xl-02c6fc6f-788b-c2ae-e9d5-647dfa039598: attempt to delete existing nodes targetSize:7 delta:-1 existingNodes: 7

...
NOW I DELETED THE NODE FROM EC2 CONSOLE
...

E0313 15:15:27.466600       1 static_autoscaler.go:439] Failed to fix node group sizes: failed to decrease eks-mli_inference_g4dn4xl-02c6fc6f-788b-c2ae-e9d5-647dfa039598: attempt to delete existing nodes targetSize:7 delta:-1 existingNodes: 7
I0313 15:15:37.796147       1 static_autoscaler.go:709] Decreasing size of eks-mli_inference_g4dn4xl-02c6fc6f-788b-c2ae-e9d5-647dfa039598, expected=7 current=6 delta=-1
E0313 15:15:37.796180       1 static_autoscaler.go:439] Failed to fix node group sizes: failed to decrease eks-mli_inference_g4dn4xl-02c6fc6f-788b-c2ae-e9d5-647dfa039598: attempt to delete existing nodes targetSize:7 delta:-1 existingNodes: 7
I0313 15:15:48.264982       1 static_autoscaler.go:709] Decreasing size of eks-mli_inference_g4dn4xl-02c6fc6f-788b-c2ae-e9d5-647dfa039598, expected=7 current=6 delta=-1
E0313 15:15:48.265018       1 static_autoscaler.go:439] Failed to fix node group sizes: failed to decrease eks-mli_inference_g4dn4xl-02c6fc6f-788b-c2ae-e9d5-647dfa039598: attempt to delete existing nodes targetSize:7 delta:-1 existingNodes: 8
I0313 15:15:58.763859       1 static_autoscaler.go:709] Decreasing size of eks-mli_inference_g4dn4xl-02c6fc6f-788b-c2ae-e9d5-647dfa039598, expected=7 current=6 delta=-1
E0313 15:15:58.763892       1 static_autoscaler.go:439] Failed to fix node group sizes: failed to decrease eks-mli_inference_g4dn4xl-02c6fc6f-788b-c2ae-e9d5-647dfa039598: attempt to delete existing nodes targetSize:7 delta:-1 existingNodes: 8

NO MORE ENTRIES RELATED TO THIS ISSUE

I believe the fundamental issue is that cluster-autoscaler does not handle instances without node objects, that is, it does not support deleting nodes when the instance is still running, and similarly, does not handle instances that fail to produce a node object in the first place (e.g., misconfigured launch templates).

Nodes are not supposed to be deleted manually. The kubelet creates the initial node object and keeps it updated, but does not react to the node object being removed. Cluster-autoscaler uses cloud APIs to scale up ASGs when needed or shutdown instances when drained, but does not intervene in node object lifecycle. The cloud node controller is supposed to remove the node object only after the instance is shutdown and relinquished, but has no way to tell whether a still-running instance is supposed to be a node of the cluster. So, manually deleting a node creates a zombie instance. (The control plane responds by redirecting service traffic and rescheduling workloads elsewhere, but the instance is left running.)

Ideally kubelet should recreate the missing node object if necessary, but it currently doesn't. Then cluster-autoscaler starts getting confused by the sustained mismatch between ASG sizes and the node object inventory. This initially presents as cluster-autoscaler's Failed to find readiness information for <ASG> message, while tending to leave pods indefinitely in a pending state (as if it thinks a scale-up is already in progress). If later the cluster manages to satisfy capacity due to some other trigger (either by manually bumping up the ASG desired size, or waiting for a transient workload spike to scale the cluster up further, or if enough pods just happen to finish) then all workloads may be restored to health, but nonetheless (after a scaling delay) cluster-autoscaler instead starts emitting Failed to fix node group sizes: failed to decrease <ASG>... messages containing the telltale clue that targetSize == existingNodes while delta < 0.

To actually clean up the problem you need to manually identify the zombie instances and individually terminate them (or, more disruptively, just bump that ASG's desired size to zero briefly - then manually bump it back again in case that's the same ASG that cluster-autoscaler itself is run on). And then stop letting people manually delete nodes (nor run untested configs).

A possible fix here would be, for all ASGs that cluster-autoscaler is configured to manage, to have cluster-autoscaler terminate any instance beyond a certain age if it doesn't correspond to a node object. (This would treat them like failed start-ups. An alternative solution would be for kubelet to recreate the node object it syncs to whenever necessary. Another alternative would be if kubelet reacted by exiting and shutting down its host, letting the ASG clean up such instances.)

Note this proposed fix will also clean up instances that fail to connect to the cluster, which is precisely the source of the problem for some of the reports above in this thread. (If the ASG launch template is misconfigured for the cluster, it makes sense that cluster-autoscaler should be allowed to periodically retry creating the instance, effectively checking for corrections to that configuration.)

We noticed this with a spot instance node group on EKS. Could a preempted instance also trigger this behavior?

FYI, we are also seeing this with one of our users. It starts again with the timeout.

05:26:42.532720       1 clusterstate.go:266] Scale-up timed out for node group eks-something-worker-80c733af6-1e9e-534e-a483-918f39b0bc05 after 15m2.467913257s

We observed the same issue with cluster autoscaler 1.28.0. It started with a timeout error. The instance got created in ASG but was not registered as a node in kubernetes. I attempted to recreate the issue which was difficult to do so I tried a different method of getting the same error which was deleting a node manually on kubernetes.
For Cluster autoscaler 1.28.0:
I manually deleted one of the nodes that a worker pod was running on. The instance is still present in ASG but this deletes in corresponding node in k8. The pod running on the deleted node crashes first but then goes into an indefinite pending state. Autoscaler attempts to decrease the size of the nodegroup but is unable to and it leads to the "Failed to fix node group size error".

I tested the same scenario on autoscaler 1.27.3. I manually deleted one of the nodes that a worker pod was running on. The instance is still present in ASG but this deletes in corresponding node in k8. This led to the pod crashing and then going to pending state. However the cluster autoscaler recognized the zombie instance and printed a message in the logs 1 unregistered node present. The autoscaler deleted the zombie instance and created a new one and also created a new node in kubernetes. The autoscaler resolved the problem on its own.

In 1.27.3 autoscaler gets unregistered nodes correctly hence it is able to identify zombie instances.

func getNotRegisteredNodes(allNodes []*apiv1.Node, cloudProviderNodeInstances map[string][]cloudprovider.Instance, time time.Time) []UnregisteredNode {
	registered := sets.NewString()
	for _, node := range allNodes {
		registered.Insert(node.Spec.ProviderID)
	}
	notRegistered := make([]UnregisteredNode, 0)
	for _, instances := range cloudProviderNodeInstances {
		for _, instance := range instances {
			if !registered.Has(instance.Id) {
				notRegistered = append(notRegistered, UnregisteredNode{
					Node:              fakeNode(instance, cloudprovider.FakeNodeUnregistered),
					UnregisteredSince: time,
				})
			}
		}
	}
	return notRegistered
}

Just had this occur on one of our EKS 1.28 clusters, fixed it by:

• Look at the node group node list to get current instance names (double confirm instances in the node group within k8s directly, these should 100% match) - in our case this node group had 4 actual in-service nodes
• Went into the ASG directly and to the Instance Management tab - in our case this showed 5 in service instances.
• Look at each instance's details in EC2 directly to check the names/IP's, and in our case one of them in the ASG didn't have a corresponding node in EKS/K8s
• Detach that instance from the ASG - warning about "ASG may replace this instance", but it didn't.
• Instance expected counts now match between ASG and EKS/K8s
• Errors on the cluster-autoscaler pod stopped and normal operation resumed on it's own, once it had picked up the ASG updated status

Everything looks stabilized for this cluster now and can probably repeat if it does return.

Just had this occur on one of our EKS 1.28 clusters, fixed it by:

• Look at the node group node list to get current instance names (double confirm instances in the node group within k8s directly, these should 100% match) - in our case this node group had 4 actual in-service nodes
• Went into the ASG directly and to the Instance Management tab - in our case this showed 5 in service instances.
• Look at each instance's details in EC2 directly to check the names/IP's, and in our case one of them in the ASG didn't have a corresponding node in EKS/K8s
• Detach that instance from the ASG - warning about "ASG may replace this instance", but it didn't.
• Instance expected counts now match between ASG and EKS/K8s
• Errors on the cluster-autoscaler pod stopped and normal operation resumed on it's own, once it had picked up the ASG updated status

Everything looks stabilized for this cluster now and can probably repeat if it does return.

This method works fine for EKS 1.29 also, thanks.

I'd add some tips how to deduce this "cattle out of herd".

Error is:
static_autoscaler.go:449] Failed to fix node group sizes: failed to decrease gtlb: attempt to delete existing nodes targetSize:2 delta:-1 existingNodes: 2

To see what CAS "thinks" about this ASG see Configmap cluster-autoscaler-status. Something like this:
ScaleUp: NoActivity (ready=1 cloudProviderTarget=2 ...

Than see instance list for this ASG. There should be one spare, ready=1 from the ConfigMap is wrong. To check which one I used Annotations on Nodes csi.volume.kubernetes.io/nodeid={"ebs.csi.aws.com":"i-abcdabcddfv345"}. Compare to ASG list - got instance(s) to detach.

Open question is what circumstances lead to such a situation with manual fix needed. Will try 1.29.2.

Just had this occur on one of our EKS 1.28 clusters, fixed it by:

• Look at the node group node list to get current instance names (double confirm instances in the node group within k8s directly, these should 100% match) - in our case this node group had 4 actual in-service nodes
• Went into the ASG directly and to the Instance Management tab - in our case this showed 5 in service instances.
• Look at each instance's details in EC2 directly to check the names/IP's, and in our case one of them in the ASG didn't have a corresponding node in EKS/K8s
• Detach that instance from the ASG - warning about "ASG may replace this instance", but it didn't.
• Instance expected counts now match between ASG and EKS/K8s
• Errors on the cluster-autoscaler pod stopped and normal operation resumed on it's own, once it had picked up the ASG updated status

Everything looks stabilized for this cluster now and can probably repeat if it does return.

Thanks for help, these solution working perfectly.

Just had this occur on one of our EKS 1.28 clusters, fixed it by:

* Look at the node group node list to get current instance names (double confirm instances in the node group within k8s directly, these should 100% match) - in our case this node group had 4 actual in-service nodes

* Went into the ASG directly and to the Instance Management tab - in our case this showed 5 in service instances.

* Look at each instance's details in EC2 directly to check the names/IP's, and in our case one of them in the ASG didn't have a corresponding node in EKS/K8s

* Detach that instance from the ASG - warning about "ASG may replace this instance", but it didn't.

* Instance expected counts now match between ASG and EKS/K8s

* Errors on the cluster-autoscaler pod stopped and normal operation resumed on it's own, once it had picked up the ASG updated status

Everything looks stabilized for this cluster now and can probably repeat if it does return.

Thank you! This works perfectly running EKS 1.28 and CA 1.28.2. Any plan to work on a fix? Thanks in advance.

After encountering this issue on EKS 1.28 and CA 1.28.2, we reproduced the issue as described above by manually removing one of the Kubernetes nodes.
Taking a look at the difference in behavior with 1.27.3, I think the issue is caused by this change:
e5bc070#diff-d404c1cb5f21589e42a6372990c673fcb42738f61d35bda2f5eeccdd0a7c3abeL1000-R1002

When the autoscaler is able to recover from this disconnect between ASG state and Kubernetes state, you will see this in the log:

I0504 14:22:56.465489       1 static_autoscaler.go:405] 1 unregistered nodes present

This message is never present in 1.28.x versions.

To test this, I modified the getNotRegisteredNodes method to expand the check:

if (!registered.Has(instance.Id) && expRegister) || (!registered.Has(instance.Id) && instance.Status == nil)

And sure enough, on the next run of the cluster-autoscaler:

I0504 14:22:56.465489       1 static_autoscaler.go:405] 1 unregistered nodes present
...
I0504 14:25:57.187609       1 clusterstate.go:633] Found longUnregistered Nodes [aws:///us-east-1a/i-1234ab5678901a12b]
I0504 14:25:57.187651       1 static_autoscaler.go:405] 1 unregistered nodes present
I0504 14:25:57.187662       1 static_autoscaler.go:746] Marking unregistered node aws:///us-east-1a/i-1234ab5678901a12b for removal
I0504 14:25:57.187678       1 static_autoscaler.go:755] Removing 1 unregistered nodes for node group general
I0504 14:25:57.407027       1 auto_scaling_groups.go:343] Terminating EC2 instance: i-1234ab5678901a12b
I0504 14:25:57.407055       1 aws_manager.go:162] DeleteInstances was called: scheduling an ASG list refresh for next main loop evaluation
I0504 14:25:57.407118       1 static_autoscaler.go:413] Some unregistered nodes were removed

We also encountered this problem, and I refactored part of the logic. It can solve this problem very well #6729. Works well on our large cluster.

For AWS, instance has only two states. (expectedToRegister is incorrect)

1. instance.Status = nil
2. instance.State = InstanceCreating and instance.State.ErrorInfo = OutOfResourcesErrorClass

Therefore removeOldUnregisteredNodes will be invalid. Finally enter fixNodeGroupSize and report an error.

Just had this occur on one of our EKS 1.28 clusters, fixed it by:

* Look at the node group node list to get current instance names (double confirm instances in the node group within k8s directly, these should 100% match) - in our case this node group had 4 actual in-service nodes

* Went into the ASG directly and to the Instance Management tab - in our case this showed 5 in service instances.

* Look at each instance's details in EC2 directly to check the names/IP's, and in our case one of them in the ASG didn't have a corresponding node in EKS/K8s

* Detach that instance from the ASG - warning about "ASG may replace this instance", but it didn't.

* Instance expected counts now match between ASG and EKS/K8s

* Errors on the cluster-autoscaler pod stopped and normal operation resumed on it's own, once it had picked up the ASG updated status

Everything looks stabilized for this cluster now and can probably repeat if it does return.

I am using EKS 1.29 with CA 1.29.2, tried this but i detach the instance, ASG automatically adds newer instances which are also not present in the cluster and i ended up with more extra zombie instances, I uncheck the instance replacement option and the instance is k8s starts to disappear.

any ETA on fix?