fail to scale EKS anywhere cluster for bare metal

[ygao-armada]

What happened:
I try to scale my cluster (from 1 cp node to 3 cp nodes out of 4 cp nodes, hardware.csv has 4 nodes, and I want to use 3 of them as cp nodes) with command:

eksctl anywhere upgrade cluster 
-f cluster.yaml 
--hardware-csv hardware.csv
--kubeconfig mgmt/mgmt-eks-a-cluster.kubeconfig

If I keep cp count 1 in the cluster.yaml, the command completes in 2 minutes, however, nothing change, even I don't see new hardware with follow command:
kubectl get hardware -n eksa-system --show-labels

If I change cp count to 3 in the cluster.yaml, I keep seeing this message:

2024-02-15T17:26:49.783Z    V6   Executing command    {"cmd": "/usr/bin/docker exec -i eksa_1708017136282361003 kubectl get --ignore-not-found -o json --kubeconfig mgmt/mgmt-eks-a-cluster.kubeconfig Cluster.v1alpha1.anywhere.eks.amazonaws.com --namespace default mgmt02"}
2024-02-15T17:26:49.884Z    V9   Cluster generation and observedGeneration    {"Generation": 2, "ObservedGeneration": 1}
2024-02-15T17:26:49.884Z    V5   Error happened during retry   {"error": "cluster generation (2) and observedGeneration (1) differ", "retries": 782}
2024-02-15T17:26:49.884Z    V5   Sleeping before next retry   {"time": "1s"}

What you expected to happen:
I expect the command to succeed with new hardware added

How to reproduce it (as minimally and precisely as possible):

Anything else we need to know?:

Environment:

• EKS Anywhere Release: v0.18.2
• EKS Distro Release: 1.26

[ph-armada]

Hi admin - any update? Thanks.

[ph-armada]

Hi - EKSA team - may I know if you have a Slack channel or something that's more live for Q&A aside from GitHub issue triage as well? Thanks!