CKS: new created nodes are not added to CKS cluster if scaling fails

[weizhouapache]

problem

this happens with ACS 4.22 and 4.21, but might also with older versions

• create CKS cluster (1 control node + 1 worker node) as regular user
• set resource limit (e.g. memory) of regular user, so that the user can deploy a new worker node, but not two worker nodes
• scale the CKS cluster with two more worker nodes (1 -> 3)
• a worker node is created, but not 2 due to resource limitation

expected (option 1):

• CKS cluster node size is increased to 2 (2 worker nodes)
• network rules are configured for the new worker node
• CKS cluster is Running state

expected (option 2):

• scaling fails
• No new worker node is deployed
• CKS cluster node size is unchanged (still 1)
• CKS cluster is Running state

Actual:

• a new worker node is deployed
• CKS cluster node size is still 1 (1 worker node)
• network rules are not configured for the new worker node
• CKS cluster is Alert state

versions

The versions of ACS, hypervisors, storage, network etc..

The steps to reproduce the bug

...

What to do about it?

No response

[weizhouapache]

I updated the description with two options of expectation
both are fine for me. cc @DaanHoogland @Pearl1594

expected (option 1):

CKS cluster node size is increased to 2 (2 worker nodes)
network rules are configured for the new worker node
CKS cluster is Running state

expected (option 2):
scaling fails
No new worker node is deployed
CKS cluster node size is unchanged (still 1)
CKS cluster is Running state

[DaanHoogland]

looking at #12167 , we are choosing option 2, right @Pearl1594 ? (fine by me)

[Pearl1594]

Yes @DaanHoogland selected option 2, because having the cluster partially scale may lead to end users believing it scaled and then later realizing that it didn't scale up to the nodes they expected it to. That was my thought when I chose option 2 to address the issue.