Control plane nodes not upgraded for same kube version

[jaxesn]

What happened:
Created a cluster with 0.9.2, kube version 1.22. Upgraded with 0.10.0, keeping cluster spec and kube version the same. Worker nodes were updated to new kube version but control plane node was not.

jgw-vsphere-6nx52                  Ready    control-plane,master   49m     v1.22.6-eks-bb942e6    195.17.16.78     195.17.16.78     Ubuntu 20.04.4 LTS   5.4.0-110-generic   containerd://1.5.9
jgw-vsphere-md-0-d84fd4cfc-z6mn5   Ready    <none>                 9m42s   v1.22.10-eks-7dc61e8   195.17.130.106   195.17.130.106   Ubuntu 20.04.4 LTS   5.4.0-120-generic   containerd://1.5.9

What you expected to happen:

Control plane nodes should also be upgraded.

How to reproduce it (as minimally and precisely as possible):

Anything else we need to know?:

Environment:

• EKS Anywhere Release: 0.9.2/0.10.0
• EKS Distro Release: 1.22

[jaxesn]

There is only one vspheremachinetemplate created for cp, there should be two since the upgrade flow should have created a new one with the new template

  ~/testing/cli/5-14-22-vsphere ❯ k --kubeconfig jgw-vsphere/jgw-vsphere-eks-a-cluster.kubeconfig get VSphereMachineTemplate -A                                                                                                                                                                                                                        dev at  12:34:23 PM
NAMESPACE     NAME                                               AGE
eksa-system   jgw-vsphere-control-plane-template-1656602683095   67m
eksa-system   jgw-vsphere-etcd-template-1656602683095            67m
eksa-system   jgw-vsphere-etcd-template-1656605890695            67m
eksa-system   jgw-vsphere-md-0-1656602683097                     67m
eksa-system   jgw-vsphere-md-0-1656605889612                     67m

[jaxesn]

I used only one VSphereMachineConfig for both the cp and worker nodes, the working theory is this is could be the issue. @jiayiwang7 tested with separate configs and it worked correctly.

[jiayiwang7]

tested with same VSphereMachineConfig for cp and worker nodes, and stacked etcd. Upgrade works fine.

[jiayiwang7]

Tested with same VSphereMachineConfig for cp, worker and unstacked etcd. Upgrade works correctly..

create w 0.9.2

➜  eks-anywhere git:(main) ✗ k get nodes
NAME                            STATUS   ROLES                  AGE     VERSION
joeywang-md-0-b7c74cd96-vj6m4   Ready    <none>                 4m37s   v1.22.6-eks-bb942e6
joeywang-nk9br                  Ready    control-plane,master   6m10s   v1.22.6-eks-bb942e6

upgrade w 0.10.0

➜  eks-anywhere git:(main) ✗ k get nodes
NAME                             STATUS   ROLES                  AGE     VERSION
joeywang-fn6tb                   Ready    control-plane,master   3m8s    v1.22.10-eks-7dc61e8
joeywang-md-0-694c564b9c-m4tl4   Ready    <none>                 9m13s   v1.22.10-eks-7dc61e8

➜  eks-anywhere git:(main) ✗ k get vspheremachinetemplates -A
NAMESPACE     NAME                                            AGE
eksa-system   joeywang-control-plane-template-1656614863460   78s
eksa-system   joeywang-control-plane-template-1656616374263   78s
eksa-system   joeywang-etcd-template-1656614863460            80s
eksa-system   joeywang-etcd-template-1656616377314            79s
eksa-system   joeywang-md-0-1656614863463                     78s
eksa-system   joeywang-md-0-1656616375887                     82s