[Release-1.28] - manual upgrade of rke2 on arm setup gets several pods restarting and alternate between running/crashloopbackoff state

[dereknola]

Backport fix for manual upgrade of rke2 on arm setup gets several pods restarting and alternate between running/crashloopbackoff state

• manual upgrade of rke2 on arm setup gets several pods restarting and alternate between running/crashloopbackoff state #6235

[aganesh-suse]

Verified with release-1.28 branch with version: v1.28.11-rc7+rke2r1

rke2 -v
rke2 version v1.28.11-rc7+rke2r1 (6b12d7a783238b72da8450fa1b6ec587cebb79ed)
go version go1.21.11 X:boringcrypto

kubectl get nodes
NAME              STATUS   ROLES                       AGE   VERSION
ip-172-31-12-8    Ready    control-plane,etcd,master   21m   v1.28.11+rke2r1
ip-172-31-6-234   Ready    control-plane,etcd,master   20m   v1.28.11+rke2r1
ip-172-31-7-6     Ready    <none>                      20m   v1.28.11+rke2r1
ip-172-31-9-39    Ready    control-plane,etcd,master   23m   v1.28.11+rke2r1

kubectl get pods -A
NAMESPACE        NAME                                                   READY   STATUS      RESTARTS      AGE
dnsutils         dnsutils                                               1/1     Running     0             16m
kube-system      cloud-controller-manager-ip-172-31-12-8                1/1     Running     0             14m
kube-system      cloud-controller-manager-ip-172-31-6-234               1/1     Running     0             13m
kube-system      cloud-controller-manager-ip-172-31-9-39                1/1     Running     1 (12m ago)   12m
kube-system      etcd-ip-172-31-12-8                                    1/1     Running     0             14m
kube-system      etcd-ip-172-31-6-234                                   1/1     Running     0             13m
kube-system      etcd-ip-172-31-9-39                                    1/1     Running     0             12m
kube-system      helm-install-rke2-canal-vzbv8                          0/1     Completed   0             14m
kube-system      helm-install-rke2-coredns-85ztv                        0/1     Completed   0             14m
kube-system      helm-install-rke2-ingress-nginx-l762n                  0/1     Completed   1             14m
kube-system      helm-install-rke2-metrics-server-r89bx                 0/1     Completed   1             14m
kube-system      helm-install-rke2-snapshot-controller-9sxll            0/1     Completed   1             14m
kube-system      helm-install-rke2-snapshot-controller-crd-7qpph        0/1     Completed   1             14m
kube-system      helm-install-rke2-snapshot-validation-webhook-8nk78    0/1     Completed   2             14m
kube-system      kube-apiserver-ip-172-31-12-8                          1/1     Running     0             14m
kube-system      kube-apiserver-ip-172-31-6-234                         1/1     Running     0             13m
kube-system      kube-apiserver-ip-172-31-9-39                          1/1     Running     0             12m
kube-system      kube-controller-manager-ip-172-31-12-8                 1/1     Running     1 (14m ago)   14m
kube-system      kube-controller-manager-ip-172-31-6-234                1/1     Running     1 (13m ago)   13m
kube-system      kube-controller-manager-ip-172-31-9-39                 1/1     Running     1 (12m ago)   12m
kube-system      kube-proxy-ip-172-31-12-8                              1/1     Running     0             14m
kube-system      kube-proxy-ip-172-31-6-234                             1/1     Running     0             13m
kube-system      kube-proxy-ip-172-31-7-6                               1/1     Running     0             12m
kube-system      kube-proxy-ip-172-31-9-39                              1/1     Running     0             11m
kube-system      kube-scheduler-ip-172-31-12-8                          1/1     Running     0             14m
kube-system      kube-scheduler-ip-172-31-6-234                         1/1     Running     0             13m
kube-system      kube-scheduler-ip-172-31-9-39                          1/1     Running     0             12m
kube-system      rke2-canal-29hmg                                       2/2     Running     0             12m
kube-system      rke2-canal-fklf4                                       2/2     Running     0             13m
kube-system      rke2-canal-hklw5                                       2/2     Running     0             12m
kube-system      rke2-canal-jf5vz                                       2/2     Running     1 (13m ago)   13m
kube-system      rke2-coredns-rke2-coredns-84b9cb946c-n976d             1/1     Running     0             21m
kube-system      rke2-coredns-rke2-coredns-84b9cb946c-tqcvp             1/1     Running     0             23m
kube-system      rke2-coredns-rke2-coredns-autoscaler-b49765765-jg2f6   1/1     Running     0             23m
kube-system      rke2-ingress-nginx-controller-5txgm                    1/1     Running     0             10m
kube-system      rke2-ingress-nginx-controller-gblsq                    1/1     Running     0             11m
kube-system      rke2-ingress-nginx-controller-nhb8l                    1/1     Running     0             9m55s
kube-system      rke2-ingress-nginx-controller-rp58j                    1/1     Running     0             9m1s
kube-system      rke2-metrics-server-655477f655-xcm54                   1/1     Running     0             22m
kube-system      rke2-snapshot-controller-59cc9cd8f4-pnnm8              1/1     Running     1 (13m ago)   22m
kube-system      rke2-snapshot-validation-webhook-54c5989b65-hbt69      1/1     Running     0             22m
test-clusterip   test-clusterip-674bf76598-7lzcx                        1/1     Running     0             16m
test-clusterip   test-clusterip-674bf76598-jz8dh                        1/1     Running     0             16m
test-daemonset   test-daemonset-4z4zb                                   1/1     Running     0             16m
test-daemonset   test-daemonset-5dlz7                                   1/1     Running     0             16m
test-daemonset   test-daemonset-l7485                                   1/1     Running     0             16m
test-daemonset   test-daemonset-tz4bn                                   1/1     Running     0             16m
test-ingress     test-ingress-kdz8q                                     1/1     Running     0             16m
test-ingress     test-ingress-tx4bm                                     1/1     Running     0             16m