Custom clusters broken in k3s HA install

[aaronyeeski]

What kind of request is this (question/bug/enhancement/feature request):
Bug

Steps to reproduce (least amount of steps as possible):
Steps:
Install Rancher master-head in HA using k3s nodes.
https://staging.rancher.com/docs/rancher/v2.x/en/installation/k8s-install/create-nodes-lb/
Rancher install command:

helm install rancher rancher-alpha/rancher --namespace cattle-system --set hostname=<hostname> --version 2.4.0-alpha1  --set rancherImageTag=master-head

In master-head, provision a custom 3 node cluster using Linux instances (3 nodes, all roles each)

Result:
Cluster is stuck Updating with

[workerPlane] Failed to upgrade Worker Plane: [host ip-172-31-0-173 not ready]; [workerPlane] Failed to upgrade Worker Plane: [host ip-172-31-0-173 not ready]

The issue is a provisioning run that doesn't succeed in the first try, will leave nodes in NotReady, which then will block the next provisioning run because it needs all nodes as Ready when there is an upgrade.

Other details that may be helpful:

Environment information

• Rancher version (rancher/rancher/rancher/server image tag or shown bottom left in the UI):
  rancher\rancher:master-head version 1ce03e7
• Installation option (single install/HA): HA

Cluster information

• Cluster type (Hosted/Infrastructure Provider/Custom/Imported): Custom
• Kubernetes version (use kubectl version):

`v1.17.2-rancher1-2`

[aaronyeeski]

In some clusters, nodes that were provisioned as all roles are provisioned as Control Plane.
Cluster setup:

Installation command and docker ps from a Control Plane node:

May be due to the way RKE up is run every time there is an update. Sequence:

1. Etcd  - Etcd components upgrade one after other - e1,e2,e3. 
* Etcd components upgrade
2. Control plane - 
*Drain c1
*Control plane components (apiserver , scheduler.,,)  gets upgraded

3. kubelet , kubeproxy upgrades for Etcd and Control plane nodes  (No particular order)  1 at a time
* Drain nodes 
* kubectl, kubeproxy gets upgraded

4.  kubelet, kubeproxy upgrades for worker nodes  -w1, w2,w3 ( in batches of max unavailable)
*Drain node
*kubectl , kubeproxy gets upgraded

[aaronyeeski]

The bug fix is verified with Rancher server rancher\rancher:master-head version ca71db485

Steps:
Install Rancher master-head in HA using k3s nodes.
https://staging.rancher.com/docs/rancher/v2.x/en/installation/k8s-install/create-nodes-lb/
Rancher install command:

helm install rancher rancher-alpha/rancher --namespace cattle-system --set hostname=<hostname> --version 2.4.0-alpha1  --set rancherImageTag=master-head

In master-head, provision a custom 3 node cluster using Linux instances (3 nodes, all roles each)

Custom cluster is provisioned successfully.