Cluster does not come up in Rancher when flatcar-linux-stable-2605.6.0 nodes are used

[sowmyav27]

What kind of request is this (question/bug/enhancement/feature request): bug

Steps to reproduce (least amount of steps as possible):

• Bring up a custom cluster on 2.5-head using flatcar-linux-stable-2605.6.0 nodes
• 1 etcd, 1 control and 3 worker nodes.
• etcd and worker nodes are seen Active state.
• But the Cluster is in error state with error Cluster health check failed: Failed to communicate with API server: Get "https://<controlplane ip>:6443/api/v1/namespaces/kube-system?timeout=45s": context deadline exceeded

Expected Result:
Cluster should deploy successfully.

Other details that may be helpful:

Environment information

• Rancher version (rancher/rancher/rancher/server image tag or shown bottom left in the UI): 2.5-head - commit id: 0465f0
• Installation option (single install/HA): Single node

Cluster information

• Cluster type (Hosted/Infrastructure Provider/Custom/Imported): custom cluster
• Kubernetes version (use kubectl version):

1.19.3

[superseb]

@sowmyav27 @aaronrancher Can we get the exact AMI and instance types used for this?

[superseb]

Can we also confirm this was applied? rancher/rke#1744 (comment)

[sowmyav27]

Post offline discussion with @superseb Will validate this usecase after applying this cluster.yml

[sowmyav27]

On 2.5.2, when this cluster.yml was applied, the clusters failed to come up.

• Deploy custom cluster with ^^^ cluster.yaml
• instance_type: t3a.medium/t3.xlarge. 1 etcd/control plane 2 worker nodes
• Cluster shows error Cluster health check failed: Failed to communicate with API server: Get "https://<controlplane-ip>:6443/api/v1/namespaces/kube-system?timeout=45s": context deadline exceeded

[sowmyav27]

• On 2.5.2, Apply this cluster.yml and after running systemctl enable docker.service on the node, the node registered in the cluster successfully.
• instance type: t3a.medium

[sowmyav27]

On 2.5-head - commit id: 9ee39b9f
Downstream clusters

• Verified deploying downstream clusters using cluster.yml.
• Validation tests passed on these clusters(automation check)

HA setup:

• Deployed HA using this network setting in rke local cluster using cluster.yml.
• Deployed a downstream cluster using the same config. Validation tests passed on this cluster.