Rancher RKE upstream cluster have DNS flipping issue

[prasannavasan]

Rancher Server Setup

• Rancher version: 2.7.1
• Installation option (Docker install/Helm Chart): Docker

Information about the Cluster

• Kubernetes version: V1.24.9
• Cluster Type (Local/Downstream): Downstream (Custom)

• Custom = Running a docker command on a node
• Using - sudo docker run -d --restart=unless-stopped -p 80:80 -p 443:443 --privileged rancher/rancher

User Information

• What is the role of the user logged in? Default Admin

Describe the bug
I am creating a custom cluster from Rancher UI using 3 EC2 instances as nodes. On the first node after installing docker I am running rancher agent and then logging into it to create a cluster. I am keeping the default and creating the cluster. I am also registering the node as All roles to the cluster.

I connect to the cluster and do a kubectl exec -i -t dnsutils -- nslookup kubernetes.default (To check DNS) - https://kubernetes.io/docs/tasks/administer-cluster/dns-debugging-resolution/

DNS is resolved fine.

I register and add my second node as a worker role to existing cluster. Now the DNS is flipping. Attached image below.

EC2 instance config

3 VMs of small size with 4 cpu cores 16GB memory and 100GB disk. Ubuntu 22.04 LTS.

To Reproduce
Setup a RKE cluster from a EC2 node
Connect and verify DNS lookup works fine
Register another node
DNS starts flipping

Result

Expected Result
Expect to resolve dns successfully.

Screenshots

Additional context
This issue prevents setting up yugabyte db on the cluster with DNS issue in my usecase.