[Release-1.28] - Unable to access metrics-server pod ip from all nodes

[brandond]

Backport fix for Unable to access metrics-server pod ip from all nodes

• Unable to access metrics-server pod ip from all nodes #5518

[ShylajaDevadiga]

Validated fix on rke2 version v1.28.7-rc3+rke2r1

Environment Details

Infrastructure
Cloud EC2 instance

Node(s) CPU architecture, OS, and Version:

cat /etc/os-release 
ec2-user@ip-172-31-4-114:~> cat /etc/os-release
NAME="SLES"
VERSION="15-SP4"
VERSION_ID="15.4"
PRETTY_NAME="SUSE Linux Enterprise Server 15 SP4"

Cluster Configuration:
3 server 1 agent

Config.yaml:

cat /etc/rancher/rke2/config,yaml
write-kubeconfig-mode: "0644"
tls-san:
  - fake.fqdn.value
profile: cis-1.23
cni: calico

Steps to validate

1. Install rke2 in HA setup
2. Check metrics on all nodes

ec2-user@ip-172-31-3-248:~> rke2 -v
rke2 version v1.28.7-rc3+rke2r1 (fe8538a165d25755050d2a0827dbf982f72611fa)
go version go1.21.7 X:boringcrypto
ec2-user@ip-172-31-3-248:~> kubectl get pods -A -o wide|grep metrics
kube-system       helm-install-rke2-metrics-server-sxkn4                                0/1     Completed   0               5h29m   10.42.146.71    ip-172-31-3-248.us-east-2.compute.internal   <none>           <none>
kube-system       rke2-metrics-server-544c8c66fc-p2xr9                                  1/1     Running     0               5h27m   10.42.146.73    ip-172-31-3-248.us-east-2.compute.internal   <none>           <none>
ec2-user@ip-172-31-3-248:~> 

ec2-user@ip-172-31-0-54:~> kubectl top nodes
NAME                                         CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%   
ip-172-31-0-54.us-east-2.compute.internal    254m         12%    1637Mi          42%       
ip-172-31-12-86.us-east-2.compute.internal   74m          3%     755Mi           19%       
ip-172-31-14-20.us-east-2.compute.internal   199m         9%     1552Mi          40%       
ip-172-31-3-248.us-east-2.compute.internal   297m         14%    2327Mi          60%