Networking issue between nodes

[jvnvenu]

Environmental Info:
RKE2 Version:
v1.30.2 +rke2r1

Node(s) CPU architecture, OS, and Version:
3 Linux Nodes (RHEL 8), 2 Windows Nodes (Server 2019)
The above machines are VMWare vms

Cluster Configuration:
1 server (Linux)
4 agents (2 Linux and 2 Windows)
Flannel CNI

Describe the bug:
except first node all other nodes not able to access Kube APIs and unable to resolve DNS.
Between nodes directly it can reachable without any issues.

Expected behavior:
Should not see any connectivity issues between nodes

Actual behavior:
Lot of connectivity issues

[brandond]

except first node all other nodes not able to access Kube APIs and unable to resolve DNS.

This indicates that VXLAN CNI traffic between nodes is being dropped. You should:

1. Ensure that all of your nodes have the flannel vxlan port open to each other - see https://docs.rke2.io/install/requirements#inbound-network-rules

   UDP	4789	All RKE2 nodes	All RKE2 nodes	Flannel CNI with VXLAN

2. Ensure that your traffic is not being dropped due to a well-known bug in the vmware virtual ethernet driver that affects vxlan traffic.
   You can work around this by running /usr/sbin/ethtool -K flannel.4096 tx-checksum-ip-generic off on the linux nodes.

[jvnvenu]

great. it works. Thanks a lot. Will it be taken care in future version without this workaround.

[brandond]

No. The workaround disables hardware checksum offload, and imposes a significant hit to performance. It should only be used on nodes with buggy NIC drivers that fail to correctly calculate checksums for vxlan packets. Preferably you would use a different virtual interface type, or seek help from vmware to update the driver to a version not affected by this bug.

[dfaltum]

@brandond So this issue is related to VMware (and some other drivers)? I thought this problem is about a kernel bug.

[brandond]

That was one cause of it, iirc there is another more common bug in one of the virtual nic drivers, which is why it's most commonly seen on VMware.

projectcalico/calico#4727 (comment)