Massive packet loss with Flannel on Robot nodes

[erendogan51]

For a long period of time, i was struggling with TCP connection timeout on pods that are scheduled on Robot nodes. My cluster uses Flannel as CNI with Wireguard as its backend, with my Robot nodes connected over a vSwitch. While investigating, i tried many different configuration options, like creating a net-conf on my Robot to configure MTU to 1400, or changing from the Wireguard backend to Vxlan, but all yielding not remedy to my issue. I am still experiencing massive packet loss:

mtr --tcp --port 443 github.com

Interestingly the issue only occurs with TCP packet not with IP:

mtr github.com

The issue only happens on Robot nodes, while cloud nodes are fine. I could replicate the issue with a fresh cluster and a new Robot node. Another solution i tried on this fresh setup was switching Flannel to Cilium, which eliminated the issue entirely. While i would be okay doing that, a migration is just pain.

Has anyone else encountered this issue and could provide feedback?

[felix-oberressl]

This sounds potentially related to an issue I’m seeing in one of my K3s clusters on Hetzner Robot/vSwitch, although my symptoms are not exactly the same.

In my case, the problem only appears briefly and mainly on the busiest worker node. The node gets marked NotReady for a few seconds, pod deletion starts and then gets canceled once connectivity comes back. I also see short k3s-agent/API timeouts and temporary Rook-Ceph mon session loss during those moments.

Did your issue also depend on workload, or did it happen even on mostly idle nodes?

[erendogan51]

no my issue was not load dependent unfortunately and ultimately i could not figure it out so i migrated my cluster to Cilium with some downtime

some things dont work after doing such a migration like you cannot replace the kube proxy or use another IPAM aside from Kubernetes Host Scope but now its at least stable

one thing i added to my Cilium values is that even the Cloud node use the MTU of 1400, i did not test with the default 1450 unfortunately so cannot tell for certain if just that fixed my issue or Cilium in general