node-aware reachability probes + infra-symptom fingerprinting (a real cross-node debugging saga Radar could have shortened)

[nadaverell]

The motivating case

This came from a real thread in a DevOps community group (details anonymized, translated from Hebrew). An experienced operator spent two days debugging a fresh on-prem cluster:

Environment: RKE2 · Canal · vSphere (not Tanzu, manually provisioned VMs, no CPI) · RHEL · SELinux disabled · firewalld disabled · broken from the first deployment.

Symptoms:

• Pods on different nodes cannot talk over TCP/UDP — but ping works
• nmap shows filtered; tcpdump shows the packets never arrive at all — except ICMP
• TCP/UDP to the node's own IP works fine
• Pod-to-pod on the same node works fine

The triage journey the group went through (each step hours apart):

1. "Manually provisioned RHEL VMs? Once I had a ghost firewalld process still running even though the service was down — kill it and harden systemd." → checked, not it.
2. "NetworkPolicy?" → deleted them all, no change.
3. Found a strange zombie flannel process → investigated, not it.
4. "NetworkManager on RHEL — you may need to tell flannel which interface to use." → verified flannel was on the node network interface, looked fine.
5. Swapped the entire CNI to Cilium on the same infrastructure → identical failure. Conclusion in the thread: "it must be vSphere dropping packets."
6. Latest theory being chased: ESXi promiscuous mode ("maybe the vSwitch drops frames destined for the MAC of the flannel sub-interface?"). Another member confirmed their working multi-node vSphere cluster has promiscuous mode off — and for VXLAN overlays this theory doesn't hold anyway: the inner frame is wrapped inside a UDP payload, the vSwitch only ever sees the outer frame with the VM's own MAC.
7. Parallel theories about the org's external firewalls ("maybe it treats the pod CIDR as a foreign VLAN and drops it") — hard to check because the vSphere environment is managed by another team the operator has no access to.

What it most likely actually is (fits every symptom, and the CNI swap reproducing it is the tell): the well-known vmxnet3 UDP tunnel offload issue on ESXi. Both Canal and default Cilium encapsulate with VXLAN (UDP 8472); the vmxnet3 NIC advertises tx-udp_tnl-segmentation offload, ESXi mishandles the encapsulated TCP/UDP packets on non-NSX VXLAN ports, and they're dropped at the sender's hypervisor — so the receiver's tcpdump sees nothing. ICMP doesn't ride the offload path, so ping survives. The classic fix, on every node's uplink interface:

ethtool -K ens192 tx-udp_tnl-segmentation off tx-udp_tnl-csum-segmentation off

(The runner-up suspect in a managed environment like this is an NSX Distributed Firewall policy allowing ICMP but default-denying TCP/UDP.)

Why Radar (with #1037) wouldn't have saved them — yet

The reachability work in #1037 is deliberately honest about what it verifies, and here it would honestly report: config clean, live TCP flaky. True — but this operator established that in the first hour with nmap and tcpdump. The two facts that actually cracked the case were:

• same-node ✓ / cross-node ✗ — a matrix they built by hand, pod pair by pod pair
• CNI-agnostic → the break is below Kubernetes

Today's in-cluster probes can't produce either: probe Jobs are scheduled wherever the scheduler puts them and dial the Service name (internal/reachability/incluster.go), so on a cluster like this the result is nondeterministically green or red depending on where the probe pod lands and which endpoint kube-proxy picks. "Flaky" is less precise than what the operator already knew.

Proposal

1. Node-aware probe matrix

Extend the in-cluster test with a topology mode:

• Pin probe pods to specific nodes (one per node, or a bounded sample on large clusters — same spirit as the existing MaxInClusterProbes cap)
• Dial per-endpoint pod IPs (and optionally node IPs) in addition to the Service name
• Project results into a source-node × target-node matrix

When cross-node probes fail while same-node probes succeed, emit a first-class finding:

Cross-node overlay break — pods can reach same-node peers but not peers on other nodes. Your Kubernetes config is fine; the break is in the CNI/encapsulation layer or below.

This fits #1037's honesty model precisely: the same-node/cross-node asymmetry is measured evidence, not inference, and it's exactly the fact that reframes the whole investigation from "my manifests" to "my infrastructure."

2. Symptom → known-cause shortlist

cross-node-only + TCP/UDP-only + ICMP-passes + config-clean is a fingerprint with a short, famous list of causes. Once the matrix detects it, surface a curated shortlist as hypotheses (never verdicts — we can't see below the K8s API), each with the exact next command:

Hypothesis	Check / fix
VXLAN checksum/segmentation offload (VMware vmxnet3, some bare-metal NICs)	ethtool -K <uplink> tx-udp_tnl-segmentation off tx-udp_tnl-csum-segmentation off
Encapsulation port blocked between nodes	tcpdump -ni <uplink> udp port 8472 (flannel/Cilium) / 4789 (Calico VXLAN) on the receiver
MTU mismatch (overlay overhead)	compare interface MTU vs CNI MTU; test with small vs large payloads
RHEL: NetworkManager managing CNI interfaces / nm-cloud-setup	RKE2 docs: rke2-canal.conf ignore rules; disable nm-cloud-setup
External/distributed firewall (e.g. NSX DFW) between nodes	ask the infra team for policies on inter-node UDP + the pod CIDR

Detection can be sharpened by what we already have in the cache: node providerID / nodeInfo identifying vSphere VMs, the CNI identified from running DaemonSets — enough to rank "VMware offload bug" first for exactly this environment.

Why it's worth it

This failure class (cross-node overlay breakage on on-prem/VM infrastructure) is one of the most common and most expensive K8s networking failures — and the thread above shows how it goes without tooling: a competent operator, two days, a group chat crowd-sourcing theories, and at the end they were converging on a wrong theory (promiscuous mode) while the likely fix was a one-line ethtool. Radar could have produced the matrix in one probe run and put the ethtool command on screen in minute five.

Rough implementation surface: internal/reachability/ (node pinning + per-endpoint targets), internal/trace/coverage.go (matrix projection + new finding), the Reachability tab matrix UI, and the MCP diagnose payload.

Feedback welcome — especially from anyone running multi-node clusters on vSphere/Proxmox/bare metal who's hit this class of problem: what signal would have saved you the most time?