Pod Connectivity Issue on Oracle Linux 9 with Calico (rke2 custom cluster)

[panstou]

Description:
I deployed an RKE2 custom downstream cluster using Rancher Manager and tested multiple network configurations:

• Canal (Flannel + Calico) → Not working on Oracle Linux 9
• Calico-only → Not working on Oracle Linux 9
• Flannel-only → Working on Oracle Linux 9
• Same configurations on Oracle Linux 8 → All working

On Oracle Linux 9, when using Canal (Flannel + Calico) or Calico-only, pods on different nodes cannot communicate. However, with Flannel-only, inter-node pod communication works fine. The same configurations work correctly on Oracle Linux 8.

Steps to Reproduce:

1. Deploy an RKE2 custom downstream cluster via Rancher Manager.
2. Configure networking with using the default configuration:

• Canal (Flannel + Calico)
• Calico-only
• Flannel-only

3. Deploy test pods across multiple nodes.
4. Attempt pod-to-pod communication across nodes.

Expected Behavior:

Pods on different nodes should be able to communicate with each other in all configurations.

Actual Behavior:

• Canal (Flannel + Calico) and Calico-only on OL9: Pods cannot communicate across nodes.
• Flannel-only on OL9: Pods communicate successfully across nodes.
• All configurations on OL8: Everything works fine.

Environment:

Operating System:
    Not Working: Oracle Linux Server 9.4, Kernel Version: 5.15.0-300.163.18.el9uek.x86_64
    Working: Oracle Linux Server 8.9, Kernel Version: 5.4.17-2136.327.2.el8uek.x86_64
Rancher Manager Version: 2.10.1
RKE2 Version: rke2 version v1.30.8+rke2r1 (https://github.com/rancher/rke2/commit/3e6fb8634f164d623f53f7b61d07b27726565d4e)
Rke2-calico Version: rke2-calico-v3.29.100

Additional Notes:

• firewalld and SELinux disabled
• NetworkManager is configured to ignore calico/flannel related network interfaces
• Flannel-only works fine, so the issue might be Calico-specific.

[panstou]

# ethtool -k  vxlan.calico | grep generic
        tx-checksum-ip-generic: off

This issue seems to be related to Calico networking inside the node.

# ping -I 10.171.96.192 10.171.96.195
PING 10.171.96.195 (10.171.96.195) from 10.171.96.192 : 56(84) bytes of data.
From 10.171.96.192 icmp_seq=1 Destination Host Unreachable
From 10.171.96.192 icmp_seq=2 Destination Host Unreachable
From 10.171.96.192 icmp_seq=3 Destination Host Unreachable
^C
--- 10.171.96.195 ping statistics ---
5 packets transmitted, 0 received, +3 errors, 100% packet loss, time 4088ms
pipe 4

# ip -o -4 a
1: lo    inet 127.0.0.1/8 scope host lo\       valid_lft forever preferred_lft forever
2: ens192    inet 10.136.106.70/23 brd 10.136.107.255 scope global noprefixroute ens192\       valid_lft forever preferred_lft forever
28: vxlan.calico    inet 10.171.96.192/32 scope global vxlan.calico\       valid_lft forever preferred_lft forever

# ip route
default via 10.136.106.1 dev ens192 proto static metric 100
10.136.106.0/23 dev ens192 proto kernel scope link src 10.136.106.70 metric 100
10.171.33.0/26 via 10.171.33.1 dev vxlan.calico onlink
10.171.44.192/26 via 10.171.44.195 dev vxlan.calico onlink
10.171.54.128/26 via 10.171.54.133 dev vxlan.calico onlink
blackhole 10.171.96.192/26 proto 80
10.171.96.193 dev cali60f27bfb689 scope link
10.171.96.194 dev calid6fbbc33834 scope link
10.171.96.195 dev calif4156e36b37 scope link
10.171.160.64/26 via 10.171.160.68 dev vxlan.calico onlink
10.171.249.192/26 via 10.171.249.193 dev vxlan.calico onlink

# ip link show | grep cali
28: vxlan.calico: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
31: cali60f27bfb689@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP mode DEFAULT group default qlen 1000
32: calid6fbbc33834@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP mode DEFAULT group default qlen 1000
33: calif4156e36b37@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP mode DEFAULT group default qlen 1000

# arp -a | grep cali
? (10.171.54.133) at 66:79:ec:44:9c:ae [ether] PERM on vxlan.calico
? (10.171.44.195) at 66:62:82:83:47:01 [ether] PERM on vxlan.calico
? (10.171.96.193) at <incomplete> on cali60f27bfb689 <------------------------- Host Unreachable
? (10.171.33.1) at 66:37:95:22:c7:66 [ether] PERM on vxlan.calico
? (10.171.96.195) at <incomplete> on calif4156e36b37 <------------------------- Host Unreachable
? (10.171.249.193) at 66:cd:88:17:a2:71 [ether] PERM on vxlan.calico
? (10.171.96.194) at 96:8d:52:0b:93:88 [ether] on calid6fbbc33834
? (10.171.160.68) at 66:5d:3e:73:0c:6d [ether] PERM on vxlan.calico

# calicoctl node  checksystem
Checking kernel version...
                5.15.0-300.163.18.el9uek.x86_64                                 OK
Checking kernel modules...
                xt_conntrack                                            OK
                ip_set                                                  OK
                ip6_tables                                              OK
                vfio-pci                                                OK
                ipt_REJECT                                              OK
                xt_rpfilter                                             OK
                xt_u32                                                  OK
                ip_tables                                               OK
                ipt_ipvs                                                OK
                ipt_rpfilter                                            OK
                nf_conntrack_netlink                                    OK
                xt_set                                                  OK
                ipt_set                                                 OK
                xt_addrtype                                             OK
                xt_icmp                                                 OK
                xt_multiport                                            OK
                xt_bpf                                                  OK
                xt_icmp6                                                OK
                xt_mark                                                 OK
System meets minimum system requirements to run Calico!
------------------------------------------------------------------------------------------
# calicoctl get workloadendpoints --allow-version-mismatch
WORKLOAD             NODE           NETWORKS            INTERFACE
overlaytest-4qvfq    XXXXXXXXX7   10.171.160.77/32    calided4d8b4d24
overlaytest-4zh5s    XXXXXXXXX5   10.171.96.193/32    cali60f27bfb689
overlaytest-6vqsx    XXXXXXXXX0   10.171.54.134/32    calia54632ff81d
overlaytest-dcclp    XXXXXXXXX8   10.171.44.196/32    cali4a9733d25a0
overlaytest-hwkqv    XXXXXXXXX6   10.171.33.2/32      cali98e7a4c5a9b
overlaytest-j27zm    XXXXXXXXX9   10.171.249.192/32   cali9145ed38b5a
overlaytest2-bkbbz   XXXXXXXXX8   10.171.44.197/32    cali91f50bef26b
overlaytest2-cfv9g   XXXXXXXXX7   10.171.160.78/32    cali1b33d777534
overlaytest2-d9jsm   XXXXXXXXX9   10.171.249.194/32   cali9fac0499149
overlaytest2-tmvgz   XXXXXXXXX6   10.171.33.3/32      calica89439d255
overlaytest2-v2dcp   XXXXXXXXX5   10.171.96.195/32    calif4156e36b37
overlaytest2-z6cgm   XXXXXXXXX0   10.171.54.135/32    calie5b402fca47
---------------------------------------------------------------------------------------------

[coutinhop]

@panstou have you raised this with the RKE2 folks as well? from the info you posted, it's still a bit hard to pinpoint what could be causing this issue... have Calico versions changed when you upgraded from OL8 to OL9?

Could you post the output of kubectl get felixconfigurations -o yaml, kubectl get ippools -o yaml, kubectl get installations -o yaml?

[panstou]

Hi @coutinhop! I have already raised this issue with the RKE2 team.
rancher/rke2#7932

When deploying Calico via RKE2, the Calico version is tied to the RKE2 version. I tested the same installation with different RKE2 (and corresponding Calico) versions, but the issue persisted. However, the same versions worked fine on OL8.
I am using the default rke2-calico values, and below are the command outputs:

• kubectl get felixconfigurations -o yaml

apiVersion: v1
items:
- apiVersion: crd.projectcalico.org/v1
  kind: FelixConfiguration
  metadata:
    annotations:
      operator.tigera.io/bpfEnabled: "false"
      projectcalico.org/metadata: '{"generation":2,"creationTimestamp":"2025-04-01T08:21:37Z","labels":{"app.kubernetes.io/managed-by":"Helm"},"annotations":{"meta.helm.sh/release-name":"rke2-calico","meta.helm.sh/release-namespace":"kube-system"},"managedFields":[{"manager":"helm","operation":"Update","apiVersion":"crd.projectcalico.org/v1","time":"2025-04-01T08:21:37Z","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app.kubernetes.io/managed-by":{}}},"f:spec":{".":{},"f:defaultEndpointToHostAction":{},"f:featureDetectOverride":{},"f:healthPort":{},"f:logSeveritySys":{},"f:wireguardEnabled":{},"f:xdpEnabled":{}}}},{"manager":"operator","operation":"Update","apiVersion":"crd.projectcalico.org/v1","time":"2025-04-01T08:21:47Z","fieldsType":"FieldsV1","fieldsV1":{"f:spec":{"f:nftablesMode":{},"f:vxlanVNI":{}}}}]}'
    name: default
  spec:
    bpfConnectTimeLoadBalancing: TCP
    bpfEnabled: false
    bpfHostNetworkedNATWithoutCTLB: Enabled
    bpfLogLevel: ""
    defaultEndpointToHostAction: Drop
    featureDetectOverride: ChecksumOffloadBroken=true
    floatingIPs: Disabled
    healthPort: 9099
    logSeverityScreen: Info
    logSeveritySys: Info
    nftablesMode: Disabled
    reportingInterval: 0s
    vxlanVNI: 4096
    wireguardEnabled: false
    xdpEnabled: true
kind: List
metadata:
  resourceVersion: ""

• kubectl get ippools -o yaml

apiVersion: v1
items:
- apiVersion: crd.projectcalico.org/v1
  kind: IPPool
  metadata:
    labels:
      app.kubernetes.io/managed-by: tigera-operator
    name: default-ipv4-ippool
  spec:
    allowedUses:
    - Workload
    - Tunnel
    blockSize: 26
    cidr: 10.171.0.0/16
    ipipMode: Never
    natOutgoing: true
    nodeSelector: all()
    vxlanMode: Always
kind: List
metadata:
  resourceVersion: ""

• kubectl get installations -o yaml

apiVersion: v1
items:
- apiVersion: operator.tigera.io/v1
  kind: Installation
  metadata:
    annotations:
      meta.helm.sh/release-name: rke2-calico
      meta.helm.sh/release-namespace: kube-system
    labels:
      app.kubernetes.io/managed-by: Helm
    name: default
  spec:
    calicoNetwork:
      bgp: Disabled
      hostPorts: Enabled
      ipPools:
      - allowedUses:
        - Workload
        - Tunnel
        blockSize: 26
        cidr: 10.171.0.0/16
        disableBGPExport: false
        disableNewAllocations: false
        encapsulation: VXLAN
        name: default-ipv4-ippool
        natOutgoing: Enabled
        nodeSelector: all()
      linuxDataplane: Iptables
      linuxPolicySetupTimeoutSeconds: 0
      multiInterfaceMode: None
      nodeAddressAutodetectionV4:
        firstFound: true
      windowsDataplane: Disabled
    cni:
      ipam:
        type: Calico
      type: Calico
    controlPlaneReplicas: 2
    controlPlaneTolerations:
    - effect: NoSchedule
      key: node-role.kubernetes.io/control-plane
      operator: Exists
    - effect: NoExecute
      key: node-role.kubernetes.io/etcd
      operator: Exists
    flexVolumePath: /var/lib/kubelet/volumeplugins/
    imagePath: rancher
    imagePrefix: mirrored-calico-
    kubeletVolumePluginPath: None
    kubernetesProvider: RKE2
    logging:
      cni:
        logFileMaxAgeDays: 30
        logFileMaxCount: 10
        logFileMaxSize: 100Mi
        logSeverity: Info
    nodeUpdateStrategy:
      rollingUpdate:
        maxUnavailable: 1
      type: RollingUpdate
    nonPrivileged: Disabled
    variant: Calico
kind: List
metadata:
  resourceVersion: ""

[joseluisgonzalezca]

@panstou Were you able to resolve your issue? I'm facing the same problem and I've tried everything without success. My current setup consists of 3 nodes with Debian 12, using RKE2 with Kubernetes v1.31.5 and Calico 3.29.1.

I've tried to:

• Disable checksum validation in both physical interfaces (internal and external) and vxvlan.calico as stated in other issues.
• Change configuration of mtuIfacePattern to match enX0 and enX1 and auto detect MTU.
• Restart calico daemonset

My problem is related to Nodeport inter-node communication. Traffic is working as expected if the target pod is running in the same node where to the original request is sent. If the node has to reroute the package to other node because the pod is not available in-place, the communication does not work.

[tomastigera]

See rancher/rke2#7932 (comment) hope it fixes your issue.

[skoryk-oleksandr]

@joseluisgonzalezca Have you tried to change the default vxlan port. It is a vmware limitation 🤷‍♂ see #8349 (comment)
@panstou Is this issue still exist for you?

[joseluisgonzalezca]

Hi @skoryk-oleksandr!

I'm using Xen as the underlying hypervisor so the VMware issue #8349 that you have mentioned does not apply to my case.

I opened another issue #10149 because I realized that my problem was similar but not the same as the one stated in this issue. In my case, the problem was related to kube-proxy. Some of the iptables rules were deleted by mistake and I had to recreate them by restarting the nodes.

Apart from that, I made some adjustments in the Calico configuration in order to property bind the vxlan interface to my internal physical interfaces (my nodes have multiple adapters and the traffic was being sent using the wrong interface).

It's all explained in the latest messages of the issue.

Kind regards!

[bulsiel]

Hi
@panstou try to change option:
featureDetectOverride: ChecksumOffloadBroken=true -> featureDetectOverride: ChecksumOffloadBroken=false

Similar problem is described by SUSE

[lucastigera]

Hey @panstou !
Have you tried the fix suggested by @tomastigera and @skoryk-oleksandr , or the configuration recommended by @bulsiel ?
Are you still experiencing the issue?

[lucastigera]

Since things have been quiet, I’m going to go ahead and close this issue.
Feel free to reopen it if the problem still persists.