IPIP: decapsulated inner packets arrive at dest node but never delivered to pod netns (cross-node pod-to-pod broken, cluster-wide)

[jjozzietech]

Environment

• Calico v3.31.5, IPIP mode
• Kubernetes v1.34.8, 23 nodes, single-NIC
• Pod CIDR 10.244.0.0/16
• Ubuntu 24.04, kernels mixed (6.8.0-106 / -117 / -124)

Symptom
Cross-node pod-to-pod traffic fails (100% packet loss) for tenant workload pods. Same-node pod-to-pod works. Infrastructure pods (e.g. ingress nginx) remain reachable cross-node; tenant workload pods do not. Onset ~24h ago, no intentional Calico change at that time.

Key evidence

• ip route get <podIP> resolves correctly via tunl0 on the source node; /26 subnet routes present via bird; blackhole routes correct on owning nodes.
• tcpdump on the destination node shows the decapsulated inner ICMP echo request ARRIVING:
  IP <srcNode> > <dstNode>: IP <srcPod> > <dstPod>: ICMP echo request
  ...but no echo reply is ever generated.
• The destination pod's eth0 rx_packets counter does NOT increment during a cross-node ping → the inner packet dies between the node root namespace and the pod netns.
• conntrack -E shows no entry for the failing ICMP (never tracked).
• net.ipv4.ip_forward=1, all.forwarding=1, rp_filter=0.
• ipsets present: cali40all-hosts-net exists (revision 7); the IPIP ACCEPT rule (/* Allow IPIP packets from Calico hosts */) shows 20M+ packets.
• felix logs: only routine HostMetadata resync, no dataplane errors.

Not the cause (ruled out by testing)

• Kernel version: a node on 6.8.0-124 fails identically to nodes on -106/-117.
• Stale pod state: freshly recreated pods (new IP, new veth) fail identically.
• Routes/BGP: all present and correct.

Tried, did NOT fix (now part of cluster state)

• kubectl rollout restart daemonset/calico-node (full, completed)
• Reboot of a single node (returns Ready, still broken)
• rp_filter=0 on one node

Question
What causes IPIP decapsulated inner packets to arrive at the destination node and decap successfully, but never be delivered into the target pod's network namespace — cluster-wide, surviving a calico-node rollout, and independent of kernel version? Is there a known fix short of a full simultaneous fleet reboot?

Happy to provide any specific command output (iptables -L -nv, ip route, calicoctl node status, felix config, etc.).