Pod to ingress with backend hosted on the same node not working in certain configurations

[giorio94]

Is there an existing issue for this?

• I have searched the existing issues

What happened?

Running an extended version of the connectivity tests to validate pod to ingress service with both the backend pod hosted on the same or a different node highlighted connectivity issues when the backend is hosted on the same node in certain configurations. In particular, this issue occurred on the Conformance Cluster Mesh E2E tests, although not clustermesh related (as the backend is hosted in the local cluster as well), and more specifically in the following configurations:

          - name: '2'
            tunnel: 'disabled'
            ipfamily: 'dual'
            encryption: 'wireguard'
            kube-proxy: 'none' (KPR + BPF masquerade)

          - name: '4'
            tunnel: 'disabled'
            ipfamily: 'ipv6'
            encryption: 'disabled'
            kube-proxy: 'none' (KPR + BPF masquerade)

❌ 3/63 tests failed (5/654 actions), 12 tests skipped, 0 scenarios skipped:
connectivity test failed: 3 tests failed
Test [pod-to-ingress-service]:
  ❌ pod-to-ingress-service/pod-to-ingress-service/curl-0: cilium-test/client-69748f45d8-vbwdx (10.242.1.231) -> cilium-test/cilium-ingress-same-node (cilium-ingress-same-node.cilium-test:80)
  ❌ pod-to-ingress-service/pod-to-ingress-service/curl-2: cilium-test/client2-ccd7b8bdf-z8fmw (10.242.1.100) -> cilium-test/cilium-ingress-same-node (cilium-ingress-same-node.cilium-test:80)
Test [pod-to-ingress-service-allow-ingress-identity]:
  ❌ pod-to-ingress-service-allow-ingress-identity/pod-to-ingress-service/curl-1: cilium-test/client-69748f45d8-vbwdx (10.242.1.231) -> cilium-test/cilium-ingress-same-node (cilium-ingress-same-node.cilium-test:80)
  ❌ pod-to-ingress-service-allow-ingress-identity/pod-to-ingress-service/curl-2: cilium-test/client2-ccd7b8bdf-z8fmw (10.242.1.100) -> cilium-test/cilium-ingress-same-node (cilium-ingress-same-node.cilium-test:80)
Test [outside-to-ingress-service]:
  ❌ outside-to-ingress-service/outside-to-ingress-service/curl-ingress-service-0: cilium-test/host-netns-non-cilium-nz9t8 (172.18.0.3) -> cilium-test/cilium-ingress-same-node (cilium-ingress-same-node.cilium-test:80)

❌ 3/57 tests failed (5/342 actions), 18 tests skipped, 8 scenarios skipped:
Test [pod-to-ingress-service]:
  ❌ pod-to-ingress-service/pod-to-ingress-service/curl-1: cilium-test/client-69748f45d8-r8rpj (fd00:10:242:2::20b0) -> cilium-test/cilium-ingress-same-node (cilium-ingress-same-node.cilium-test:80)
  ❌ pod-to-ingress-service/pod-to-ingress-service/curl-2: cilium-test/client2-ccd7b8bdf-mjxhv (fd00:10:242:2::d899) -> cilium-test/cilium-ingress-same-node (cilium-ingress-same-node.cilium-test:80)
Test [pod-to-ingress-service-allow-ingress-identity]:
  ❌ pod-to-ingress-service-allow-ingress-identity/pod-to-ingress-service/curl-0: cilium-test/client-69748f45d8-r8rpj (fd00:10:242:2::20b0) -> cilium-test/cilium-ingress-same-node (cilium-ingress-same-node.cilium-test:80)
  ❌ pod-to-ingress-service-allow-ingress-identity/pod-to-ingress-service/curl-2: cilium-test/client2-ccd7b8bdf-mjxhv (fd00:10:242:2::d899) -> cilium-test/cilium-ingress-same-node (cilium-ingress-same-node.cilium-test:80)
Test [outside-to-ingress-service]:
  ❌ outside-to-ingress-service/outside-to-ingress-service/curl-ingress-service-0: cilium-test/host-netns-non-cilium-x5zz8 (fc00:f853:ccd:e793::4) -> cilium-test/cilium-ingress-same-node (cilium-ingress-same-node.cilium-test:80)

Differently, it didn't occur when KPR was disabled (the ingress controller is not enabled in that case), or Cilium was configured in tunnel mode. I'm not sure why it didn't happen in the Conformance E2E tests.

Link: https://github.com/cilium/cilium/actions/runs/8456333159

Cilium Version

Tip of main

Sysdump

cilium-sysdump-20240327-180128.zip
cilium-sysdump-20240327-180201.zip

Code of Conduct

• I agree to follow this project's Code of Conduct

[sayboras]

Ingress controller is having one requirement of kpr enabled (or at least nodeport enabled). The above configurations didn't seem to satisfy such requirement 🤔

[giorio94]

Ingress controller is having one requirement of kpr enabled (or at least nodeport enabled). The above configurations didn't seem to satisfy such requirement 🤔

kube-proxy: none corresponds to KPR enabled. Sorry for not being clear there, that's the setting for kind which disables the kube-proxy daemonset, and then Cilium is configured with KPR enabled.

[sayboras]

Copied the comment here so that it will not get lost.

In short, with the re-introduce of from proxy route 2005, there is still issue on the return traffic if bpf.masquerade is enabled, the workaround is to enable bpf.hostLegacyRouting as well.

#32367 (comment)

[jspaleta]

Just fyi, I'm seeing similar reproducible behavior in a Kind cluster environment but bpf.hostLegacyRouting doesn't appear to be a viable workaround for me...so far. I'll continue testing in #32525