Cilium doesn't send neighbor advertisements in response to neighbor solicitiations for pod IPv6 IPs

[jschwinger233]

Is there an existing issue for this?

• I have searched the existing issues

What happened?

#12086 introduced a feature that cilium can respond to IPv6 NS on behalf of the pods on the same host, but this feature doesn't work on v1.15.

Cilium Version

v1.15.1

Kernel Version

Linux gray-Latitude-5530 6.5.0-18-generic #18~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC Wed Feb 7 11:40:03 UTC 2 x86_64 x86_64 x86_64 GNU/Linux

Kubernetes Version

Client Version: version.Info{Major:"1", Minor:"26", GitVersion:"v1.26.1", GitCommit:"8f94681cd294aa8cfd3407b8191f6c70214973a4", GitTreeState:"clean", BuildDate:"2023-01-18T15:58:16Z", GoVersion:"go1.19.5", Compiler:"gc", Platform:"linux/amd64"}
Kustomize Version: v4.5.7
Server Version: version.Info{Major:"1", Minor:"27", GitVersion:"v1.27.3", GitCommit:"25b4e43193bcda6c7328a6d147b1fb73a33f1598", GitTreeState:"clean", BuildDate:"2023-06-15T00:36:28Z", GoVersion:"go1.20.5", Compiler:"gc", Platform:"linux/amd64"}

Regression

Yes, v1.13 still works.

Sysdump

No response

Relevant log output

No response

Anything else?

Repro steps:

1. write a helm.values file with following content:

bpf:
  masquerade: true
enableIPv6Masquerade: false
extraConfig:
  enable-ipv6-ndp: "true"
  ipv6-mcast-device: eth0
ipam:
  mode: cluster-pool
  operator:
    clusterPoolIPv4MaskSize: "23"
    clusterPoolIPv4PodCIDRList:
      - 10.128.0.0/10
    clusterPoolIPv6PodCIDRList:
      - fc00:f853:ccd:e793::100:0/104
    clusterPoolIPv6PodCIDRMask: 120
ipv6:
  enabled: true
k8sServiceHost: kind-control-plane
k8sServicePort: "6443"
kubeProxyReplacement: true

2. create a kind cluster

./contrib/scripts/kind.sh --xdp --secondary-network "" 3 "" "" iptables dual 0.0.0.0 6443

3. create cilium v1.15.1, without kind-worker3

kubectl patch node kind-worker3 --type=json -p='[{"op":"add","path":"/metadata/labels/cilium.io~1no-schedule","value":"true"}]'

cilium install --nodes-without-cilium=kind-worker3 --wait     --chart-directory=./install/kubernetes/cilium        -f ./helm.values  --version v1.15.1

4. create a Nginx pod

kubectl apply -f https://k8s.io/examples/application/deployment.yaml

5. add a route on kind-worker3 to make sure kind-worker3 sends IPv6 NS for nginx pod:

docker exec -it kind-worker3 ip -6 r a $ipv6_of_nginx_pod dev eth0

6. check connectivity between kind-worker3 and nginx pod, should be fine but broken

docker exec -it kind-worker3 curl [c00:f853:ccd:e793::100:18b]:80

Running tcpdump on kind-worker3 eth0, it turns out IPv6 NS for pod never got reply:

# fc00:f853:ccd:e793::100:18b is ipv6 of nginx pod

listening on eth0, link-type EN10MB (Ethernet), snapshot length 262144 bytes
12:50:21.439388 IP6 fc00:c111::4 > ff02::1:ff00:18b: ICMP6, neighbor solicitation, who has fc00:f853:ccd:e793::100:18b, length 32
12:50:22.468934 IP6 fc00:c111::4 > ff02::1:ff00:18b: ICMP6, neighbor solicitation, who has fc00:f853:ccd:e793::100:18b, length 32
12:50:23.493014 IP6 fc00:c111::4 > ff02::1:ff00:18b: ICMP6, neighbor solicitation, who has fc00:f853:ccd:e793::100:18b, length 32
^C

Cilium Users Document

• Are you a user of Cilium? Please add yourself to the Users doc

Code of Conduct

• I agree to follow this project's Code of Conduct

[jschwinger233]

This turns out to be an unexpected outcome of 6580714 (bpf: Remove ICMPv6 NS Responder on bpf_host), however things are getting complicated.

Timeline:

1.13: bpf NS responder was needed because of #23445, at the same time it caused troube like #14509
1.14: since #23445 had been closed, I thought it was a good time to remove bpf NS responder, which also resolved #14509
1.15: now we realized removing bpf NS responder brreaks #12086
this week: #30837 was trying to revert the change, but ended up with another ipv6 issue when KPR is on: #30837 (comment)

A probably feasible solution is to bring back bpf NS responder only if NS is asking for a pod IPv6 (can be found from ipcache); for any other NS (e.g. asking for node IP), we just let them go through stack.