fix: Prime gateway ARP/NDP neighbor for eBPF fib lookup - #286
Conversation
|
Same disclosure as on the rest of the stack — ecv reviewed this through an AI shell, apologies. Two questions.
|
Grants the CNI DaemonSet's credential-refresh container CAP_BPF and CAP_NET_ADMIN plus a bpf-fs hostPath mount (/sys/fs/bpf), and adds an "ebpf-datapath" gRPC health sub-service to its liveness/readiness probes -- required unconditionally now that this container also hosts the eBPF/TC-BPF uSID datapath's load/attach/pin control daemon (installer.Run, #287); there's no flag left to gate this behind, since the datapath is the only forwarding path. Pins GALACTIC_CNI_EBPF_INTERFACES=eth1 for the containerlab topology, where every lab node is dual-homed and interface auto-detection picks the wrong (management-bridge) link over the actual transit-fabric one. Mounts bpffs on Kind nodes in CI (scripts/ci.sh) and updates TestCNITapInterface to run a privileged e2e pod with its own bpf-fs mount, starting the eBPF control daemon and waiting for vrf_table to be pinned before exercising CNI ADD -- the datapath being the only forwarding path means the e2e pod needs the same maps a production node's DaemonSet would already have pinned. Adds test-unit-root to test-e2e's dependency list. Stacked on #286 (CNI eBPF cutover + gateway fix) and #287 (GC/installer wiring) -- the visible diff includes both until they merge upstream; review focuses on the deploy/e2e files listed above. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Adds SweepEBPFVRFTable, a GC pass that reconciles the eBPF uSID datapath's vrf_table map entries against live BGPVRFInstance CRDs using a generation-cutoff scheme to avoid a register/sweep race. This runs from galactic-cni's "run" container rather than galactic-router's existing GC controller: the pinned vrf_table map only exists inside that container, which has the /sys/fs/bpf hostPath mount and CAP_BPF galactic-router's DaemonSet does not need for anything else. routerNamesForNode gains a fuller sibling, routersForNode, since the sweep needs each router's full Spec.SRv6Locator, not just its name. Wires installer.Run to load/attach/pin the eBPF datapath at startup (startEBPFDatapath), serve /metrics (Prometheus), report an "ebpf-datapath" gRPC health sub-service, and run the GC sweep on its own ticker. Adds the --metrics-port CLI flag to galactic-cni run. Depends only on #283 (attach/usidmap/metrics/uformat) -- no dependency on the CNI ADD cutover in #285/#286, since this reconciles directly against BGPRouter/BGPVRFInstance CRD state rather than anything registerEBPFDatapath writes to resourceTracker. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Grants the CNI DaemonSet's credential-refresh container CAP_BPF and CAP_NET_ADMIN plus a bpf-fs hostPath mount (/sys/fs/bpf), and adds an "ebpf-datapath" gRPC health sub-service to its liveness/readiness probes -- required unconditionally now that this container also hosts the eBPF/TC-BPF uSID datapath's load/attach/pin control daemon (installer.Run, #287); there's no flag left to gate this behind, since the datapath is the only forwarding path. Pins GALACTIC_CNI_EBPF_INTERFACES=eth1 for the containerlab topology, where every lab node is dual-homed and interface auto-detection picks the wrong (management-bridge) link over the actual transit-fabric one. Mounts bpffs on Kind nodes in CI (scripts/ci.sh) and updates TestCNITapInterface to run a privileged e2e pod with its own bpf-fs mount, starting the eBPF control daemon and waiting for vrf_table to be pinned before exercising CNI ADD -- the datapath being the only forwarding path means the e2e pod needs the same maps a production node's DaemonSet would already have pinned. Adds test-unit-root to test-e2e's dependency list. Stacked on #286 (CNI eBPF cutover + gateway fix) and #287 (GC/installer wiring) -- the visible diff includes both until they merge upstream; review focuses on the deploy/e2e files listed above. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Adds SweepEBPFVRFTable, a GC pass that reconciles the eBPF uSID datapath's vrf_table map entries against live BGPVRFInstance CRDs using a generation-cutoff scheme to avoid a register/sweep race. This runs from galactic-cni's "run" container rather than galactic-router's existing GC controller: the pinned vrf_table map only exists inside that container, which has the /sys/fs/bpf hostPath mount and CAP_BPF galactic-router's DaemonSet does not need for anything else. routerNamesForNode gains a fuller sibling, routersForNode, since the sweep needs each router's full Spec.SRv6Locator, not just its name. Wires installer.Run to load/attach/pin the eBPF datapath at startup (startEBPFDatapath), serve /metrics (Prometheus), report an "ebpf-datapath" gRPC health sub-service, and run the GC sweep on its own ticker. Adds the --metrics-port CLI flag to galactic-cni run. Depends only on #283 (attach/usidmap/metrics/uformat) -- no dependency on the CNI ADD cutover in #285/#286, since this reconciles directly against BGPRouter/BGPVRFInstance CRD state rather than anything registerEBPFDatapath writes to resourceTracker. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Grants the CNI DaemonSet's credential-refresh container CAP_BPF and CAP_NET_ADMIN plus a bpf-fs hostPath mount (/sys/fs/bpf), and adds an "ebpf-datapath" gRPC health sub-service to its liveness/readiness probes -- required unconditionally now that this container also hosts the eBPF/TC-BPF uSID datapath's load/attach/pin control daemon (installer.Run, #287); there's no flag left to gate this behind, since the datapath is the only forwarding path. Pins GALACTIC_CNI_EBPF_INTERFACES=eth1 for the containerlab topology, where every lab node is dual-homed and interface auto-detection picks the wrong (management-bridge) link over the actual transit-fabric one. Mounts bpffs on Kind nodes in CI (scripts/ci.sh) and updates TestCNITapInterface to run a privileged e2e pod with its own bpf-fs mount, starting the eBPF control daemon and waiting for vrf_table to be pinned before exercising CNI ADD -- the datapath being the only forwarding path means the e2e pod needs the same maps a production node's DaemonSet would already have pinned. Adds test-unit-root to test-e2e's dependency list. Stacked on #286 (CNI eBPF cutover + gateway fix) and #287 (GC/installer wiring) -- the visible diff includes both until they merge upstream; review focuses on the deploy/e2e files listed above. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
8521a2f to
7139fc2
Compare
Summary
The eBPF uSID ingress datapath (added in #285) decapsulates SRv6 traffic and calls
bpf_fib_lookup()to resolve the egress path for the inner packet, then redirects it straight to the resolved neighbor — entirely in-kernel, never touching the normal forwarding stack.bpf_fib_lookup()does not itself trigger ARP/NDP resolution the way ordinary kernel packet forwarding does, so without a pre-existing neighbor table entry it fails withBPF_FIB_LKUP_RET_NO_NEIGHand the datapath drops the packet. In practice: every cross-region packet to a pod that had never otherwise triggered NDP for its own address was silently and permanently blackholed.installGatewayNeighborfixes this by installing a permanent neighbor table entry mapping the pod's address to its guest veth's own known MAC at CNI ADD, so this resolution never depends on dynamic ARP/NDP. The guest MAC now flows frombuildVethResultthroughpublishBGPStateintoconfigureHostGateway; it'snilfor tap attachments, which have no separate guest-side link in this netns to resolve a MAC from — that path is out of scope here since this fix targets the veth-only bug it was found from.Stacked on #285 since the neighbor entry only matters once the eBPF datapath is the one performing the fib lookup.
Test plan
go build ./...go vet ./...go test ./internal/cni/...(one pre-existing unrelated failure:TestUpdateForwardRuleInvalidActionneeds aniptablesbinary not present in this sandbox)task lint(0 issues)Part of the eBPF uSID datapath cutover stack (base: #285).