Skip to content

fix: Prime gateway ARP/NDP neighbor for eBPF fib lookup - #286

Closed
privateip wants to merge 0 commit into
pr3-cni-ebpf-cutoverfrom
pr3b-gateway-neighbor
Closed

fix: Prime gateway ARP/NDP neighbor for eBPF fib lookup#286
privateip wants to merge 0 commit into
pr3-cni-ebpf-cutoverfrom
pr3b-gateway-neighbor

Conversation

@privateip

Copy link
Copy Markdown
Contributor

Summary

The eBPF uSID ingress datapath (added in #285) decapsulates SRv6 traffic and calls bpf_fib_lookup() to resolve the egress path for the inner packet, then redirects it straight to the resolved neighbor — entirely in-kernel, never touching the normal forwarding stack. bpf_fib_lookup() does not itself trigger ARP/NDP resolution the way ordinary kernel packet forwarding does, so without a pre-existing neighbor table entry it fails with BPF_FIB_LKUP_RET_NO_NEIGH and the datapath drops the packet. In practice: every cross-region packet to a pod that had never otherwise triggered NDP for its own address was silently and permanently blackholed.

installGatewayNeighbor fixes this by installing a permanent neighbor table entry mapping the pod's address to its guest veth's own known MAC at CNI ADD, so this resolution never depends on dynamic ARP/NDP. The guest MAC now flows from buildVethResult through publishBGPState into configureHostGateway; it's nil for tap attachments, which have no separate guest-side link in this netns to resolve a MAC from — that path is out of scope here since this fix targets the veth-only bug it was found from.

Stacked on #285 since the neighbor entry only matters once the eBPF datapath is the one performing the fib lookup.

Test plan

  • go build ./...
  • go vet ./...
  • go test ./internal/cni/... (one pre-existing unrelated failure: TestUpdateForwardRuleInvalidAction needs an iptables binary not present in this sandbox)
  • task lint (0 issues)

Part of the eBPF uSID datapath cutover stack (base: #285).

@ecv

ecv commented Aug 5, 2026

Copy link
Copy Markdown

Same disclosure as on the rest of the stack — ecv reviewed this through an AI shell, apologies. Two questions.

  1. This primes the neighbor entry for veth only; tap passes nil and skips it. The comment says tap is out of scope because there is no guest-side link in this netns to read a MAC from, which I follow — the guest picks its own MAC, so the fix is not simply symmetric. But feat: Cut CNI ADD over to eBPF uSID registration #285 registers tap attachments into vrf_table with their own egress kind, and feat(ebpf): add usid TC-BPF program and generated bindings #280's docs call both modes supported, so a tap attachment reaches the same bpf_fib_lookup with the same empty neighbor table. Does it blackhole the same way until the VM happens to send something that resolves it? If it does, is "supported" the right word for tap until that path has an answer too?

  2. The comment attributes the drop to DROP_REASON_FIB_LOOKUP_FAILED, but usid.c maps BPF_FIB_LKUP_RET_NO_NEIGH to DROP_REASON_FIB_NO_NEIGH and only falls through to FIB_LOOKUP_FAILED for return codes it does not name. Anyone chasing this bug from the metrics would watch the wrong series. Worth correcting the comment — and the distinct counter is what made the diagnosis possible, so it is worth naming right.

@ecv ecv left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i'll pause now

privateip added a commit that referenced this pull request Aug 6, 2026
Grants the CNI DaemonSet's credential-refresh container CAP_BPF and
CAP_NET_ADMIN plus a bpf-fs hostPath mount (/sys/fs/bpf), and adds an
"ebpf-datapath" gRPC health sub-service to its liveness/readiness
probes -- required unconditionally now that this container also hosts
the eBPF/TC-BPF uSID datapath's load/attach/pin control daemon
(installer.Run, #287); there's no flag left to gate this behind, since
the datapath is the only forwarding path.

Pins GALACTIC_CNI_EBPF_INTERFACES=eth1 for the containerlab topology,
where every lab node is dual-homed and interface auto-detection picks
the wrong (management-bridge) link over the actual transit-fabric one.

Mounts bpffs on Kind nodes in CI (scripts/ci.sh) and updates
TestCNITapInterface to run a privileged e2e pod with its own bpf-fs
mount, starting the eBPF control daemon and waiting for vrf_table to
be pinned before exercising CNI ADD -- the datapath being the only
forwarding path means the e2e pod needs the same maps a production
node's DaemonSet would already have pinned. Adds test-unit-root to
test-e2e's dependency list.

Stacked on #286 (CNI eBPF cutover + gateway fix) and #287 (GC/installer
wiring) -- the visible diff includes both until they merge upstream;
review focuses on the deploy/e2e files listed above.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
privateip added a commit that referenced this pull request Aug 6, 2026
Adds SweepEBPFVRFTable, a GC pass that reconciles the eBPF uSID
datapath's vrf_table map entries against live BGPVRFInstance CRDs
using a generation-cutoff scheme to avoid a register/sweep race. This
runs from galactic-cni's "run" container rather than
galactic-router's existing GC controller: the pinned vrf_table map
only exists inside that container, which has the /sys/fs/bpf hostPath
mount and CAP_BPF galactic-router's DaemonSet does not need for
anything else. routerNamesForNode gains a fuller sibling,
routersForNode, since the sweep needs each router's full
Spec.SRv6Locator, not just its name.

Wires installer.Run to load/attach/pin the eBPF datapath at startup
(startEBPFDatapath), serve /metrics (Prometheus), report an
"ebpf-datapath" gRPC health sub-service, and run the GC sweep on its
own ticker. Adds the --metrics-port CLI flag to galactic-cni run.

Depends only on #283 (attach/usidmap/metrics/uformat) -- no
dependency on the CNI ADD cutover in #285/#286, since this reconciles
directly against BGPRouter/BGPVRFInstance CRD state rather than
anything registerEBPFDatapath writes to resourceTracker.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
privateip added a commit that referenced this pull request Aug 6, 2026
Grants the CNI DaemonSet's credential-refresh container CAP_BPF and
CAP_NET_ADMIN plus a bpf-fs hostPath mount (/sys/fs/bpf), and adds an
"ebpf-datapath" gRPC health sub-service to its liveness/readiness
probes -- required unconditionally now that this container also hosts
the eBPF/TC-BPF uSID datapath's load/attach/pin control daemon
(installer.Run, #287); there's no flag left to gate this behind, since
the datapath is the only forwarding path.

Pins GALACTIC_CNI_EBPF_INTERFACES=eth1 for the containerlab topology,
where every lab node is dual-homed and interface auto-detection picks
the wrong (management-bridge) link over the actual transit-fabric one.

Mounts bpffs on Kind nodes in CI (scripts/ci.sh) and updates
TestCNITapInterface to run a privileged e2e pod with its own bpf-fs
mount, starting the eBPF control daemon and waiting for vrf_table to
be pinned before exercising CNI ADD -- the datapath being the only
forwarding path means the e2e pod needs the same maps a production
node's DaemonSet would already have pinned. Adds test-unit-root to
test-e2e's dependency list.

Stacked on #286 (CNI eBPF cutover + gateway fix) and #287 (GC/installer
wiring) -- the visible diff includes both until they merge upstream;
review focuses on the deploy/e2e files listed above.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
privateip added a commit that referenced this pull request Aug 6, 2026
Adds SweepEBPFVRFTable, a GC pass that reconciles the eBPF uSID
datapath's vrf_table map entries against live BGPVRFInstance CRDs
using a generation-cutoff scheme to avoid a register/sweep race. This
runs from galactic-cni's "run" container rather than
galactic-router's existing GC controller: the pinned vrf_table map
only exists inside that container, which has the /sys/fs/bpf hostPath
mount and CAP_BPF galactic-router's DaemonSet does not need for
anything else. routerNamesForNode gains a fuller sibling,
routersForNode, since the sweep needs each router's full
Spec.SRv6Locator, not just its name.

Wires installer.Run to load/attach/pin the eBPF datapath at startup
(startEBPFDatapath), serve /metrics (Prometheus), report an
"ebpf-datapath" gRPC health sub-service, and run the GC sweep on its
own ticker. Adds the --metrics-port CLI flag to galactic-cni run.

Depends only on #283 (attach/usidmap/metrics/uformat) -- no
dependency on the CNI ADD cutover in #285/#286, since this reconciles
directly against BGPRouter/BGPVRFInstance CRD state rather than
anything registerEBPFDatapath writes to resourceTracker.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
privateip added a commit that referenced this pull request Aug 6, 2026
Grants the CNI DaemonSet's credential-refresh container CAP_BPF and
CAP_NET_ADMIN plus a bpf-fs hostPath mount (/sys/fs/bpf), and adds an
"ebpf-datapath" gRPC health sub-service to its liveness/readiness
probes -- required unconditionally now that this container also hosts
the eBPF/TC-BPF uSID datapath's load/attach/pin control daemon
(installer.Run, #287); there's no flag left to gate this behind, since
the datapath is the only forwarding path.

Pins GALACTIC_CNI_EBPF_INTERFACES=eth1 for the containerlab topology,
where every lab node is dual-homed and interface auto-detection picks
the wrong (management-bridge) link over the actual transit-fabric one.

Mounts bpffs on Kind nodes in CI (scripts/ci.sh) and updates
TestCNITapInterface to run a privileged e2e pod with its own bpf-fs
mount, starting the eBPF control daemon and waiting for vrf_table to
be pinned before exercising CNI ADD -- the datapath being the only
forwarding path means the e2e pod needs the same maps a production
node's DaemonSet would already have pinned. Adds test-unit-root to
test-e2e's dependency list.

Stacked on #286 (CNI eBPF cutover + gateway fix) and #287 (GC/installer
wiring) -- the visible diff includes both until they merge upstream;
review focuses on the deploy/e2e files listed above.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@privateip privateip closed this Aug 6, 2026
@privateip
privateip force-pushed the pr3b-gateway-neighbor branch from 8521a2f to 7139fc2 Compare August 6, 2026 21:17
@privateip
privateip deleted the pr3b-gateway-neighbor branch August 6, 2026 21:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants