feat: Cut CNI ADD over to eBPF uSID registration - #285
Conversation
9734305 to
25cf4ad
Compare
|
Same disclosure as elsewhere in the stack — ecv reviewed this through an AI shell, apologies. Questions, not verdicts. Two pieces of care worth naming. Checking
|
The eBPF uSID ingress datapath decapsulates SRv6 traffic and calls bpf_fib_lookup() to resolve the egress path for the inner packet, then redirects it straight to the resolved neighbor entirely in-kernel, never touching the normal forwarding stack. bpf_fib_lookup() does not itself trigger ARP/NDP resolution the way ordinary kernel packet forwarding does, so without a pre-existing neighbor table entry it fails with BPF_FIB_LKUP_RET_NO_NEIGH and the packet is dropped -- every cross-region packet to a pod that had never otherwise triggered NDP for its own address was silently and permanently blackholed. installGatewayNeighbor installs a permanent neighbor table entry mapping the pod's address to its guest veth's own known MAC at CNI ADD, so this resolution never depends on dynamic ARP/NDP. guestHWAddr now flows from buildVethResult through publishBGPState into configureHostGateway; nil for tap attachments, which have no separate guest-side link in this netns to resolve a MAC from. Stacked on #285 (eBPF uSID registration cutover), since the neighbor entry only matters once the eBPF datapath is the one doing the fib lookup. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Adds SweepEBPFVRFTable, a GC pass that reconciles the eBPF uSID datapath's vrf_table map entries against live BGPVRFInstance CRDs using a generation-cutoff scheme to avoid a register/sweep race. This runs from galactic-cni's "run" container rather than galactic-router's existing GC controller: the pinned vrf_table map only exists inside that container, which has the /sys/fs/bpf hostPath mount and CAP_BPF galactic-router's DaemonSet does not need for anything else. routerNamesForNode gains a fuller sibling, routersForNode, since the sweep needs each router's full Spec.SRv6Locator, not just its name. Wires installer.Run to load/attach/pin the eBPF datapath at startup (startEBPFDatapath), serve /metrics (Prometheus), report an "ebpf-datapath" gRPC health sub-service, and run the GC sweep on its own ticker. Adds the --metrics-port CLI flag to galactic-cni run. Depends only on #283 (attach/usidmap/metrics/uformat) -- no dependency on the CNI ADD cutover in #285/#286, since this reconciles directly against BGPRouter/BGPVRFInstance CRD state rather than anything registerEBPFDatapath writes to resourceTracker. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The eBPF uSID ingress datapath decapsulates SRv6 traffic and calls bpf_fib_lookup() to resolve the egress path for the inner packet, then redirects it straight to the resolved neighbor entirely in-kernel, never touching the normal forwarding stack. bpf_fib_lookup() does not itself trigger ARP/NDP resolution the way ordinary kernel packet forwarding does, so without a pre-existing neighbor table entry it fails with BPF_FIB_LKUP_RET_NO_NEIGH and the packet is dropped -- every cross-region packet to a pod that had never otherwise triggered NDP for its own address was silently and permanently blackholed. installGatewayNeighbor installs a permanent neighbor table entry mapping the pod's address to its guest veth's own known MAC at CNI ADD, so this resolution never depends on dynamic ARP/NDP. guestHWAddr now flows from buildVethResult through publishBGPState into configureHostGateway; nil for tap attachments, which have no separate guest-side link in this netns to resolve a MAC from. Stacked on #285 (eBPF uSID registration cutover), since the neighbor entry only matters once the eBPF datapath is the one doing the fib lookup. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Adds SweepEBPFVRFTable, a GC pass that reconciles the eBPF uSID datapath's vrf_table map entries against live BGPVRFInstance CRDs using a generation-cutoff scheme to avoid a register/sweep race. This runs from galactic-cni's "run" container rather than galactic-router's existing GC controller: the pinned vrf_table map only exists inside that container, which has the /sys/fs/bpf hostPath mount and CAP_BPF galactic-router's DaemonSet does not need for anything else. routerNamesForNode gains a fuller sibling, routersForNode, since the sweep needs each router's full Spec.SRv6Locator, not just its name. Wires installer.Run to load/attach/pin the eBPF datapath at startup (startEBPFDatapath), serve /metrics (Prometheus), report an "ebpf-datapath" gRPC health sub-service, and run the GC sweep on its own ticker. Adds the --metrics-port CLI flag to galactic-cni run. Depends only on #283 (attach/usidmap/metrics/uformat) -- no dependency on the CNI ADD cutover in #285/#286, since this reconciles directly against BGPRouter/BGPVRFInstance CRD state rather than anything registerEBPFDatapath writes to resourceTracker. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The eBPF uSID ingress datapath decapsulates SRv6 traffic and calls bpf_fib_lookup() to resolve the egress path for the inner packet, then redirects it straight to the resolved neighbor entirely in-kernel, never touching the normal forwarding stack. bpf_fib_lookup() does not itself trigger ARP/NDP resolution the way ordinary kernel packet forwarding does, so without a pre-existing neighbor table entry it fails with BPF_FIB_LKUP_RET_NO_NEIGH and the packet is dropped -- every cross-region packet to a pod that had never otherwise triggered NDP for its own address was silently and permanently blackholed. installGatewayNeighbor installs a permanent neighbor table entry mapping the pod's address to its guest veth's own known MAC at CNI ADD, so this resolution never depends on dynamic ARP/NDP. guestHWAddr now flows from buildVethResult through publishBGPState into configureHostGateway; nil for tap attachments, which have no separate guest-side link in this netns to resolve a MAC from. Stacked on #285 (eBPF uSID registration cutover), since the neighbor entry only matters once the eBPF datapath is the one doing the fib lookup. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
25cf4ad to
7139fc2
Compare
…p-on-error
Three independent fixes found while validating the eBPF uSID cutover,
none of which touch the eBPF datapath itself:
- internal/plumbing/srv6/egress.go: switches SEG6 encapsulation from
SEG6_IPTUN_MODE_ENCAP ("full" encap, always adds a Routing Header)
to the kernel's SEG6_IPTUN_MODE_ENCAP_RED ("reduced" encap, omits it
for a single-segment list). usid.c's ingress decap strips exactly a
fixed 40-byte outer IPv6 header with no allowance for a Routing
Header, so full encap corrupted every cross-region uSID packet.
Also adds netlink.Via/family handling so an IPv4 VPC prefix
(End.DT46) can egress through an IPv6 SID next-hop without violating
netlink's same-family route constraint, and CheckSEG6EncapRed, a
startup kernel-capability probe wired into cmd/galactic-router's
tenant-mode startup (root.go).
- internal/runtime/gobgp/{monitor,paths}.go: attaches the RFC 9252 BGP
Prefix-SID path attribute to every advertised path and prefers it
over the EVPN NLRI's GWIPAddress field on receive, since GWIPAddress
can't carry an IPv6 SID alongside an IPv4 EVPN Type-5 prefix.
- internal/reconcile/reconcile.go: BuildDesiredRouter now skips (logs +
continue) a single BGPAdvertisement whose SID can't be computed
instead of failing the whole router build.
Depends on #285's usid.go rewrite (ComputeSID onto the uFMT 48+16
layout): reconcile.go calls ComputeSID directly, and its tests assert
against uFMT-encoded values.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Summary
Fourth PR in the eBPF uSID datapath stack (on top of #283, the datapath control-plane packages).
Removes the legacy per-route netlink SRv6 ingress mechanism (
srv6.RouteIngressAdd/Del, all ofsrv6.go) entirely and replaces it with registration against the eBPF uSID datapath's pinned maps. The eBPF/TC-BPF datapath is now the only ingress/decap path for both veth and tap attachments — there's no dual-path coexistence to maintain.ComputeSIDis rewritten onto the shared uFMT 48+16 bit layout (internal/plumbing/ebpf/uformat, #281) instead of its previous ad hoc NodeID/VRFID/Function suffix, so the BGP control plane and the eBPF dataplane can never drift on bit positions.The CNI ADD path no longer derives the VRFID straight from the VPCAttachment identifier; it now allocates a 12-bit uFMT Argument per node from live
BGPVRFInstanceCRD state (allocateArgument), with a collision check (checkArgumentCollision) covering the allocate-then-create race between two concurrent ADDs.registerEBPFDatapath/unregisterEBPFDatapathwrite and roll back the three eBPF map entries (locator_table,function_table,vrf_table) for each attachment.Note
This branch is built directly on #283, not on #284 (the independent BGPAdvertisement prefix-merge fix) — those two touch nearby code in
bgp.gobut have no real dependency on each other. Merging #284 first and rebasing this stack is the cleanest order; otherwise expect a normal merge conflict inbgp.goat merge time.Gateway ARP/NDP neighbor priming (
installGatewayNeighbor) is a separate concern and lands in its own follow-up PR stacked on this one.Test plan
go build ./...go vet ./...go test ./internal/cni/... ./internal/plumbing/srv6/...(one pre-existing unrelated failure:TestUpdateForwardRuleInvalidActionneeds aniptablesbinary not present in this sandbox)task lint(0 issues)Part of the eBPF uSID datapath cutover stack (base: #283).