feat: eBPF vrf_table GC sweep and control-daemon startup - #287
Conversation
Adds SweepEBPFVRFTable, a GC pass that reconciles the eBPF uSID datapath's vrf_table map entries against live BGPVRFInstance CRDs using a generation-cutoff scheme to avoid a register/sweep race. This runs from galactic-cni's "run" container rather than galactic-router's existing GC controller: the pinned vrf_table map only exists inside that container, which has the /sys/fs/bpf hostPath mount and CAP_BPF galactic-router's DaemonSet does not need for anything else. routerNamesForNode gains a fuller sibling, routersForNode, since the sweep needs each router's full Spec.SRv6Locator, not just its name. Wires installer.Run to load/attach/pin the eBPF datapath at startup (startEBPFDatapath), serve /metrics (Prometheus), report an "ebpf-datapath" gRPC health sub-service, and run the GC sweep on its own ticker. Adds the --metrics-port CLI flag to galactic-cni run. Depends only on #283 (attach/usidmap/metrics/uformat) -- no dependency on the CNI ADD cutover in #285/#286, since this reconciles directly against BGPRouter/BGPVRFInstance CRD state rather than anything registerEBPFDatapath writes to resourceTracker. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
|
Rewritten after reading the rest of the stack, #283 through #289; one of my original questions was already answered by #288, and two others turned out to have direct evidence behind them. Two things worth crediting first: capturing the generation cutoff before listing the CRDs, and bailing out entirely when zero BGPRouters come back rather than treating that as "zero live attachments." The second is the kind of guard that usually only gets written after it has already wiped a node once.
Smaller: the resolved interface list is stored in the datapath state and never read again, since the health handle re-resolves internally. And the reconcile error path bumps the error count by one regardless of how many deletions failed, so the count reads as "this sweep had errors" rather than a magnitude. Same CI note as the rest of the stack: this targets |
Grants the CNI DaemonSet's credential-refresh container CAP_BPF and CAP_NET_ADMIN plus a bpf-fs hostPath mount (/sys/fs/bpf), and adds an "ebpf-datapath" gRPC health sub-service to its liveness/readiness probes -- required unconditionally now that this container also hosts the eBPF/TC-BPF uSID datapath's load/attach/pin control daemon (installer.Run, #287); there's no flag left to gate this behind, since the datapath is the only forwarding path. Pins GALACTIC_CNI_EBPF_INTERFACES=eth1 for the containerlab topology, where every lab node is dual-homed and interface auto-detection picks the wrong (management-bridge) link over the actual transit-fabric one. Mounts bpffs on Kind nodes in CI (scripts/ci.sh) and updates TestCNITapInterface to run a privileged e2e pod with its own bpf-fs mount, starting the eBPF control daemon and waiting for vrf_table to be pinned before exercising CNI ADD -- the datapath being the only forwarding path means the e2e pod needs the same maps a production node's DaemonSet would already have pinned. Adds test-unit-root to test-e2e's dependency list. Stacked on #286 (CNI eBPF cutover + gateway fix) and #287 (GC/installer wiring) -- the visible diff includes both until they merge upstream; review focuses on the deploy/e2e files listed above. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Grants the CNI DaemonSet's credential-refresh container CAP_BPF and CAP_NET_ADMIN plus a bpf-fs hostPath mount (/sys/fs/bpf), and adds an "ebpf-datapath" gRPC health sub-service to its liveness/readiness probes -- required unconditionally now that this container also hosts the eBPF/TC-BPF uSID datapath's load/attach/pin control daemon (installer.Run, #287); there's no flag left to gate this behind, since the datapath is the only forwarding path. Pins GALACTIC_CNI_EBPF_INTERFACES=eth1 for the containerlab topology, where every lab node is dual-homed and interface auto-detection picks the wrong (management-bridge) link over the actual transit-fabric one. Mounts bpffs on Kind nodes in CI (scripts/ci.sh) and updates TestCNITapInterface to run a privileged e2e pod with its own bpf-fs mount, starting the eBPF control daemon and waiting for vrf_table to be pinned before exercising CNI ADD -- the datapath being the only forwarding path means the e2e pod needs the same maps a production node's DaemonSet would already have pinned. Adds test-unit-root to test-e2e's dependency list. Stacked on #286 (CNI eBPF cutover + gateway fix) and #287 (GC/installer wiring) -- the visible diff includes both until they merge upstream; review focuses on the deploy/e2e files listed above. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…t EEXIST Three independent attach-path fixes from ecv's review of #283, bundled together since all three land in attach.go/usid.c: 1. (point 2) usid_ingress's fail-open paths (not IPv6, too short to parse, no locator_table match -- design plan R6) returned TC_ACT_OK. This filter attaches direct-action at a fixed tc priority, and Cilium attaches its own tc/bpf programs to the same native-device ingress hook on these hosts; in direct-action mode TC_ACT_OK is a final verdict that ends the qdisc's filter chain, so a packet this program doesn't claim would never reach a colocated Cilium filter at a later priority. Switched every pre-locator-match fail-open path to TC_ACT_UNSPEC, which hands off to the next filter instead. Every fail-open path *after* the locator_table match (this program has claimed the packet) is unaffected -- those are, and remain, TC_ACT_SHOT. Also made the filter's tc priority overridable via GALACTIC_CNI_EBPF_FILTER_PRIORITY (default unchanged at 1), mirroring the same override internal/vmtap/config.go already exposes for its own Cilium-priority-collision risk. 2. (point 3) Load pinned every map by name and treated any schema mismatch against an existing pin (ebpf.ErrMapIncompatible -- e.g. a changed value struct size or max_entries) as fatal. vrf_value grew egress_kind this cycle and vrf_table sizing is an open question on #287, so the next such change would crashloop every node until an operator manually deleted the stale pins. Load now unpins and recreates any incompatible map before retrying the load once -- every map here is control-plane-owned and reconstructable (usidmap.Register/the GC controller repopulate it), so losing its contents across a schema change is the correct trade-off against a crashloop. 3. (point 8) ensureClsact listed qdiscs then unconditionally added one if absent, racy against any other agent (notably Cilium) doing the same thing to the same device between the two calls. QdiscAdd returning EEXIST in that window is now treated as success, since the qdisc ensureClsact wanted to exist now does. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Grants the CNI DaemonSet's credential-refresh container CAP_BPF and CAP_NET_ADMIN plus a bpf-fs hostPath mount (/sys/fs/bpf), and adds an "ebpf-datapath" gRPC health sub-service to its liveness/readiness probes -- required unconditionally now that this container also hosts the eBPF/TC-BPF uSID datapath's load/attach/pin control daemon (installer.Run, #287); there's no flag left to gate this behind, since the datapath is the only forwarding path. Pins GALACTIC_CNI_EBPF_INTERFACES=eth1 for the containerlab topology, where every lab node is dual-homed and interface auto-detection picks the wrong (management-bridge) link over the actual transit-fabric one. Mounts bpffs on Kind nodes in CI (scripts/ci.sh) and updates TestCNITapInterface to run a privileged e2e pod with its own bpf-fs mount, starting the eBPF control daemon and waiting for vrf_table to be pinned before exercising CNI ADD -- the datapath being the only forwarding path means the e2e pod needs the same maps a production node's DaemonSet would already have pinned. Adds test-unit-root to test-e2e's dependency list. Stacked on #286 (CNI eBPF cutover + gateway fix) and #287 (GC/installer wiring) -- the visible diff includes both until they merge upstream; review focuses on the deploy/e2e files listed above. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…t EEXIST Three independent attach-path fixes from ecv's review of #283, bundled together since all three land in attach.go/usid.c: 1. (point 2) usid_ingress's fail-open paths (not IPv6, too short to parse, no locator_table match -- design plan R6) returned TC_ACT_OK. This filter attaches direct-action at a fixed tc priority, and Cilium attaches its own tc/bpf programs to the same native-device ingress hook on these hosts; in direct-action mode TC_ACT_OK is a final verdict that ends the qdisc's filter chain, so a packet this program doesn't claim would never reach a colocated Cilium filter at a later priority. Switched every pre-locator-match fail-open path to TC_ACT_UNSPEC, which hands off to the next filter instead. Every fail-open path *after* the locator_table match (this program has claimed the packet) is unaffected -- those are, and remain, TC_ACT_SHOT. Also made the filter's tc priority overridable via GALACTIC_CNI_EBPF_FILTER_PRIORITY (default unchanged at 1), mirroring the same override internal/vmtap/config.go already exposes for its own Cilium-priority-collision risk. 2. (point 3) Load pinned every map by name and treated any schema mismatch against an existing pin (ebpf.ErrMapIncompatible -- e.g. a changed value struct size or max_entries) as fatal. vrf_value grew egress_kind this cycle and vrf_table sizing is an open question on #287, so the next such change would crashloop every node until an operator manually deleted the stale pins. Load now unpins and recreates any incompatible map before retrying the load once -- every map here is control-plane-owned and reconstructable (usidmap.Register/the GC controller repopulate it), so losing its contents across a schema change is the correct trade-off against a crashloop. 3. (point 8) ensureClsact listed qdiscs then unconditionally added one if absent, racy against any other agent (notably Cilium) doing the same thing to the same device between the two calls. QdiscAdd returning EEXIST in that window is now treated as success, since the qdisc ensureClsact wanted to exist now does. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Summary
Adds
SweepEBPFVRFTable, a GC pass that reconciles the eBPF uSID datapath'svrf_tablemap entries against liveBGPVRFInstanceCRDs, using a generation-cutoff scheme to avoid a register/sweep race. This runs fromgalactic-cni'sruncontainer rather thangalactic-router's existing GC controller: the pinnedvrf_tablemap only exists inside that container, which has the/sys/fs/bpfhostPath mount andCAP_BPFthatgalactic-router's DaemonSet doesn't need for anything else.routerNamesForNodegains a fuller sibling,routersForNode, since the sweep needs each router's fullSpec.SRv6Locator, not just its name.Wires
installer.Runto load/attach/pin the eBPF datapath at startup, serve/metrics(Prometheus), report anebpf-datapathgRPC health sub-service, and run the GC sweep on its own ticker. Adds a--metrics-portCLI flag togalactic-cni run.Note
This depends only on #283 (the datapath control-plane packages), not on the CNI ADD cutover in #285/#286 — the sweep reconciles directly against
BGPRouter/BGPVRFInstanceCRD state rather than anythingregisterEBPFDatapathwrites, so it's a sibling branch in the stack rather than stacked on top of the CNI cutover.Test plan
go build ./...go vet ./...go test ./internal/gc/... ./internal/installer/... ./cmd/...task lint(0 issues)Part of the eBPF uSID datapath cutover stack (base: #283).