feat: Add TC-BPF uSID program and bit-layout library - #281
Conversation
Removes the legacy per-route netlink SRv6 ingress mechanism (srv6.RouteIngressAdd/Del, srv6.go) entirely and replaces it with registration against the eBPF uSID datapath's pinned maps (attach, usidmap -- #283): the eBPF/TC-BPF datapath is now the only ingress/decap path for both veth and tap attachments, so there's no dual-path coexistence to maintain. ComputeSID (internal/plumbing/srv6/usid.go) is rewritten onto the shared uFMT 48+16 bit layout (internal/plumbing/ebpf/uformat -- #281) instead of its previous ad hoc NodeID/VRFID/Function suffix, so the BGP control plane and the eBPF dataplane can never drift on bit positions. The CNI ADD path no longer derives the VRFID straight from the VPCAttachment identifier (vrfIDFromAttachment); it now allocates a 12-bit uFMT Argument per-node from live BGPVRFInstance CRD state (allocateArgument), with a collision check (checkArgumentCollision) covering the allocate-then-create race between two concurrent ADDs. registerEBPFDatapath/unregisterEBPFDatapath write and roll back the three eBPF map entries (locator_table, function_table, vrf_table) for each attachment. Stacked on #283 (eBPF datapath control-plane packages) and #284 (BGPAdvertisement prefix-merge fix, needed for this PR's publishBGPStateK8s changes to apply cleanly). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
|
Apologies up front — ecv ran this review through an AI shell rather than reading it line by line himself. Please forgive him. Everything below is a question, not a verdict. (Rewritten after reading the addressing spec in datum-cloud/enhancements#740, which answered some of what I'd originally asked and sharpened the rest.) First, the part that checks out:
Smaller things, take or leave: One coverage note rather than a question: every test that gets past Separately: #280 looks superseded by this one — the |
|
Thanks for the thorough pass — going through in order: 1. Behavior gating — real bug, fixed. 2. 3. Counter semantics — you're right that "claimed" packets showing healthy while everything actually gets dropped was a real blind spot. Added 4. PMTUD loss — accepted as a known cost of this cutover for now; added it to 5. No 6. Duplicate drop-reason enum — removed 7. License string — changed the ELF license to Smaller things — parenthesized Coverage note — agreed, tracked and not addressed here; will keep visible once #282's root CI job lands. #280 — agreed, it's superseded by this one; closing it. |
Adds the compiled TC-BPF usid_ingress program (uFMT 48+16 uSID decode/decap/redirect pipeline) with bpf2go-generated Go bindings, and the pure-Go uformat bit-layout library (Block/Node-ID/Function/Argument encode/decode) used by both the eBPF control plane and the BGP control plane so they never drift on bit positions. Pulls in github.com/cilium/ebpf. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Fixes and clarifications from ecv's review: - Gate on function_value.behavior before decap: a function_table hit whose behavior isn't BEHAVIOR_END_DT46 is now dropped (new DROP_REASON_UNSUPPORTED_BEHAVIOR) instead of falling through to DT46 decap, since #740 makes 0xE/0xF independent service universes. - Add vrf_value.dropped_packets, bumped at every post-vrf_table-match drop site, so packets-dropped_packets reflects actually-forwarded traffic instead of just claimed traffic. - Add bpf_skb_pull_data(skb, 0) at program entry so direct data/ data_end reads are safe regardless of the skb's arriving layout. - Change the BPF ELF license string from "Dual BSD/GPL" to "AGPL-3.0-or-later" to match the file's own SPDX header, with a comment on the gpl_only-helper tradeoff that follows from it. - Remove usid_test.go's duplicate drop-reason enum in favor of dropreason.go's exported constants. - Tighten the vrf_table max_entries comment to state the ~2-Block cap explicitly. - Document docs/agents/ARCHITECTURE.md's PMTUD/ICMPv6 PTB gap as a known constraint. - Minor: parenthesize Encode's bit composition, correct uformat's "no shift" package doc claim, fix dropreason.go's copyright year, document vrf_value.bytes's pre-decap semantics. Note: usid_bpfel.o/usid_bpfeb.o and their generated Go bindings still need `go generate ./internal/plumbing/ebpf/prog/...` (requires clang) before these usid.c changes take effect in the compiled datapath. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
a30784e to
ef73ddf
Compare
Teaches task build to regenerate the TC-BPF usid program's Go bindings (bpf2go via go generate, split into a new build:ebpf step ahead of build:binaries) and the CNI container image to do the same at image-build time, so clang/llvm is a build-time-only dependency and never ships in the runtime image. Adds a root-privileged rerun of the unit test suite in CI (test-unit-root) so tests gated on requireRoot(t) -- the BPF_PROG_TEST_RUN cases in usid_test.go, plus later root-gated tests -- actually execute instead of skipping, and a step that diffs the regenerated eBPF artifacts against the committed ones so usid.c and its compiled output can't silently drift apart. Stacked on #281 (adds the usid.c program this generates from). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Teaches task build to regenerate the TC-BPF usid program's Go bindings (bpf2go via go generate, split into a new build:ebpf step ahead of build:binaries) and the CNI container image to do the same at image-build time, so clang/llvm is a build-time-only dependency and never ships in the runtime image. Adds a root-privileged rerun of the unit test suite in CI (test-unit-root) so tests gated on requireRoot(t) -- the BPF_PROG_TEST_RUN cases in usid_test.go, plus later root-gated tests -- actually execute instead of skipping, and a step that diffs the regenerated eBPF artifacts against the committed ones so usid.c and its compiled output can't silently drift apart. Stacked on #281 (adds the usid.c program this generates from). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
ca05eaf to
3227471
Compare
Teaches task build to regenerate the TC-BPF usid program's Go bindings (bpf2go via go generate, split into a new build:ebpf step ahead of build:binaries) and the CNI container image to do the same at image-build time, so clang/llvm is a build-time-only dependency and never ships in the runtime image. Adds a root-privileged rerun of the unit test suite in CI (test-unit-root) so tests gated on requireRoot(t) -- the BPF_PROG_TEST_RUN cases in usid_test.go, plus later root-gated tests -- actually execute instead of skipping, and a step that diffs the regenerated eBPF artifacts against the committed ones so usid.c and its compiled output can't silently drift apart. Stacked on #281 (adds the usid.c program this generates from). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
ef73ddf changed the BPF ELF license section from "Dual BSD/GPL" to "AGPL-3.0-or-later" on the premise that "none of the helpers this program calls today are gpl_only." That premise is false: the VRF FIB lookup already calls bpf_fib_lookup(), which the kernel marks gpl_only (net/core/filter.c's bpf_fib_lookup_proto). The kernel's license_is_gpl_compatible() check only recognizes a fixed whitelist of exact strings, and "AGPL-3.0-or-later" isn't on it, so the program now fails to load at all: "cannot call GPL-restricted function from non-GPL compatible program". This surfaced as PR #282's new CI jobs failing -- the Build job's drift check flagged the stale committed .o/.go (usid.c had changed but they hadn't been regenerated since), and once regenerated, Unit Tests (root) then hit the verifier rejection above on every FIB-lookup-path test. Set the license string to plain "GPL" -- not "Dual BSD/GPL"; this program isn't itself dual-licensed, so it declares the license it actually means rather than a disjunction it doesn't. Correct the surrounding comment accordingly: the ELF license section governs which helpers the verifier allows and is independent of (doesn't relicense) the file's own AGPL-3.0-or-later SPDX header, same as Cilium, Katran, and every other AGPL/Apache/BSD-licensed project embedding a BPF datapath. Regenerate usid_bpfel.o/usid_bpfeb.o and their Go bindings from the corrected usid.c (also picking up the vrf_value.dropped_packets field ef73ddf added, which needed the same regeneration its own commit message flagged as still pending). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
3227471 to
d666878
Compare
Teaches task build to regenerate the TC-BPF usid program's Go bindings (bpf2go via go generate, split into a new build:ebpf step ahead of build:binaries) and the CNI container image to do the same at image-build time, so clang/llvm is a build-time-only dependency and never ships in the runtime image. Adds a root-privileged rerun of the unit test suite in CI (test-unit-root) so tests gated on requireRoot(t) -- the BPF_PROG_TEST_RUN cases in usid_test.go, plus later root-gated tests -- actually execute instead of skipping, and a step that diffs the regenerated eBPF artifacts against the committed ones so usid.c and its compiled output can't silently drift apart. Stacked on #281 (adds the usid.c program this generates from). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Teaches task build to regenerate the TC-BPF usid program's Go bindings (bpf2go via go generate, split into a new build:ebpf step ahead of build:binaries) and the CNI container image to do the same at image-build time, so clang/llvm is a build-time-only dependency and never ships in the runtime image. Adds a root-privileged rerun of the unit test suite in CI (test-unit-root) so tests gated on requireRoot(t) -- the BPF_PROG_TEST_RUN cases in usid_test.go, plus later root-gated tests -- actually execute instead of skipping, and a step that diffs the regenerated eBPF artifacts against the committed ones so usid.c and its compiled output can't silently drift apart. Stacked on #281 (adds the usid.c program this generates from). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Removes the legacy per-route netlink SRv6 ingress mechanism (srv6.RouteIngressAdd/Del, srv6.go) entirely and replaces it with registration against the eBPF uSID datapath's pinned maps (attach, usidmap -- #283): the eBPF/TC-BPF datapath is now the only ingress/decap path for both veth and tap attachments, so there's no dual-path coexistence to maintain. ComputeSID (internal/plumbing/srv6/usid.go) is rewritten onto the shared uFMT 48+16 bit layout (internal/plumbing/ebpf/uformat -- #281) instead of its previous ad hoc NodeID/VRFID/Function suffix, so the BGP control plane and the eBPF dataplane can never drift on bit positions. The CNI ADD path no longer derives the VRFID straight from the VPCAttachment identifier (vrfIDFromAttachment); it now allocates a 12-bit uFMT Argument per-node from live BGPVRFInstance CRD state (allocateArgument), with a collision check (checkArgumentCollision) covering the allocate-then-create race between two concurrent ADDs. registerEBPFDatapath/unregisterEBPFDatapath write and roll back the three eBPF map entries (locator_table, function_table, vrf_table) for each attachment. Stacked on #283 (eBPF datapath control-plane packages) and #284 (BGPAdvertisement prefix-merge fix, needed for this PR's publishBGPStateK8s changes to apply cleanly). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Teaches task build to regenerate the TC-BPF usid program's Go bindings (bpf2go via go generate, split into a new build:ebpf step ahead of build:binaries) and the CNI container image to do the same at image-build time, so clang/llvm is a build-time-only dependency and never ships in the runtime image. Adds a root-privileged rerun of the unit test suite in CI (test-unit-root) so tests gated on requireRoot(t) -- the BPF_PROG_TEST_RUN cases in usid_test.go, plus later root-gated tests -- actually execute instead of skipping, and a step that diffs the regenerated eBPF artifacts against the committed ones so usid.c and its compiled output can't silently drift apart. Stacked on #281 (adds the usid.c program this generates from). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Teaches task build to regenerate the TC-BPF usid program's Go bindings (bpf2go via go generate, split into a new build:ebpf step ahead of build:binaries) and the CNI container image to do the same at image-build time, so clang/llvm is a build-time-only dependency and never ships in the runtime image. Adds a root-privileged rerun of the unit test suite in CI (test-unit-root) so tests gated on requireRoot(t) -- the BPF_PROG_TEST_RUN cases in usid_test.go, plus later root-gated tests -- actually execute instead of skipping, and a step that diffs the regenerated eBPF artifacts against the committed ones so usid.c and its compiled output can't silently drift apart. Stacked on #281 (adds the usid.c program this generates from). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Adds the Go control-plane library for the TC-BPF uSID datapath: - preflight: kernel capability gate (BTF, HASH maps, SCHED_CLS, bpf_fib_lookup with VRF-tbid support) run before attempting to load the program at all. - usidmap: typed read/write/reconcile API for the three kernel maps (locator_table, function_table, vrf_table) the program consults. - attach: load/pin/attach/detach/watch lifecycle for the TC-BPF ingress hook, including netlink-driven re-attachment on interface or route change, gated by the new GALACTIC_CNI_EBPF_INTERFACES env var (internal/config) for multi-homed nodes where auto-detection is ambiguous. - metrics: Prometheus collector plus load/attach event counters. Every package here is independently unit-testable against fakes (faketable_test.go, a mock kernel prober, fake closers) -- nothing outside this tree calls any of it yet. Stacked on #281 (usid.c/uformat) and #282 (codegen toolchain). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Removes the legacy per-route netlink SRv6 ingress mechanism (srv6.RouteIngressAdd/Del, srv6.go) entirely and replaces it with registration against the eBPF uSID datapath's pinned maps (attach, usidmap -- #283): the eBPF/TC-BPF datapath is now the only ingress/decap path for both veth and tap attachments, so there's no dual-path coexistence to maintain. ComputeSID (internal/plumbing/srv6/usid.go) is rewritten onto the shared uFMT 48+16 bit layout (internal/plumbing/ebpf/uformat -- #281) instead of its previous ad hoc NodeID/VRFID/Function suffix, so the BGP control plane and the eBPF dataplane can never drift on bit positions. The CNI ADD path no longer derives the VRFID straight from the VPCAttachment identifier (vrfIDFromAttachment); it now allocates a 12-bit uFMT Argument per-node from live BGPVRFInstance CRD state (allocateArgument), with a collision check (checkArgumentCollision) covering the allocate-then-create race between two concurrent ADDs. registerEBPFDatapath/unregisterEBPFDatapath write and roll back the three eBPF map entries (locator_table, function_table, vrf_table) for each attachment. Stacked on #283 (eBPF datapath control-plane packages) and #284 (BGPAdvertisement prefix-merge fix, needed for this PR's publishBGPStateK8s changes to apply cleanly). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Adds the Go control-plane library for the TC-BPF uSID datapath: - preflight: kernel capability gate (BTF, HASH maps, SCHED_CLS, bpf_fib_lookup with VRF-tbid support) run before attempting to load the program at all. - usidmap: typed read/write/reconcile API for the three kernel maps (locator_table, function_table, vrf_table) the program consults. - attach: load/pin/attach/detach/watch lifecycle for the TC-BPF ingress hook, including netlink-driven re-attachment on interface or route change, gated by the new GALACTIC_CNI_EBPF_INTERFACES env var (internal/config) for multi-homed nodes where auto-detection is ambiguous. - metrics: Prometheus collector plus load/attach event counters. Every package here is independently unit-testable against fakes (faketable_test.go, a mock kernel prober, fake closers) -- nothing outside this tree calls any of it yet. Stacked on #281 (usid.c/uformat) and #282 (codegen toolchain). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Removes the legacy per-route netlink SRv6 ingress mechanism (srv6.RouteIngressAdd/Del, srv6.go) entirely and replaces it with registration against the eBPF uSID datapath's pinned maps (attach, usidmap -- #283): the eBPF/TC-BPF datapath is now the only ingress/decap path for both veth and tap attachments, so there's no dual-path coexistence to maintain. ComputeSID (internal/plumbing/srv6/usid.go) is rewritten onto the shared uFMT 48+16 bit layout (internal/plumbing/ebpf/uformat -- #281) instead of its previous ad hoc NodeID/VRFID/Function suffix, so the BGP control plane and the eBPF dataplane can never drift on bit positions. The CNI ADD path no longer derives the VRFID straight from the VPCAttachment identifier (vrfIDFromAttachment); it now allocates a 12-bit uFMT Argument per-node from live BGPVRFInstance CRD state (allocateArgument), with a collision check (checkArgumentCollision) covering the allocate-then-create race between two concurrent ADDs. registerEBPFDatapath/unregisterEBPFDatapath write and roll back the three eBPF map entries (locator_table, function_table, vrf_table) for each attachment. Stacked on #283 (eBPF datapath control-plane packages) and #284 (BGPAdvertisement prefix-merge fix, needed for this PR's publishBGPStateK8s changes to apply cleanly). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Summary
This is the first PR in a stack that cuts SRv6 ingress over from a per-route netlink mechanism to a shared eBPF/TC-BPF datapath. It lands the lowest-level, dependency-free pieces on their own so the rest of the stack has something to build on.
Adds the compiled TC-BPF
usid_ingressprogram (decodes/decaps/redirects uSID traffic) with its bpf2go-generated Go bindings, and a pure-Gouformatlibrary that encodes/decodes the uFMT 48+16 SID layout (Block / Node-ID / Function / Argument). Both the eBPF datapath and the BGP control plane will depend onuformatso they can never drift on bit positions. Pulls ingithub.com/cilium/ebpf.Nothing outside
internal/plumbing/ebpf/{prog,uformat}changes here — no call sites exist yet, so this is inert until the next PR in the stack wires it up.Test plan
go build ./...go test ./internal/plumbing/ebpf/...(BPF-map/BPF_PROG_TEST_RUNcases skip without root/CAP_BPF, as expected in this environment)Part of the eBPF uSID datapath cutover (stack: this PR → codegen toolchain → datapath control-plane packages → CNI cutover → GC/installer wiring → deploy/e2e wiring).