Skip to content

feat(ebpf): add usid TC-BPF program and generated bindings #280

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
privateip wants to merge 1 commit into from
Closed

feat(ebpf): add usid TC-BPF program and generated bindings #280

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion AGENTS.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,7 @@ task test:e2e # Kind cluster lifecycle test
task lint # golangci-lint; lint-fix applies safe auto-fixes
```

There is no production release image build in this repo (`task docker-build` and the release workflow were removed after the shared image was found to advertise `galactic-router` without ever building it — see [docs/agents/ARCHITECTURE.md](docs/agents/ARCHITECTURE.md#known-constraints)). `containers/galactic-cni/Dockerfile` exists solely for `task test:e2e`.
Production images are built by `.github/workflows/publish.yaml`: `publish-galactic-cni-image` and `publish-galactic-router-image` each build and push their own image (`ghcr.io/datum-cloud/galactic-cni`, `ghcr.io/datum-cloud/galactic-router`) from their respective `containers/*/Dockerfile`, and `publish-kustomize-bundles` pushes `config/` as an OCI Kustomize bundle with each job's real published tag stamped in. This replaced the old single-image `release.yaml`, which built one shared image that advertised `galactic-router` without ever building it — see [docs/agents/ARCHITECTURE.md](docs/agents/ARCHITECTURE.md#cicd) for that history. `containers/galactic-cni/Dockerfile` is used by both `task test:e2e` and `publish.yaml`.

**Before every PR:** `task ci` (lint → build → test:unit → test:e2e).

Expand Down
308 changes: 180 additions & 128 deletions docs/agents/ARCHITECTURE.md
View file
Open in desktop

Large diffs are not rendered by default.

30 changes: 15 additions & 15 deletions docs/cni-cmd-sequence.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -64,18 +64,18 @@ sequenceDiagram
CNI->>CNI: AddrAdd(gateway/128) on host veth
CNI->>CNI: RouteAdd(subnet to host veth) in VRF table

CNI->>CNI: decode VPC hex + VRFID
CNI->>CNI: decode VPC hex
CNI->>K8s: newK8sClient()

CNI->>CNI: publishBGPStateK8s() (retry loop)
activate CNI
CNI->>K8s: lookupBGPRouter(node)
CNI->>CNI: resolveSRv6SID(locator, nodeID, vrfID)
CNI->>SRv6: RouteIngressAdd(sid, vpc, attachment)
activate SRv6
SRv6->>SRv6: seg6local End.DT46 route
SRv6-->>CNI: ok
deactivate SRv6
CNI->>CNI: allocateArgument(ctx, k8s, namespace, routerName, vrfInstanceName) -> vrfID (local per-node Argument)
CNI->>CNI: registerEBPFDatapath(bgp, vpc, attachment, ifaceType, vrfID, PinDir)
activate EBPF
EBPF->>EBPF: Locator.Register / Function.Register / VRF.Register (locator_table, function_table, vrf_table)
EBPF-->>CNI: ok (fatal to ADD on error -- this is the only forwarding path)
deactivate EBPF
CNI->>K8s: CreateOrUpdate BGPVRFInstance
CNI->>K8s: CreateOrUpdate BGPAdvertisement(prefix, annotations)
CNI-->>Runtime: ok
Expand Down Expand Up @@ -128,18 +128,18 @@ sequenceDiagram

CNI->>CNI: buildTapResult(ipamResult) + PrintResult()

CNI->>CNI: decode VPC hex + VRFID
CNI->>CNI: decode VPC hex
CNI->>K8s: newK8sClient()

CNI->>CNI: publishBGPStateK8s() (retry loop)
activate CNI
CNI->>K8s: lookupBGPRouter(node)
CNI->>CNI: resolveSRv6SID(locator, nodeID, vrfID)
CNI->>SRv6: RouteIngressAdd(sid, vpc, attachment)
activate SRv6
SRv6->>SRv6: seg6local End.DT46 route
SRv6-->>CNI: ok
deactivate SRv6
CNI->>CNI: allocateArgument(ctx, k8s, namespace, routerName, vrfInstanceName) -> vrfID (local per-node Argument)
CNI->>CNI: registerEBPFDatapath(bgp, vpc, attachment, ifaceType, vrfID, PinDir)
activate EBPF
EBPF->>EBPF: Locator.Register / Function.Register / VRF.Register (locator_table, function_table, vrf_table)
EBPF-->>CNI: ok (fatal to ADD on error -- this is the only forwarding path)
deactivate EBPF
CNI->>K8s: CreateOrUpdate BGPVRFInstance
CNI->>K8s: CreateOrUpdate BGPAdvertisement(prefix, annotations)
CNI-->>Runtime: ok
Expand Down Expand Up @@ -175,7 +175,7 @@ sequenceDiagram
end
end

Note over CNI: Shared resources (VRF, interface, routes, SRv6,<br/>BGPAdvertisement, BGPVRFInstance) are NOT deleted here.<br/>They may be in use by another pod on the same (vpc, attachment).<br/>The GC controller collects orphans periodically.
Note over CNI: Shared resources (VRF, interface, routes,<br/>eBPF vrf_table entry, BGPAdvertisement, BGPVRFInstance) are NOT deleted here.<br/>They may be in use by another pod on the same (vpc, attachment).<br/>The GC controller (and, for vrf_table specifically, gc.SweepEBPFVRFTable) collects orphans periodically.

CNI->>CNI: slog.Info("DEL: skipping shared resource cleanup (handled by GC)")
CNI->>CNI: print empty result
Expand Down
57 changes: 54 additions & 3 deletions docs/cni/configuration.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,8 +4,11 @@
(or any CNI manager), plus node-local settings resolved at runtime from the
conflist, environment variables, and (as a last resort) the Kubernetes API.

> Last verified: 2026-07-28 against the current working tree of `internal/cni/config.go`,
> `internal/cni/ipam_ops.go`, and `internal/installer/installer.go`.
> Last verified: 2026-08-02 against the current working tree of `internal/cni/config.go`,
> `internal/cni/ipam_ops.go`, `internal/installer/installer.go`, `internal/config/cni.go`,
> and `internal/plumbing/ebpf/prog/usid.c` — the eBPF uSID datapath is now the only
> forwarding path (direct cutover, not a phased rollout); `GALACTIC_CNI_ENABLE_EBPF_DATAPATH`
> and `GALACTIC_CNI_EBPF_OBSERVE_ONLY` no longer exist.

## Runtime Configuration

Expand Down Expand Up @@ -84,6 +87,54 @@ and this environment variable has no effect on the allocation behavior.
**Type:** bool
**Default:** `false`

### eBPF uSID datapath

The eBPF/TC-BPF `uFMT 48+16` uSID datapath is the only forwarding path for
SRv6 uSID traffic — there is no legacy static-route fallback and no feature
flag to disable it. The DaemonSet's
long-lived `run` container (`internal/installer.Run`, via
`internal/plumbing/ebpf/attach`) always loads/pins/attaches the compiled
`usid_ingress` program at startup; a kernel preflight-check failure
(`internal/plumbing/ebpf/preflight`) is fatal to that container. The CNI
plugin binary's ADD path (`internal/cni/bgp.go`'s `registerEBPFDatapath`)
always registers this attachment's `vrf_table` entry; a registration
failure is fatal to the ADD.

**Argument allocation.** `registerEBPFDatapath` uses a real,
per-node-allocated 12-bit Argument value (`internal/cni/bgp.go`'s
`allocateArgument`) — the same value the router independently recomputes
the BGP-advertised SID from (`internal/reconcile`).

**Both `veth` and `tap` modes supported.** The datapath's final redirect
step (`internal/plumbing/ebpf/prog/usid.c`) picks a redirect helper per
`vrf_table` entry: `bpf_redirect_peer` for `veth` attachments (the
resolved egress interface's peer lives in the container's netns) or plain
`bpf_redirect` for `tap` attachments (`internal/cni/tap` never moves the
interface out of this netns, so there is no peer to cross into).
`registerEBPFDatapath` sets this per-entry from the CNI's own
`interface_type`, so no manual configuration is needed. The branch logic
itself is simple and verifier-accepted, but a real FIB-lookup-and-redirect
success/failure by egress kind requires a live route/interface (a real
net_device backing the packet) to observe — `BPF_PROG_TEST_RUN`, used for
this program's other unit tests, cannot simulate that without one, so this
specific behavior is verified in a live cluster (ContainerLab or e2e), not
by a kernel-level unit test.

| Variable | Description | Type | Default |
| ------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------ | --------------- |
| `GALACTIC_CNI_EBPF_INTERFACES` | Comma-separated list of interface names the eBPF datapath attaches its TC-BPF ingress hook to, overriding auto-detection (the interface(s) currently carrying the default IPv6 route). For multi-homed nodes where auto-detection is ambiguous. | string (comma-separated) | _(auto-detect)_ |

The `run` container also exposes Prometheus metrics (packets/bytes per
Argument, drops by reason, load/attach/detach event counts, and per-Block
Argument-space utilization — `internal/plumbing/ebpf/metrics`) at
`/metrics` on the port set by `galactic-cni run --metrics-port`
(default `9091`; alongside the existing `--grpc-health-port`, default
`5180`), regardless of whether the flag above is set — datapath-specific
series are simply absent/zero until it is. A separate gRPC health service
named `ebpf-datapath` (distinct from the always-serving `""` overall
service) reports the live result of `internal/plumbing/ebpf/attach.Health`
once the datapath has actually started.

## CNI Configuration JSON

The CNI configuration is a JSON object passed at pod creation time. It extends
Expand All @@ -93,7 +144,7 @@ the standard CNI `PluginConf` with Galactic-specific fields.

| Field | Required | Type | Description |
| ---------------- | -------- | --------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `vpc` | **Yes** | `string` | Base62-encoded VPC identifier (48-bit value). Used to derive VRF names, interface names, and BGP route targets. |
| `vpc` | **Yes** | `string` | Base62-encoded VPC identifier (16-bit value, cluster-scoped). Used to derive VRF names, interface names, and BGP route targets. |
| `vpcattachment` | **Yes** | `string` | Base62-encoded VPC attachment identifier (16-bit value). Paired with `vpc` for deterministic VRF/BGP naming. |
| `interface_type` | No | `string` | Interface mode: `"veth"` (default, for containers) or `"tap"` (for VMs such as Kata, Firecracker, QEMU). Both modes run IPAM and SRv6/BGP publish; `tap` mode only skips host-device delegation and guest-netns configuration (see the Tap mode section below). |
| `mtu` | No | `int` | MTU for the host-side interface. For `veth` mode this applies to both veth endpoints; for `tap` mode it applies to the tap interface. |
Expand Down
153 changes: 153 additions & 0 deletions docs/ebpf-datapath-sequence.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,153 @@
# eBPF uSID Datapath Sequence Diagrams

Sequence diagrams for the eBPF/TC-BPF `uFMT 48+16` uSID datapath, covering the
`run` container's startup/load/attach path and the CNI ADD path's map
registration. This is the only forwarding path — there is no legacy
static-route fallback and no feature flag to disable it (removed in the
2026-08-02 direct cutover; see
[docs/cni/configuration.md](cni/configuration.md#ebpf-usid-datapath)).

See [docs/cni-cmd-sequence.md](cni-cmd-sequence.md) for the pre-existing
`cmdAdd`/`cmdDel` diagrams this one supplements, not replaces.

## `run` container startup — datapath load, attach, health, GC sweep

```mermaid
sequenceDiagram
autonumber
participant Main as cmd/galactic-cni (run)
participant Installer as internal/installer.Run
participant Attach as plumbing/ebpf/attach
participant Preflight as plumbing/ebpf/preflight
participant Kernel
participant Metrics as plumbing/ebpf/metrics
participant GC as internal/gc.SweepEBPFVRFTable
participant K8s

Main->>Installer: Run(ctx, grpcHealthPort, metricsPort)
activate Installer
Installer->>Installer: startEBPFDatapath(ctx, m)
Installer->>Attach: SetHooks(m.Events.Hooks())
Installer->>Attach: StartWatching(ctx, PinDir)
activate Attach
Attach->>Preflight: Check()
activate Preflight
Preflight->>Kernel: probe SCHED_CLS, HASH maps, BTF, fib_lookup+tbid
Kernel-->>Preflight: capabilities present/absent
Preflight-->>Attach: nil, or an actionable aggregated error
deactivate Preflight
alt preflight failed
Attach-->>Installer: error
Installer-->>Main: fatal error (container crashes, CrashLoopBackOff -- no fallback path exists)
else preflight passed
Attach->>Kernel: load compiled usid_ingress + pin maps under PinDir
Attach->>Attach: ResolveInterfaces() (GALACTIC_CNI_EBPF_INTERFACES override or auto-detect default-route ifaces)
Attach->>Kernel: Attach TC-BPF ingress filter to resolved interfaces
Attach->>Attach: spawn Watch() goroutine (netlink link/route subscriptions)
Attach-->>Installer: *prog.UsidObjects, ifaces, nil
end
deactivate Attach
Installer->>Metrics: RegisterDatapathCollector(objs)
Installer->>K8s: newK8sClientFn() (best-effort, for the GC sweep below)
Installer->>Installer: loadHostConf(HostConflist) -> namespace, nodeName

Installer->>Installer: serve /metrics (metricsPort), gRPC health (grpcHealthPort)
Installer->>Installer: SetServingStatus("", SERVING) -- SetServingStatus("ebpf-datapath", SERVING)

loop every ebpfHealthCheckInterval (10s)
Installer->>Attach: Health(objs, ifaces)
Attach->>Kernel: confirm TC filter still attached + program/maps still reachable
Kernel-->>Attach: ok / error
Attach-->>Installer: nil / error
Installer->>Installer: SetServingStatus("ebpf-datapath", SERVING/NOT_SERVING)
end

loop every ebpfGCSweepInterval (5m)
Installer->>GC: SweepEBPFVRFTable(ctx, k8sClient, namespace, nodeName, PinDir)
activate GC
GC->>Kernel: VRF.Generation() (cutoff, captured before listing CRDs)
GC->>K8s: list BGPRouters (this node) + BGPVRFInstances
GC->>GC: derive live (Block, Argument) set via uformat.Block + inst.Spec.VRFID directly
GC->>Kernel: VRF.Reconcile(live, cutoff) -- deletes stale entries, keeps Generation>=cutoff
GC-->>Installer: CleanupResult{EBPFVRFEntriesRemoved, Errors}
deactivate GC
end

Note over Installer: ctx.Done() -> graceful shutdown -- deferred datapath.Close() releases this process's map/program fds (pinned maps persist for the next restart)
deactivate Installer
```

## CNI ADD — eBPF `vrf_table` registration

```mermaid
sequenceDiagram
autonumber
participant Runtime
participant CNI as internal/cni (cmdAdd)
participant BGP as internal/cni/bgp.go
participant USIDMap as plumbing/ebpf/usidmap
participant PinnedMaps as pinned vrf_table/locator_table/function_table

Runtime->>CNI: ADD
activate CNI
Note over CNI: VRF, veth/tap, IPAM as in docs/cni-cmd-sequence.md

CNI->>BGP: publishBGPStateK8s(...)
activate BGP
BGP->>BGP: lookupBGPRouter() -> srv6Locator, nodeID
BGP->>BGP: allocateArgument(ctx, k8s, namespace, routerName, vrfInstanceName) -> vrfID (12-bit Argument, local per-node allocation)
BGP->>BGP: egressKindForInterfaceType(pluginConf.InterfaceType) -> EgressKindVeth | EgressKindTap
BGP->>BGP: ComputeSID(srv6Locator, nodeID, vrfID, FunctionEndDT46) (for the router's independent BGP-advertised SID recomputation -- the CNI no longer installs a kernel route from it)

BGP->>BGP: registerEBPFDatapath(bgp, vpc, vpcAttachment, ifaceType, vrfID, attach.PinDir)
activate BGP
alt BGPRouter not configured (no srv6Locator/nodeID)
BGP-->>BGP: registered=false, nil (SRv6 intentionally not set up for this attachment)
else configured
BGP->>BGP: uformat.Block(netip.ParsePrefix(srv6Locator).Addr())
BGP->>USIDMap: OpenPinnedRegistry(PinDir)
USIDMap->>PinnedMaps: ebpf.LoadPinnedMap x3 (open, don't create)
PinnedMaps-->>USIDMap: map handles
USIDMap-->>BGP: Registry, closer
BGP->>USIDMap: Locator.Register(block, nodeID)
BGP->>USIDMap: Function.Register(block, FunctionEndDT46)
BGP->>USIDMap: VRF.Register(block, vrfID, vrf.TableID(vpc, vpcAttachment), egressKind)
USIDMap->>PinnedMaps: Put x3
BGP->>USIDMap: closer.Close() (this process's own fd only -- pinned maps persist)
BGP-->>BGP: registered=true, block, nil
end
deactivate BGP
BGP->>BGP: on error, return it -- fatal to the ADD (no fallback path exists)
Note over BGP: on registered=true, tracker.ebpfRegistered/ebpfBlock/ebpfArgument recorded for rollback (see below)
deactivate BGP
deactivate CNI
```

## Failed-ADD rollback — unregistering the eBPF entry

```mermaid
sequenceDiagram
autonumber
participant CNI as internal/cni (cmdAdd, failure path)
participant Tracker as resourceTracker.cleanup
participant BGP as internal/cni/bgp.go
participant USIDMap as plumbing/ebpf/usidmap

CNI->>Tracker: cleanup(ctx)
activate Tracker
Note over Tracker: reverse creation order
alt tracker.ebpfRegistered
Tracker->>BGP: unregisterEBPFDatapath(block, argument, attach.PinDir)
BGP->>USIDMap: OpenPinnedRegistry(PinDir)
BGP->>USIDMap: VRF.Unregister(block, argument)
Note over BGP: idempotent -- not an error if already absent
end
Note over Tracker: veth/tap delete, VRF delete follow, as in docs/cni-cmd-sequence.md
deactivate Tracker
```

Steady-state (non-failed-ADD) teardown of the `vrf_table` entry is
deliberately **not** part of `cmdDel` — matching this repo's existing
"DEL is intentionally minimal" design (`docs/agents/ARCHITECTURE.md`'s
Known Constraints) — it is instead the `run` container's periodic
`gc.SweepEBPFVRFTable` shown in the first diagram above.
Loading