AF_XDP implementation per ADR-002 (3 commits)#19
Merged
Conversation
Replaces the recvfrom/sendto scaffold in `crates/lb-io/src/af_xdp.rs`
with a real XSK-ring implementation built on `xdpilone` 1.2 (chosen in
ADR-002). Linux + `--features af-xdp` gets a working backend; non-Linux
or feature-disabled keeps returning `Unsupported` so dev builds stay
quiet.
Architecture:
* UMEM is a page-aligned heap allocation (default 8 MiB = 4096 frames ×
2048 B), allocated via `std::alloc::alloc(layout)` because
`Box::new_uninit_slice` doesn't honour custom alignment.
* Frame pool split: half the frames are pre-filled into FILL at init
(the "RX pool"), half are held in `free_frames` for outbound TX. The
partition is strict — kernel-recycled frames go back to their own
side. This prevents the two paths from starving each other under
load.
* `recv_batch` drains RX → copies into caller `PacketBuf`s → recycles
exact frames back to FILL within the same call.
* `send_batch` drains COMPLETION first to reclaim TX frames, pops a
free frame per packet, copies into UMEM, posts on TX, then `wake()`s
the kernel iff `XDP_USE_NEED_WAKEUP` was reported. Returns `Ok(0)` on
backpressure rather than erroring — caller can retry.
Drop-order is load-bearing: rings → umem → backing alloc. Field
declaration order in `AfXdpIo` enforces this; reordering it is a
use-after-free.
`unsafe impl Send for AfXdpIo` is sound because the only `!Send` field
is xdpilone's `NonNull<[u8]>` pointing at memory we own; the
hand-off in `MultiThreadedForwarder::start` is single-owner.
Out of scope (commits 2 + 3):
* Loading the XDP redirect program from lb-node itself
* Refactoring MultiThreadedForwarder to take a single bidirectional
PacketIo (currently binds the same queue twice for AF_XDP)
* veth-pair integration test
Verified:
cargo check --workspace --tests (macOS, default features)
cargo check -p lb-io --features af-xdp \
--target aarch64-unknown-linux-gnu
cargo clippy --workspace --all-targets -- -D warnings
cargo test --workspace
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The forwarder needs a kernel-side BPF program that redirects packets
into the AF_XDP socket. xdpilone (the userspace XSK wrapper) doesn't
load BPF programs; aya/libbpf-rs would each be ~30+ kLOC of dep just
for that one task. Pragmatic split:
* **Bundle the BPF program**: `deploy/xdp/redirect.bpf.c` is the
minimal XDP_REDIRECT-to-XSKMAP program (12 LOC of C). Built with
`make -C deploy/xdp` (clang -target bpf), output is gitignored.
* **Bundle the deploy tooling**: `load.sh` attaches the program and
pins the XSKMAP at `/sys/fs/bpf/lb/xsks_map`; `unload.sh` reverses
everything cleanly. Run once per host as part of provisioning.
* **Self-register from `lb-node`**: at AF_XDP socket bind time,
`AfXdpIo::new` opens the pinned XSKMAP and inserts its own XSK fd
at index `queue_id`. No manual `bpftool map update` needed in
steady state. The bpf(2) syscall is called directly via libc;
only two cmd codes (BPF_OBJ_GET, BPF_MAP_UPDATE_ELEM) are needed,
so we avoid pulling libbpf-sys back in.
Operator-facing flow:
sudo apt install clang libbpf-dev bpftool
make -C deploy/xdp
sudo deploy/xdp/load.sh eth0 native
sudo systemctl start lb-node # self-registers
`xskmap_pin = None` in `AfXdpConfig` opts out of self-registration —
useful for early bring-up, custom XDP setups, or operators who prefer
to drive the map by hand.
The XDP program intentionally does no L2/L3 filtering — that belongs
in userspace `VipMatcher` where the full configured-VIP set lives.
Doing it kernel-side would mean recompiling and reloading on every
config change, a much bigger blast radius.
Verified:
cargo check -p lb-io --features af-xdp \
--target aarch64-unknown-linux-gnu
cargo clippy -p lb-io --features af-xdp \
--target aarch64-unknown-linux-gnu -- -D warnings
cargo test --workspace (all suites still pass)
bash -n deploy/xdp/{load,unload}.sh
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Wires the AfXdpIo from commits 1+2 into a proper end-to-end production
path. Three things land together:
* **Per-queue forwarder model** in `lb-forwarder`:
`MultiThreadedForwarder::start_per_queue(ios, config, shared)` spawns
one rewriter thread per `PacketIo`, each owning its IO handle and
running `recv_batch → process_batch → send_batch` inline. No
userspace steering, no muxer, no PacketPool, no inter-thread queues.
Mirrors how AF_XDP actually wants to be driven — the kernel's RSS
already partitions traffic across kernel queues, and binding the
same (iface, queue) socket from two threads is `EBUSY`. The original
steering+muxer `start()` stays for `MockIo` (used by tests).
* **lb-node main wiring**: switched to `start_per_queue` for
`IoBackend::AfXdp`. Each rewriter binds queue `i` (so rewriter 0 →
queue 0, rewriter 1 → queue 1, …). The mock backend keeps the
steering+muxer model. A failed `AfXdpIo::new` now points at
`deploy/xdp/load.sh` in the error message.
* **Smoke test + ops docs**: `deploy/xdp/smoke-test.sh` sets up a
netns + veth pair, attaches the redirect program, runs `lb-node`,
injects 1000 packets via scapy, and asserts the loss rate stays
below 10%. Manual to run (needs sudo + Linux + scapy); not in CI.
`.docs/operations.md` thread-layout section now documents both
models, and the `io_backend` config row points at the XDP setup
procedure.
After this commit: `cargo build --release --features af-xdp` produces a
binary that, given a Linux host with the redirect program loaded, will
forward real traffic. AF_XDP is the production-ready path.
Verified:
cargo check --workspace --tests (macOS, default features)
cargo check -p lb-io --features af-xdp \
--target aarch64-unknown-linux-gnu
cargo clippy --workspace --all-targets -- -D warnings
cargo test --workspace (all 23 suites pass)
bash -n deploy/xdp/smoke-test.sh
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Lands the AF_XDP backend per ADR-002. Three sequential commits, each compiles and tests cleanly:
Commit 1 — `lb-io`: xdpilone-backed AfXdpIo (`2579ed0`)
Replaces the recvfrom/sendto scaffold with real XSK-ring plumbing via `xdpilone` 1.2:
Commit 2 — XDP redirect program + self-XSKMAP-register (`e3cc115`)
Commit 3 — Per-queue forwarder + smoke test (`631dcf7`)
Net result
`cargo build --release --features af-xdp` produces a binary that — on a Linux host with the redirect program loaded — forwards real traffic at AF_XDP speeds. This unblocks the first production deploy.
Test plan
Out of scope (follow-ups)
🤖 Generated with Claude Code