-
Notifications
You must be signed in to change notification settings - Fork 7
Architecture Decisions
This page records the reasoning behind the main data-plane choices in Auto XDP. It describes decisions reflected in the current implementation, including their trade-offs and limits. It is not a replacement for the architecture and packet flow or BPF maps and low-level API references.
Some of these decisions predate this page. The records below capture the current rationale and consequences; they do not claim to reproduce every original design discussion or commit date.
The current design follows four constraints:
- Keep packet admission in the kernel hot path. A packet should not need a synchronous userspace round trip.
- Use the shape of the key space when choosing a map. Port numbers have a fixed 16-bit range; connection and source state does not.
- Keep optional protocol logic replaceable. The base parser and admission policy should not need to be rebuilt for every application handler.
- Keep one policy model across backends, while documenting where XDP/eBPF provides capabilities that nftables cannot provide.
Status: Accepted
TCP, UDP, and SCTP port admission uses a port number as its key. The valid port range is fixed at 0–65535, and the lookup occurs for every packet that reaches the admission decision.
The project moved its port whitelist from a hash-based map to BPF_MAP_TYPE_ARRAY. tcp_whitelist, udp_whitelist, and sctp_whitelist now use 65,536 entries. The port number is the array index. A value of 1 admits the port; a value of 0 closes it.
Per-port totals, such as tsc_port, use the same direct-indexed shape where the key space is also the complete port range. The implementation keeps connection tuples and source-prefix state in hash-based maps because those keys are sparse and not bounded by the port range.
- The array gives a direct lookup for every possible port.
- The map has no hash-bucket or collision behavior in the admission lookup.
- A fixed map makes the update semantics simple: set a value to open or close a port. There is no delete operation for an absent port.
- The same model is easy to inspect and keeps the port-policy ABI stable across XDP and userspace.
A hash map would store only open ports and would be a natural choice for a sparse set. It was not selected for the hot admission path because the key domain is already small, fixed, and directly indexable. The fixed array costs more reserved map space, but removes the need to hash a port and gives a predictable lookup shape.
This would save kernel map space but would put a userspace operation on the packet path. It would not meet the latency and flood-resistance goals of the XDP backend.
- Port whitelist maps reserve the complete port range instead of growing with the number of open ports.
- Other state maps remain hash-based where their keys are tuples, prefixes, or sparse policy entries.
- Changing a whitelist map's type or value layout is an ABI migration. Update the BPF objects, userspace wrappers, required-map manifest, tests, and rollback validation together.
Status: Accepted
Per-source SYN and UDP limits need state keyed by source address, but the limits also need to remain isolated by destination port. A single flat map would combine every port and source into one larger key space.
The outer maps syn4, syn6, udprt4, and udprt6 are BPF_MAP_TYPE_ARRAY_OF_MAPS, indexed by destination port. Each occupied slot points to an LRU hash map for source state. Inner-map capacities can differ for IPv4 and IPv6 and are controlled by configuration.
- Port selection is direct, like the admission whitelist.
- Source state remains sparse and uses an LRU hash map where an address is the natural key.
- Each port gets an explicit state boundary and capacity instead of sharing one unbounded-looking pool.
- Userspace can create, validate, and replace the inner maps as part of a generation.
This design depends on map-in-map support, including the BPF_F_INNER_MAP behavior used by the current XDP backend. That is why the native XDP compatibility baseline is Linux 5.10+. Hosts that cannot provide this support use the nftables backend instead.
The outer map is large even when few ports have rate state. The benefit is predictable port selection and per-port isolation; the cost is more map setup and more ABI surface.
Status: Accepted
The main XDP program must parse packets and enforce common admission rules. Some protocols and applications need additional validation, such as GRE, ESP, SCTP, or a multi-packet application protocol. Embedding every optional validator in the main program would make the verifier workload and update path grow with every handler.
Auto XDP exposes BPF_PROG_TYPE_XDP program arrays for optional handlers:
-
proto_handlers, indexed by final IP protocol number 0–255. -
tcp_port_handlers, indexed by TCP destination port 1–65535. -
udp_port_handlers, indexed by UDP destination port 1–65535.
The main program prepares an xdp_slot_ctx with already-parsed fields, then dispatches to a selected handler with a BPF tail call. Handler programs can be loaded, validated, and swapped without rebuilding the main program. The parsed context is read through get_slot_ctx(ctx); native XDP uses packet metadata and generic XDP uses the per-CPU slot_ctx_map fallback.
A single program could contain every protocol branch. It would keep control flow in one object, but each new handler would require rebuilding and re-verifying the main program. It would also make optional application logic part of the base firewall's verifier and update surface.
Static branches avoid program-array lookups, but disabled and protocol-specific code would still be compiled into the main program. They do not provide the same load, unload, and per-port replacement model.
Userspace inspection is too late for the XDP decision path. nftables remains the compatibility backend, but it cannot execute these BPF handler programs or provide the same per-packet BPF handler state machines.
- Handler changes can be localized to a program-array slot and its persistent object.
- Handler programs must satisfy the XDP verifier and use the shared context ABI correctly.
- Tail-call handlers are XDP-only. A fallback to nftables preserves common firewall policy, not custom BPF handler behavior.
- A missing protocol handler follows
slots.default_action; operators must account for tunnels and VPN protocols before choosing a default drop action. - The handler loader uses a load, validate, swap, and rollback sequence so a failed replacement does not discard the active handler.
Status: Accepted
XDP sees ingress packets before the normal socket and conntrack path. It cannot infer from an inbound packet alone whether the packet is a reply to a connection initiated by the host. A simple inbound whitelist would either drop legitimate replies or allow too much unsolicited traffic.
tc_flow_track.o observes host egress and writes reverse TCP and UDP tuples into pinned conntrack maps. The XDP program reads the same maps during ingress admission. Install and reload also seed existing TCP sessions when possible.
- A userspace flow tracker would introduce scheduling latency and races between egress and ingress.
- A broad inbound allow rule would preserve replies but weaken the default-deny policy.
- Native kernel conntrack is available to the nftables backend, but it is not a map that the XDP program can directly consult in its early ingress path.
- XDP and
tcmust agree on tuple layouts, byte order, timestamps, and cleanup rules. - The
tcegress attachment is part of the XDP runtime's health and recovery path. - Conntrack maps are shared ABI objects. Changes require coordinated updates across the XDP program,
tcprogram, wrappers, and tests.
Status: Accepted
Native XDP is not available or safe on every virtual machine, driver, or kernel. The project still needs automatic listener synchronization on those hosts.
Userspace builds one desired policy and reconciles it through either the XDP backend or a dedicated nftables table. The nftables backend mirrors common admission, trust, ACL, and rate-limit semantics where practical. XDP-specific features remain explicitly XDP-only.
This would simplify the implementation but exclude hosts where XDP cannot attach or where the required map-in-map support is unavailable.
Maintaining two unrelated policy models would make listener changes, configuration, and security behavior drift between backends. A shared desired-state model keeps the operational contract closer across hosts.
- Native XDP provides early ingress filtering, BPF handler dispatch, and BPF ring-buffer telemetry.
- nftables provides compatibility but does not execute custom BPF handlers or expose the same packet-event stream.
- Backend capability differences must remain visible in documentation and diagnostics.
Status: Accepted
Reloading a firewall can fail during compilation, verifier checks, map validation, interface attachment, or service startup. A detach-first or in-place update can leave an interface with no protection or with only part of the intended policy.
Auto XDP prepares candidate programs, maps, handlers, and policy state before switching the active generation. It verifies map names and ABI requirements, seeds required state, switches attachments, and retains the previous generation for rollback. The nftables backend commits a complete candidate table in one batch.
- Mutating the active maps and program in place is simpler, but a partial failure can leave mixed generations.
- Detaching first creates a protection gap during reload.
- Rebuilding only the failed component makes recovery logic depend on which component failed and can leave cross-component state inconsistent.
- Install and reload code must manage pinned generations, active identity, and recovery metadata.
- Map ABI validation is part of deployment safety, not just a build check.
- The system uses more disk and pin-management logic to keep the previous generation available.
- The extra coordination makes upgrades safer across multiple interfaces and across XDP,
tc, handlers, and userspace state.
Auto XDP documentation · Repository · Releases · MPL-2.0