Carnx

XDP-based network statistics digger.

What is XDP ?

XDP (for eXpress Data Path) is a Linux Kernel feature (>=4.8) providing an early hook in the incoming packets queue (RX).

The hook is placed in the network interface controller (NIC) driver just after the interrupt processing, and before any memory allocation needed by the network stack itself (Wikipedia). The following diagram (source) details the position of XDP in the incoming packet data flow.

The main assets of XDP are the following:

• XDP does not require memory allocation
• XDP hook code is run in the kernel cutting down its CPU usage

Due to this design, XDP is then rather competitive with kernel-bypass methods (like DPDK or PF_RING) but easier to integrate. In comparison with userspace methods, it can process roughly 5x more packets than classical tools (like iptables). You can have a look to the benchmarks in this repository. See the original paper and the corresponding presentation for a deeper description of XDP.

Architecture of carnx

The design of XDP constraints its use. In particular carnx is divided into several components to manage both the kernel hook and the incoming requests of the server.

Kernel hook

The program processing the incoming packets is carnx.bpf. Its sources are written in C and are then compiled to eBPF with clang (XDP hooks are run by the eBPF virtual machine). This code is very critical as it runs in kernel mode, therefore it is verified by the kernel when we want to load it. There are many constraints to pass the verifier: limited program size, no loop, buffer bounds must be checked before accesses... This program is implemented so as to update some counters from incoming packets.

As an example, if you have built the program, you can fetch the list of the counters through:

# grpcurl -plaintext -emit-defaults -unix /run/carnx.sock api.Carnx/GetCounterNames
{
  "counters": [
    "PKT",
    "IP",
    "IP6",
    "TCP",
    "UDP",
    "ICMP",
    "ICMP6",
    "ARP",
    "ACK",
    "SYN"
  ]
}

BPF Map

The kernel hook increments some counters but naturally we want to fetch these values to a user-space application (our server). For this purpose XDP can use all the BPF ecosystem (recall that XDP is merely a BPF program) which notably provides maps to share memory between the kernel and the user-space.

Several map types exist. Carnx use currently a single map (XDP_CARNX_MAP) storing counter values: counters are referenced by an index i and their value is merely XDP_CARNX_MAP[i]. So, our map behaves like an array.

Actually, there is not a single map but one for each CPU core. Why? In a Linux system, you have not a single RX queue but one for every core. Packets are well dispatched to the cores and are then processed in parallel (see this post for a more detailed view of the linux networking stack receiving data).

User-space interface

While the hook updates the map, the counter values are fetched from the kernel by the user-space library libcarnx.so. This library mainly uses libbpf.so to interact with the kernel objects. So it can read XDP_CARNX_MAP but it is also responsible of loading carnx.bpf into the kernel (and attaching the program to the desired network interface).

In addition, the library maintains a context (defined below) to track the load/attach operations.

struct context
{
    struct bpf_object *obj;
    int prog_fd;
    int map_fd;
    struct bpf_map *map;
    char iface[IFACE_LENGTH];
    unsigned int xdp_flags;
    bool is_loaded;
    bool is_attached;
};

Server

Finally a server written in Go (carnxd) exposes a gRPC API to manage load/attach operations and provide counter values. It basically wraps around libcarnx.so. By default carnxd listens to a unix socket to avoid polluting a network interface with API calls (recall that the XDP hook monitors a network interface).

Build

Currently you can only get carnx from sources. First you have to compile libbpf.so and then you can build both libcarnx.so and carnxd.

$ make libbpf
$ make

Then you can test carnx (as root) through

# tests/unix-test.sh 

In particular it check the API. If there is a problem, you will see it :)

Install

After the build, you can install everything with the following command. You should naturally check the Makefile to ensure it is not harmful :)

# make install

In details it does the following:

• The BPF library files (libbpf.so and libbpf.so.0) are installed to /usr/lib
• The carnx library (libcarnx.so) too
• The carnx server (carnxd) is installed to /usr/bin
• The systemd files (carnx.socket and carnx.service) are installed to /lib/systemd/system/

You can remove the installed files by calling

# make uninstall

Get started

After installing carnx, you can start the server by defining the path to the BPF program and the network interface to monitor:

# carnxd --interface lo --load /var/lib/carnx/carnx.bpf

The program cannot be put into background (daemonize). For this purpose, you should use the systemd service. By default it will listen to the localhost interface lo. Currently you must modify it by editing /lib/systemd/system/carnx.service.

To test the server, you can install grpcurl and request a snapshot from the server (the current values of the counters).

# grpcurl -plaintext -emit-defaults -unix /run/carnx.sock api.Carnx/Snapshot

API

The gRPC API is detailed in the api sub-directory.

What's next?

Many things can be done. I don't know precisely what I tend to do:

• Improve carnx (config file, non-root rights to the socket...)
• Build something upon carnx
• Package carnx
• Improve the doc