1.0.2

BNG Blaster 1.0.2

This release improves the AF_XDP performance and robustness, allows to run many BNG Blaster instances in parallel on one host, and fixes the LACP negotiation.

New Features

• Startup status in test-info: The control socket is now opened before the configuration is loaded, and test-info reports the status (starting, running, stopping), the current task (e.g. wait-startup-lock, init-interfaces, running, teardown, cleanup) and the pid. This allows a controller to follow the startup and shutdown of an instance.
• New threads command: Shows the main thread and all I/O threads with the CPU they are pinned to, whether the CPU was selected automatically or is shared with another thread, and the Linux thread ID.
• Abort the startup: The startup can be aborted with test-stop, terminate or the signals SIGINT, SIGTERM and SIGHUP, also while waiting for other instances. The interface configuration is restored and BNG Blaster exits with status 3.
• AF_XDP mode: The new link option af-xdp-mode (auto, zero-copy, copy, generic) selects the AF_XDP socket mode explicitly. Copy mode helps if the zero-copy mode of a driver performs badly, e.g. the Intel ice driver with an IOMMU in translation mode.
• L2TP connect speeds: The new l2tp-client (LAC) options tx-connect-speed and rx-connect-speed set the TX (AVP 24) and RX (AVP 38) Connect Speed sent in ICCN (#407).

Improvements

• RX and TX performance: RX and TX threads prefetch packets and stream state, so that the cache misses of multiple packets overlap. With AF_XDP on an Intel X710 (PPPoE QinQ), a single RX thread now receives 10G line rate (9.7 Mpps) with up to 1M flows instead of 4.2 to 6.6 Mpps, and a single TX thread sends 8.0 Mpps with 1M flows instead of 6.5 Mpps.
• Automatic CPU placement: The automatic CPU placement (rx-auto-cpuset, tx-auto-cpuset) now avoids CPUs handling the interrupts of the used NICs, uses only CPUs allowed for the process (taskset, systemd AllowedCPUs, container cpusets) and pins the main thread to a free CPU as well. A warning is logged for manually pinned threads running on CPUs handling NIC interrupts.
• Parallel instances: Instances running in parallel (e.g. one per SR-IOV virtual function) no longer select the same CPUs. Each instance locks its CPUs with /run/lock/bngblaster_cpu<N>.<PID>.lock, and CPUs are shared evenly if no CPU is free. Instances started at the same time are initialized one after another (/run/lock/bngblaster_startup.lock), which is logged while waiting.
• Lock files: Interface and CPU lock files use kernel locks (flock), which are released if an instance is killed or crashes. Stale lock files no longer block further tests.
• AF_XDP NIC configuration: Channels, RSS indirection table, NIC RX ring and RX VLAN offload are restored on exit, also if the startup fails. The NIC RX ring is increased to 4096 descriptors (a larger ring is kept), and the RX frames (io-slots-rx) are sized automatically to at least twice the NIC RX ring, unless configured explicitly. Based on the work of Andreas Zimber in #408.
• AF_XDP diagnostics: An interface MTU larger than the AF_XDP frame size is now accepted with a warning. A warning is logged if an XDP program is already attached. If the driver supports AF_XDP on fewer queues than required by the threads (e.g. i40e only on as many queues as CPUs), this is reported clearly.
• LACP: The LACP negotiation follows IEEE 802.1AX. A member collects once the partner is in sync and distributes once the partner collects as well, and changes of the own state are sent immediately. Standby members (lacp-max-active-links) are not in sync, so that the partner does not use them either. Two BNG Blaster instances can now be connected back to back with LACP.

Bug Fixes

• AF_XDP: Fixed RX stalling forever if the NIC RX ring ran empty, which happened e.g. with the Intel ice driver (E810) and an IOMMU in translation mode. The driver is now woken up also if nothing was received.
• AF_XDP: Fixed one of two instances failing if started at the same time with libbpf before 1.0. The failed instance also left an XDP program attached, which blocked all further tests on this interface. BNG Blaster now detaches an XDP program left behind on exit.
• AF_XDP: Fixed the XDP program remaining attached after exit with libxdp.
• AF_XDP: Fixed promiscuous mode being lost on SR-IOV VFs (e.g. iavf) after changing the number of queues.
• I/O threads: Fixed I/O threads whose CPU affinity could not be set, which kept running on any CPU while further I/O threads were not started at all.
• L2TP: The LNS showed connect speeds above 2^31 bps as negative numbers.
• JSON schema: rx-cpuset and tx-cpuset now accept a single CPU, as the configuration does.

Behavior Changes

• Control socket: The control socket is opened before the startup. During the startup, only test-info, commands, threads, test-stop and terminate are available, all other commands are answered with status code 503. Controllers should wait for the test-info status running instead of the control socket file.
• Startup: Instances started at the same time start one after another, and an aborted startup exits with status 3.
• AF_XDP: The NIC RX ring is changed during the test and more RX frames are used by default (32 MB per RX queue for a ring of 4096). Configure io-slots-rx explicitly to keep the previous size.
• LACP: BNG Blaster signals collecting once the partner is in sync, and standby members are no longer in sync.

Full Changelog: 1.0.1...1.0.2