NSS Offload Explained

NSS offload explained

This page explains, from the ground up, what NSS hardware offload is and how it works, before the rest of the wiki gets into how this project implements it. No prior NSS knowledge is assumed. If you already know what ECM and the PPE are, skip to Architecture.

The problem: a CPU forwarding every packet

A router's job is to move packets between interfaces — NAT a LAN host out to the internet, route IPv6, bridge two LAN ports, shape an uplink. On a plain Linux router the CPU does all of it. Every packet traverses the network stack: the driver hands it up, the bridge or routing table picks an output, netfilter runs conntrack and NAT, a qdisc schedules it, and the driver sends it out.

That is a lot of work per packet, and there are a lot of packets — a 300 Mbit/s line is on the order of 25,000–300,000 packets per second depending on size. On a small ARM SoC the CPU saturates well before the line does. Two things suffer:

• Throughput is capped by how fast the CPU can push packets, not by the link.
• Latency under load (bufferbloat) climbs, because packets queue in front of a busy CPU. This is what makes a video call stutter while someone else is downloading.

The Cortex-A53 cores in an IPQ807x can route maybe a few hundred Mbit/s of NATed traffic before softirq eats a whole core. That is the wall offload removes.

The idea: learn a flow once, forward it in hardware

Almost all router traffic is flows: long-lived conversations (a download, a video stream, an ssh session) that look identical packet to packet — same addresses, same ports, same output decision. The CPU re-derives that decision for every single packet, which is wasteful: the answer only changed on the first packet.

Flow offload exploits this. The first packet of a flow goes up the normal path; software works out what to do with it (route, NAT translation, output port, VLAN tag) and programs that decision into a dedicated hardware forwarding engine as a rule. Every subsequent packet of that flow is matched and forwarded by the hardware, never touching the CPU. The CPU now sees only first-packets and traffic that genuinely cannot be offloaded (new connections, broadcast, locally-destined packets).

The Linux kernel has a generic version of this idea (the nftables flowtable software/hardware fast path). NSS is Qualcomm's much more capable, dedicated implementation built into the IPQ807x silicon.

What NSS is

NSS — Network Sub-System — is a block of dedicated networking hardware in Qualcomm IPQ807x SoCs, separate from the ARM application cores. It has three parts that matter here:

• The PPE (Packet Processing Engine) — a hardware switch and classifier. Given a table of flow rules, it does the actual line-rate forwarding, NAT rewriting, VLAN handling, L2 bridging and traffic shaping itself. This is where the offloaded packets flow.
• Two UBI32 packet-processor cores running NSS firmware. The firmware manages the PPE, owns the DMA rings that move packets, runs the shaper, and speaks a message protocol to the host. The firmware is a proprietary Qualcomm blob (this project uses NSS.FW.12.5-210-HK.R; see Firmware and Source Pins).
• The EDMA engine — the DMA block that moves packets between the SoC and the ethernet ports. It is shared between the host CPU and the NSS firmware, which is the central complication this project had to solve (see Architecture).

NSS on its own is inert silicon. Making it useful needs software on the host to (a) drive the firmware and (b) decide which flows to offload.

The software pieces

flowchart TB
    ct["Linux conntrack / bridge / routing"]
    ecm["ECM (connection mgr)<br/>host kernel modules"]
    drv["qca-nss-drv<br/>host↔firmware messaging"]
    fw["NSS firmware (UBI32)<br/>drives the PPE + EDMA"]
    hw["PPE + EDMA hardware<br/>forwards the packets"]

    ct -->|"watches flows"| ecm
    ecm -->|"accelerate this flow:<br/>src/dst, NAT, port, VLAN"| fw
    drv <-->|"h2n / n2h"| fw
    fw --> hw

Loading

• qca-nss-drv — the Linux driver for the NSS cores. It boots the firmware, sets up the host↔firmware message rings (h2n = host-to-NSS, n2h = NSS-to-host), and exposes an API the rest of the stack uses. It does not itself decide what to offload.
• ECM (Enhanced Connection Manager) — the brain of the offload. ECM hooks into Linux conntrack, the bridge, and the routing path. When it sees an established connection that can be accelerated, it builds a hardware rule (addresses, NAT translation, output interface, VLAN tag, DSCP) and sends it to the firmware via qca-nss-drv. When the connection ends or changes, ECM tears the rule down. ECM is what turns "the hardware can forward flows" into "the router's actual NAT/routing/PPPoE traffic is forwarded in hardware."
• The connection managers — small helpers ECM and the firmware need for specific encapsulations: qca-nss-pppoe (PPPoE sessions), nss-bridge-mgr (Linux-bridge L2 offload), qca-mcs (IGMP/MLD multicast snooping). Each is loaded only when that feature is wanted.
• The NSS qdiscs (kmod-qca-nss-drv-qdisc/-igs) — let Linux tc/SQM program the firmware's hardware shaper, so traffic shaping (SQM) is offloaded too. See SQM and Shaping.

The lifecycle of an offloaded packet

1. A new connection's first packet arrives. The PPE has no rule for it, so it is delivered up to the host CPU (the slow path).
2. Linux processes it normally — conntrack records it, NAT/routing decide the output. ECM is watching.
3. Once the connection is established, ECM pushes a rule to the firmware describing how to forward this flow.
4. Subsequent packets of that flow are matched by the PPE and forwarded in hardware — NAT-translated, VLAN-tagged, shaped, sent out — without the CPU (the fast path). On an accelerated bulk download the host CPU stays essentially idle.
5. Traffic that cannot be offloaded — ARP/ND, broadcast/unknown-multicast flooding, fragments, locally-terminated packets, the first packet of every new flow — always takes the slow path. That residual is expected, not a fault.

Why "offload" is opt-in here, and what reboot does

On this project the router boots as plain OpenWrt: at the moment the network comes up the firmware is not running and the CPU forwards everything, exactly like stock. The NSS data plane is armed a moment later by the nss service, which runs the nss-up script at the end of boot (see Runtime Operation); once it is up, established flows ride the hardware. No NSS kernel module ever autoloads, and arming is never in the boot critical path.

Because every boot comes up on the plain host stack before NSS is armed, a reboot is always a safe way back to a clean, fully-working stock router — the universal recovery path if anything about the offload misbehaves. To keep the device on the host stack for good, disable auto-arming with uci set nss.general.enabled='0'.

How this stack differs from "normal" NSS builds

Every other NSS build replaces OpenWrt's ethernet driver with Qualcomm's out-of-tree qca-nss-dp driver and the qca-ssdk switch library. This project instead keeps OpenWrt's own upstream qca_edma/qca_ppe drivers (from openwrt/openwrt#22381) and attaches the NSS firmware to them through one small glue module, kmod-qca-ppe-nss. The payoff is staying on the mainline driver — no vendor ethernet stack to carry — while still getting full NSS offload. How that is done, and the hardware facts that make it tricky (the shared EDMA engine, the firmware's takeover of packet delivery), are the subject of Architecture.

Where to go next

• Architecture — how the firmware data plane and the upstream drivers share one SoC; the glue module and the firmware behaviors that shape the design.
• Runtime Operation — bringing the data plane up, and the rules that keep a remote device reachable.
• Firmware and Source Pins — which firmware and host sources are used and why.