Skip to content

Architecture

Jump to bottom
Julius Bairaktaris edited this page Jun 18, 2026 · 3 revisions

Architecture

How the NSS firmware data plane and OpenWrt main's upstream ethernet drivers share one SoC. Everything in this page was established by measurement on an IPQ8071A (AX3600) running NSS.FW.12.5-210-HK.R; register-level claims come from live register dumps, not documentation.

The classic stack vs. this stack

The vendor NSS stack replaces the ethernet driver: qca-nss-dp registers the GMAC netdevs and hands the data plane to the NSS cores, qca-ssdk programs the PPE switch, and nss-bridge-mgr/vlan-mgr mirror Linux bridge state into hardware tables.

This stack keeps OpenWrt main's upstream drivers from PR #22381:

  • qca_edma — the ethernet driver for the shared EDMA DMA engine,
  • qca_ppe — the PPE switch driver (DSA, out-of-band tagging),

and inserts one small glue module, kmod-qca-ppe-nss, that impersonates the nss-dp API towards qca-nss-drv. The vendor driver (qca-nss-drv) loads unmodified apart from packaging-level patches.

            ┌────────────────────┐      ┌──────────────────┐
            │     qca-nss-drv    │◄────►│  NSS cores (fw)  │
            │  (vendor, feeds)   │ h2n/ │  UBI32 @0x39/38M │
            └───────┬────────────┘ n2h  └────────┬─────────┘
              nss-dp API (6 fns)                 │ owns PPE queue
            ┌───────┴────────────┐               │ delivery + own
            │   kmod-qca-ppe-nss │               │ EDMA rings
            │   (glue, this repo)│               ▼
            └──┬──────────────┬──┘      ┌──────────────────┐
       TX redirect      VSI getters     │  PPE @0x3a000000 │
            ┌──▼─────┐  ┌─────▼────┐    │  EDMA @0x3ab00000│
            │qca_edma│  │ qca_ppe  │───►│  (shared block)  │
            │(usptream PR #22381)  │    └──────────────────┘
            └────────┴────────────┘

Fact 1: the EDMA block is shared, and the firmware takes the RX plane

The IPQ807x EDMA engine is one block shared between the host driver and the NSS firmware, with a ring-partition convention baked into the firmware. Measured behavior when the firmware boots:

  • It remaps the PPE QID2RID (queue → ring) delivery tables. Measured mapping after firmware boot: queues 0–3 → fw ring 0, 4–7 → fw ring 3, 8–11 → fw ring 1, 12–127 → fw ring 0, 128–143 → fw ring 2. The host's RX ring is unmapped from all queues: every wired RX packet now lands in firmware-owned rings.
  • It enables its own rings (rxdesc 0–3, rxfill 0/2, txdesc 0–7 plus flow-control groups) but never touches host ring registers or EDMA globals. Host rings stay armed-idle; no host-side EDMA reprogramming is needed on attach.
  • The firmware requires an operating host EDMA to complete its own boot: with qca_edma unbound, the firmware never finishes initialization.

Consequences:

  • There is no mixed mode. With firmware up, a physical port is either attached to the firmware data plane or it has no RX. All ports must attach.
  • Unloading qca-nss-drv does not restore the RX plane. The QID2RID takeover persists after rmmod; wired RX stays dead until reboot. Reboot fully restores the host stack.

Fact 2: host driver hardening the shared block requires

Three qca_edma behaviors were fatal or wrong once firmware shares the block (commit "make shared-EDMA register usage NSS-firmware safe"); the first is reachable host-only as well:

  1. Misc-IRQ storm (the big one). Probe left EDMA_REG_MISC_INT_MASK = 0x1ff with a handler that reads MISC_INT_STAT but never acks or masks. Any sticky misc condition becomes an interrupt storm that wedges the SoC with no oops — the firmware's RX-ring bring-up trips one deterministically (this masqueraded as a "NoC deadlock" for a full debugging day). The legacy driver ran with mask 0 once a port opened. Fix: mask = 0 at init plus a self-disarming handler.
  2. TX interrupt mask loop overrun. edma_irq_disable_all() iterated TX_INT_MASK up to txdesc 23, but IPQ807x has 8 TX interrupt slots — the loop aliased into RXFILL_PROD_IDX/ RXFILL_INT_MASK registers. Bounded to the txcmpl range.
  3. Too-broad hardware stop. edma_hw_stop() swept all rings and cleared PORT_CTRL, killing firmware state (the firmware needs the global enable). Now host-rings-only.

The glue additionally enables clocks the legacy stack enabled unconditionally and nothing on the upstream stack does: nssnoc crypto, crypto_ppe, uniphy1/2. The firmware touches these blocks at runtime (core 1 boots crypto features; PPE init may iterate all six ports), and an access through a gated clock is a silent NoC stall.

Address map (no overlap, verified): PPE 0x3a000000–0x3a900000, EDMA 0x3ab00000, NSS cores 0x39xxxxxx and 0x38000000.

Fact 3: the nss-dp surface is six functions

qca-nss-drv's entire dependency on the ethernet driver is six functions, all consumed in nss_data_plane/nss_data_plane.c:

nss_dp_get_netdev_by_nss_if_num   nss_dp_is_in_open_state
nss_dp_override_data_plane        nss_dp_start_data_plane
nss_dp_restore_data_plane         nss_dp_receive   (N2H RX injection)

The glue provides all six; a staged nss_dp_api_if.h in the glue's src/exports/ defines the ABI both sides compile against. This is what makes the whole approach tractable: no qca-nss-dp, no qca-ssdk, one small module.

Port attach lifecycle

Arming happens strictly at runtime (see Runtime Operation):

  1. Glue loads; a port bitmask is written to debugfs qca-ppe-nss/fw_mask. For each armed PPE port the glue resolves the DSA user netdev and the conduit.
  2. qca-nss-drv loads and boots the firmware; its one-shot registration calls nss_dp_get_netdev_by_nss_if_num() and overrides the data plane of each armed port.
  3. nss_dp_start_data_plane() replays the bring-up sequence the old nss-dp performed, in order: vsi_assign → MAC address → MTU → open(0,0,0) → link state. The VSI used is the port's private firmware VSI (see below). A port is refused if no VSI can be resolved — a port started without a VSI gets TX but no RX (firmware delivers ingress only for VSI-assigned ports; measured).
  4. TX: a per-port RCU-protected redirect hook in qca_edma's ndo_xmit (at out-of-band port resolution) hands skbs to the glue, which submits them to the firmware. An identity mode (debugfs identity_mask) exercises the same plumbing while returning frames to the host EDMA path — useful for plumbing tests.
  5. RX: firmware delivers via nss_dp_receive(); the glue runs eth_type_trans(), updates tstats, sets offload_fwd_mark the same way the OOB tagger does, and hands the skb to napi_gro_receive().
  6. A netdev notifier tracks UP / DOWN / CHANGE / CHANGEMTU / CHANGEADDR / CHANGEUPPER / UNREGISTER. Restore is host-side-only cleanup, because nss_hal_remove tears down IRQs before calling back. Lock order is rtnl → ppe_nss_lock, everywhere (the reverse order deadlocks against the notifier path).

Per-port counters and state are exposed in debugfs (qca-ppe-nss/status): tx_redirect_pkts, rx_fw_pkts, tx_busy, rx_unexpected.

Fact 4: the firmware VSI model

VSIs (Virtual Switch Instances) are the PPE's L2 domains. Three firmware behaviors shape the design (all measured live):

  1. One port per VSI. The firmware NACKs (within milliseconds) a vsi_assign that would put a second physical port on a VSI. The shared br-lan bridge VSI therefore cannot be used for member ports — with it, only the first member ever attached. The legacy stack never hit this because qca-ssdk gave every port a unique default VSI, and bridge-VSI sharing was nss-bridge-mgr's job through the firmware's own bridge interface (not implemented here; see Limitations and Roadmap). → qca_ppe allocates a private VSI per user port on first use (qca_ppe_port_fw_vsi_get()).
  2. The firmware zeroes the member/flood masks of any VSI it is assigned. Effect: firmware-to-wire broadcast egress floods into an empty mask — host ARP requests silently never reach the wire, while unicast FDB-hit egress keeps working (a wonderfully confusing failure: ssh alive, ARP dead). → qca_ppe_port_fw_vsi_refresh() re-asserts {port, CPU} membership and flood masks after every vsi_assign.
  3. The firmware clears the hardware L3_VP VSI-valid bit when it releases a port (vsi_unassign at detach/link-down). A getter that reads only hardware therefore "forgets" never-bridged ports after their first detach. → qca_ppe_port_vsi_get() answers from the driver's own port_vsi[] intent cache first (hardware is only a fallback), and standalone ports are kept on the default VSI in that cache.

Bridging between ports currently happens in software (the Linux bridge), which soaks fine at these speeds; ECM still accelerates routed/NATed flows. Hardware bridge offload would need the bridge-manager redesign described in Limitations and Roadmap.

ECM (connection offload) on this stack

qca-nss-ecm needs no ssdk and no nss-dp (audited: zero references). It talks to qca-nss-drv only. Specifics:

  • The NSS front end is selected at runtime: front_end_selection=1 module parameter.
  • VLAN offload needs no vlan-mgr: ECM's unicast rules embed the VLAN tag in nircm->vlan_primary_rule using the physical port's interface number, so PPPoE-over-VLAN uplinks offload with the plain stack (verified end to end with a tagged PPPoE WAN).
  • ppe_vp is irrelevant: nss_ppe_vp.c is the only ssdk consumer inside qca-nss-drv and ECM does not use ppe_vp, so the driver is built with PPE-VP support disabled.
  • Conntrack event processing requires CONFIG_NF_CONNTRACK_EVENTS=y (set via the ECM package KCONFIG) and the net.netfilter.nf_conntrack_events=1 sysctl at runtime.

Interface numbering on IPQ807x: NSS physical interface numbers are 1..6 (START_IFNUM = 1, 6 ports max), and on DSA they coincide with the PPE port index and the OOB tag port number. The NSS preheader is 32 bytes.

ath11k Wi-Fi offload (wifili) on this stack

The Wi-Fi offload does not touch the EDMA/PPE port plumbing at all: ath11k registers its own dynamic interfaces (wifili SoC + per-vif vdevs) with qca-nss-drv, and the firmware then drives the WLAN DP rings directly. The pieces:

  • mac80211/ath11k patch sets (patches/nss/{subsys,ath11k} in the mac80211 package, applied only with CONFIG_ATH11K_NSS_SUPPORT) add SUPPORTS_NSS_OFFLOAD and the wifili interface. frame_mode=2 (ethernet decap) is the operating mode.
  • The probe gate. With the offload compiled in, ath11k.ko and mac80211.ko reference qca-nss-drv symbols, so the NSS driver loads at every boot. Its platform probe calls the glue's nss_dp_probe_gate() and defers until a port is armed — loading modules can never boot the firmware (which would kill all wired RX, see Fact 1). Arming re-attaches the deferred core devices and boots the firmware synchronously.
  • Two-phase Wi-Fi. ath11k's NSS setup runs during the radio probe and hard-fails if the firmware is not up, so radios start in host mode (nss_offload=0) and are re-probed (platform unbind/bind) with nss_offload=1 once the firmware data plane is armed and booted. Host-mode Wi-Fi is therefore always available as the recovery path.
  • ECM accelerates routed Wi-Fi client flows through the wifili vdevs with no extra manager; bridge-level (same-LAN) Wi-Fi↔wire traffic stays on the host bridge (bridge offload is deferred, see Limitations and Roadmap).
  • NSS mesh offload is not wired up (needs fw 11.4; this stack ships 12.5).

Clone this wiki locally