Skip to content

Diagnosing a Wedged Port

Jump to bottom
Julius Bairaktaris edited this page Jul 28, 2026 · 4 revisions

Diagnosing a Wedged Port

A wedged port carries no traffic while every status the system reports looks healthy: the cable is in, Link is Up - 1Gbps/Full is in the log, the carrier is 1, and nothing is logged when the traffic stops. WAN is the painful case — DNS stops resolving, pppd loops on Timeout waiting for PADO packets, and restarting the network changes nothing — but any port can do it.

This page is the runbook: what to look at, in what order, and what each reading means. It is deliberately biased toward capturing evidence before recovering, because most wedges are cleared by a reboot and a reboot destroys everything that would have explained them.

First: do not reboot

A reboot fixes almost every wedge and teaches you nothing. A wedged box is rare and worth more intact than working, so collect the readings below first — they take under a minute — and only then recover. If the box is someone's only internet connection, capture at minimum the one-shot snapshot before you reboot.

Nothing on this page changes the datapath; every command is read-only unless it says otherwise.

Is it actually a wedge?

A wedge means the port is up and passing nothing. Check that first, because a down link, a dead PPPoE session and an upstream outage all look like "no internet" and none of them are this.

ip -br link                       # carrier state per port
ethtool <port> | grep -i 'link detected\|speed'
logread | grep -iE 'link is (up|down)'

If the link is genuinely down, this page does not apply — that is a cable, PHY or negotiation problem. If the link is up and traffic still does not flow, continue.

One-shot snapshot

Run this and keep the output; it contains every discriminator used below.

{
  date; uptime
  cat /etc/openwrt_release
  ip -br link
  nss-status -l2
  nss-status -d
  nss-wedge-watch -1
  tc -s qdisc show dev <port>
  cat /sys/kernel/debug/qca-ppe-nss/status
  echo "--- port scheduler TDM ---"
  i=0; while [ $i -lt 50 ]; do devmem $((0x3a47a000 + i * 16)) 32; i=$((i + 1)); done
  dmesg | tail -100
  logread | tail -200
} > /tmp/wedge.txt 2>&1

The TDM dump is the switch's scheduler timetable. It is fixed at boot and never changes afterwards, so it does not describe the wedge — it identifies which scheduler behaviour your build has. Do not try to read it; just include it.

Take this capture before rebooting: a wedge is intermittent, and a reboot is the one action that destroys the evidence. If the datapath was dead long enough for the recovery service to fire, a fuller capture is already on flash in /root/auto-recovery/<timestamp>/ — take that too.

The four signatures

nss-status -l2 is the fastest way to localise a wedge. It prints one row per offloaded port:

  port   ifn  car  inj  vsi  vsi_bmp       host->fw  hostbusy   fw->host  wire->fw  fw->wire
  wan    2    1    1    5    0x05050505        2001         0       1909     35382     19946

Read it as a pipeline: the host hands frames to the firmware (host->fw), the firmware puts them on the wire (fw->wire), and the reverse for receive. Where the counters stop advancing tells you which stage failed.

1. car=1 but the qdisc tree has no leaf — the port transmits nothing

By far the most common cause, and the one that survives a network restart. Check the shaper tree:

tc qdisc show dev <port>

A healthy NSS tree always has a leaf below the root:

qdisc nsstbl 1: root ...          <- root shaper
qdisc nssprio 2: parent 1: ...    <- band split (fast lane builds)
qdisc nsspfifo 100: parent 2:1 ...
qdisc nssfq_codel 10: parent 2:2 ... set_default    <- the leaf that matters

If you see only the nsstbl root with nothing under it, that is the wedge. An NSS qdisc with no set_default leaf transmits zero packets — including ARP, so the port goes completely silent while every status stays green. This arises when SQM's shaper setup fails partway: the root is installed before the leaves, so a failed leaf leaves a half-built tree behind.

Recovery — remove the broken tree, then let SQM rebuild it:

tc qdisc del dev <port> root      # port starts passing traffic immediately
/etc/init.d/sqm restart           # rebuild the shaper properly

If the port comes back the moment the root is deleted, this was your cause. Check logread | grep -i sqm for the setup failure that produced it.

2. inj=0 — host traffic is dropped before the firmware

inj is the port's injection grant. While a port is mid-transition the conduit deliberately drops host frames, and the grant is restored once link-up completes. A port that is car=1 but stuck at inj=0 never got its grant back, so everything the router itself sends — ARP, DHCP, DNS, PPPoE discovery — is discarded before it reaches the firmware, while flows the firmware forwards on its own keep working.

grep 'if_num' /sys/kernel/debug/qca-ppe-nss/status

Recovery is a link transition, which re-runs the grant:

ip link set <port> down; sleep 2; ip link set <port> up

Transient inj=0 during the first seconds after boot is normal.

3. hostbusy climbing — the firmware stopped draining the host ring

hostbusy counts frames the firmware refused to accept. A steadily climbing value with host->fw flat means the host-to-firmware ring is full and not draining: the firmware is alive enough to refuse work but not to do it. This is a firmware-side stall; no host-side command clears it, and a reboot is the only known recovery. Capture everything before you do — this signature is the least understood of the four.

4. host->fw climbing but fw->wire flat — frames die past the firmware

The host is successfully handing frames over and they are not reaching the wire. Check the flood masks in the same row: vsi_bmp should be non-zero (each byte holds a port bitmap). An emptied vsi_bmp means the port was dropped from its own flood masks, which silently black-holes broadcast — ARP and PPPoE discovery included — while established unicast flows continue.

Also worth reading here, since the MAC counters come from hardware and do not lie the way /proc/net/dev does for offloaded flows:

ethtool -S <port> | grep -iE 'tx_(uni|broad|byte)|rx_(uni|broad|byte)'

Take two samples a few seconds apart. If the MAC transmit counters advance while the peer sees nothing, the problem is past the MAC (wire, PHY, link partner). If they stay frozen, the frames never left the switch fabric.

If the port still will not pass traffic

In escalating order, capturing between steps:

  1. Bounce the link — clears a stuck injection grant and re-syncs a lane that lost alignment: ip link set <port> down; sleep 2; ip link set <port> up
  2. Remove the shaper — clears a half-built qdisc tree: tc qdisc del dev <port> root
  3. Restart SQM — rebuilds the tree from configuration: /etc/init.d/sqm restart
  4. Reboot. Everything above failed, so the state lives in the firmware and only a fresh firmware boot clears it. Make sure you have the snapshot first.

Which step worked is itself the most valuable diagnostic: a port that returns on a link bounce failed differently from one that needs a full reboot. Note it.

Reporting a wedge

A wedge is intermittent, so a report is only actionable with state captured while the port is still dead. Five things:

  1. /tmp/wedge.txt from the snapshot above, taken before any recovery.
  2. Which recovery step worked — link bounce, shaper delete, SQM restart, or only a reboot.
  3. Was there a link event just before it died? Look in the dmesg part of the snapshot for a Link is Down or Link is Up on any port in the minute before. "No" is as useful as "yes" — they point at different faults, and this is the single most valuable line in a report.
  4. Which build — the tree and commit you built from. Different trees do not share this driver, so this decides which code you are running at all.
  5. What was running — uptime, the traffic at the time, plus uci get nss.general.enabled and uci get sqm.wan.enabled with your rates.

If you can only manage two, send 1 and 3.

Related

Clone this wiki locally