w1700k
W1700K HW2.1: routed IPv6-PD (/60+) flows blackhole with nft_flow_offload/PPE #18
-
|
I think I found a potential reproducible IPv6 hardware flow offload issue Device / build:
With hardware flow offload enabled: uci set firewall.@defaults[0].flow_offloading='1'
uci set firewall.@defaults[0].flow_offloading_hw='1'
uci commit firewall
/etc/init.d/firewall restartIPv4 works fine, but forwarded IPv6 TCP traffic almost dies. Typical results from the wired client: Example IPv6 results with HW offload enabled: With hardware flow offload disabled: uci set firewall.@defaults[0].flow_offloading='1'
uci set firewall.@defaults[0].flow_offloading_hw='0'
uci commit firewall
/etc/init.d/firewall restartIPv6 becomes stable again: This is not Wi-Fi related, since the test client is wired. The generated fw4 forward chain normally contains: If I restrict the flowtable rule to IPv4 only: nft 'replace rule inet fw4 forward handle 39 meta nfproto ipv4 meta l4proto { tcp, udp } flow add @ft'then IPv4 still gets hardware offload, and IPv6 remains stable on the software path. After that workaround: So the local workaround I am using is to make fw4 generate an IPv4-only flowtable rule. In to: After restarting firewall, the generated nft rule is: This keeps IPv4 fast with PPE/hardware offload while avoiding the broken IPv6 hardware offload path. This looks like an IPv6 PPE/NPU offload issue for routed forwarded flows. I can provide a redacted config/debug bundle if useful. |
Beta Was this translation helpful? Give feedback.
Replies: 6 comments 13 replies
-
|
Hey, |
Beta Was this translation helpful? Give feedback.
-
|
Self update |
Beta Was this translation helpful? Give feedback.
-
|
@tompointexe thanks for the update. That matches my setup. I retested on the latest UBI2 build:
The issue is still present here with the default fw4 flowtable rule: IPv4 remains fast with hardware offload, but forwarded IPv6 TCP/UDP becomes unusable or extremely unstable once it enters the hardware flow offload/PPE path. The IPv4-only flowtable workaround is still stable: With that workaround, IPv4 still uses hardware offload and IPv6 stays stable on the software path. So yes, this looks consistent with your self-update: the bug may be tied to routed delegated-prefix IPv6, at least with a /60 PD. |
Beta Was this translation helpful? Give feedback.
-
|
Yup, can confirm, it seems like it only applies when you get a prefix bigger or equal to /60, smaller prefixes doesen't seems affected (in my very small testing) |
Beta Was this translation helpful? Give feedback.
-
|
thanks, that confirms my setup; I’ll keep the IPv4-only flowtable workaround until there is a hardware/PPE fix. |
Beta Was this translation helpful? Give feedback.
-
|
Going to need some debugging to see what's going on.
|
Beta Was this translation helpful? Give feedback.
-
|
Here is the result for the script on my side, took while while loading any IPV6 Compatible website (nperf) Tryed an ip -6 route, but I think my config is ... a bit ducked Tell me if I miss anything usefull to diagnose this |
Beta Was this translation helpful? Give feedback.
-
|
I ran Setup:
Router was using the default/full flowtable rule: meta l4proto { tcp, udp } flow add @ftand: firewall.@defaults[0].flow_offloading='1'
firewall.@defaults[0].flow_offloading_hw='1'I forced DNS resolution in the client test to avoid mixing this with a DNS issue: curl -6 --resolve "proof.ovh.net:443:[2001:41d0:242:d300::]" \
-L -sS --connect-timeout 5 --max-time 30 \
-o /dev/null \
-w "exit=%{exitcode} http=%{http_code} total=%{time_total} bytes=%{size_download} speed=%{speed_download} remote=%{remote_ip}\n" \
https://proof.ovh.net/files/10Mb.datResult while full IPv4+IPv6 flowtable offload was active: The diagnostic script did show IPv6 hardware-bound entries during the stall, but After restoring the IPv4-only flowtable workaround: meta nfproto ipv4 meta l4proto { tcp, udp } flow add @ftthe same Pi IPv6 test completed immediately: So this still matches the I attached the redacted logs from this run:
ppe-v6-rpi400-client-20260705-134414.log |
Beta Was this translation helpful? Give feedback.
-
|
Thanks for running that! That run worked but landed in the "inconclusive" bucket, so I've reworked the script to capture a bit more data Update ppe-v6-diag.sh . We need push the router and try and get a sustained, forced-IPv6 download, use a big test file and pull it with -6 so you know it's actually going over v6 (not falling back to v4). Open Command Prompt or PowerShell and run: Run the @arnaud-devops Thanks I'll take a look at those logs |
Beta Was this translation helpful? Give feedback.
-
|
Excellent captures the client-side curl and the router-side together basically nail this down, especially since (checking the clocks) your ppe-v6-diag run at 11:44:14 GMT overlapped the curl that ran 13:44:16 CEST = 11:44:16 GMT. So this is a router-side snapshot of the flow while it was stalling. Thanks for lining those up. A. The definitive test disable hardware offload only (leaves software flow-offload on) and re-run the exact same curl: If the download now completes at full speed, that confirms the PPE IPv6 hardware path has the bug. Re-enable after with ='1' (commit + reload). B. One cleaner capture, if you don't mind, grab the updated ppe-v6-diag.sh (it now adds a WAN-RX-vs-LAN-TX packet table + conntrack [OFFLOAD] flags, which show the packets entering the WAN and not leaving the LAN), and this time start the diag first, then start the curl the key moment is the software hardware promotion in the first second or two, which the 25 s window mostly missed. |
Beta Was this translation helpful? Give feedback.
-
|
I reran the test with the updated ppe-v6-diag.sh and a faster IPv6 target:
A. Software flow offload only:
Result: download completed successfully:
Result: download stalled and timed out: The router-side diag captured the current Scaleway flow with [HW_OFFLOAD]. The IPv6 PPE entries are present, but PKT_DLT After restoring the IPv4-only flowtable workaround, the same Scaleway IPv6 download completed again: All attached logs are redacted. ppe-v6-rpi400-gilly-final-post-workaround-20260705-152144.log |
Beta Was this translation helpful? Give feedback.
-
|
One last confirmation. I've added a "Raw FOE entry bytes" section to the diag that prints exactly that. Could you grab the updated ppe-v6-diag.sh, put your WAN back to the broken config, start a Scaleway IPv6 download so it's stalling, and while it's stalling run:
|
Beta Was this translation helpful? Give feedback.
-
|
I reran the updated ppe-v6-diag.sh with the new "Raw FOE entry bytes" section. Procedure:
Result: The diag includes the Raw FOE section. The Scaleway IPv6 entry is present as 01a9c, with: Attached:
Both logs are redacted. I preserved the FOE fields needed for analysis. ppe-v6-rpi400-gilly-rawfoe-client-20260705-202408.log |
Beta Was this translation helpful? Give feedback.
-
|
That capture nailed it thanks, arnaud-devops! Your stalling flow is being programmed into the offload engine correctly, VLAN 100 tag and all. But the hardware then forwards zero packets on it. So this isn't a config or driver mistake: the AN7581 chip simply can't hardware-offload IPv6 traffic that needs a VLAN tag added on the WAN side. That's why it only affects people with a tagged WAN like yours, untagged WANs and LAN-side VLANs use a different path and work fine. So it's not a HW1-vs-HW2 issue. Since the chip can't handle this case (above my pay grade to change), I'll patch the kernel to skip hardware offload for that specific scenario and fall back to software instead, which should resolve the stalling. IPv4 and normal IPv6 traffic keep full hardware offload. I'll build a patched image for testing and post the link here once it's ready. |
Beta Was this translation helpful? Give feedback.
-
|
@Gilly1970 FYI, a couple of new datapoints from the OpenWrt forum may be relevant to the patch scope:
This does not change my capture: on my Bouygues setup ( But maybe this means the condition is narrower than "any IPv6 + WAN VLAN tag", or we need to confirm whether junky’s test is really entering IPv6 HW_OFFLOAD/PPE. Would it help if junky runs Edit: I asked him on the OpenWrt forum here: |
Beta Was this translation helpful? Give feedback.
-
|
Yes, it would definitely help if junky could run the same captures, if he's reporting it works, before we conclude tagged-WAN works, because "2 Gbit/s with offload enabled" doesn't yet prove his IPv6 flow is actually HW-bound. flow_offloading_hw=1 can be set while that particular flow quietly rides the software flowtable, and on this SoC that path can still move a lot of traffic. (Though 2 Gbit/s is high enough that it might well be real HW offload which is exactly why I want to see it.) |
Beta Was this translation helpful? Give feedback.
-
|
What junky's capture shows is the same black-hole fingerprint.. His flow to the Scaleway test box (2001:bc8:3::7, the same server you used): That entry is:
Both directions show it: 005bc (client to server, egress wan.2002, VLAN pushed) and 026e0 (server to client, egress br-lan, no VLAN) are both packets=0. And the conntrack shows only 69 KB moved on that connection, the exact "small burst in software, then stall once it offloads" pattern you saw (160 KB). Not a conclusive capture and would still need a clean under-load capture to call this. |
Beta Was this translation helpful? Give feedback.
-
|
Maybe OpenWRT-fanboy can spin up a test image for the guys on HW.2.1. to test out? |
Beta Was this translation helpful? Give feedback.
-
|
Now is a good time to remind everyone they can piggyback on my build infrastructure to crank out their own custom builds. |
Beta Was this translation helpful? Give feedback.
That capture nailed it thanks, arnaud-devops!
Your stalling flow is being programmed into the offload engine correctly, VLAN 100 tag and all. But the hardware then forwards zero packets on it. So this isn't a config or driver mistake: the AN7581 chip simply can't hardware-offload IPv6 traffic that needs a VLAN tag added on the WAN side. That's why it only affects people with a tagged WAN like yours, untagged WANs and LAN-side VLANs use a different path and work fine. So it's not a HW1-vs-HW2 issue.
Since the chip can't handle this case (above my pay grade to change), I'll patch the kernel to skip hardware offload for that specific scenario and fall back to software instead, which should r…