Netgear R7800 performance drop in 23.05.0-rc1

[robodogspok]

Describe the bug

This is copy of message of Openwrt forums:

Link for forum post also.
https://forum.openwrt.org/t/netgear-r7800-performance-drop-in-23-05-0-rc1/162676/1

I installed 23.05.0-rc1 today on my R7800. Network throughput for ethernet-connected LAN devices dropped substantially.
I reinstalled 22.03.5 and performance went back to normal.
ISP is Verizon Fios (Boston MA area), provisioned at 300Mbps symmetric.

From https://www.speakeasy.net/speedtest/ results, a sample below. I see similar degradation with speedtest ookla app, and with bufferbloat at https://www.waveform.com/tools/bufferbloat 1.

Switching back and forth between 22.03.5 and 23.05.0-rc1 consistently shows 23.05.0-rc1 with a throughput loss.

------------------------------ // ----------------------------

It seems there is performance related regression. I did not found bug report in Github. That why I posted bug report.

There was also message about: It is likely about the CPU cache frequency scaling changes in kernel 5.15 and 6.1.

OpenWrt version

r23069-e2701e0f33

OpenWrt target/subtarget

ipq806x/generic

Device

Netgear Nighthawk X4S R7800

Image kind

Official downloaded image

Steps to reproduce

When installed new version there much worse network performance. So network throughput for ethernet-connected LAN devices dropped substantially.

Actual behaviour

No response

Expected behaviour

It was expected that performance would be normal like in Openwrt version 22.03.5.

Additional info

It is likely about the CPU cache frequency scaling changes in kernel 5.15 and 6.1.

Diffconfig

No response

Terms

• I am reporting an issue for OpenWrt, not an unsupported fork.

[brada4]

Can you repeat your tests with 'performance' and 'powersave' governors?

[brada4]

Also install irqbalance, it may help if network card has multiple processing queues attached to multiple interrupts and some other kernel change does not balance IRQs, by moving those on 2nd CPU.

[motolav]

6f9495b is in the 23.05 tree, rc2 should have improved performance at the cost of heat and power usage.

[motolav]

If you want to test for yourself you can grab a 23.05-snapshot here, https://downloads.openwrt.org/releases/23.05-SNAPSHOT/targets/

[Luckyparty]

Tested with latest 23.05 snapshot of today. No packages installed except iperf3. Getting only 250 MBit/s in upstream direction. Downstream looks good (930 MBit/s). I have two R7800. One is still running 19.07.10. I did the same test with this router again and I am getting 930 MBit/s in both directions. So there is still a performance issue with 23.05. All routers are in the same network. The iperf server to test with is an Archer C2600 (ipq806x). Also tested with a Windows iperf server but the result is nearly identical.

Went back to 22.03.5 and did the test again. 930 MBit/s in both directions. So the problem is clearly in between kernel 5.10.176 and 5.15

[motolav]

@Luckyparty That's likely a different issue as the original performance drop was intentional through the kernel config to try keep ipq806x stable without switching to the performance governor

[Luckyparty]

@motolav I just thought this is worth mentioning since it has major impact on network performance. Maybe someone else can confirm this problem.

[brada4]

Hint:
/sys/devices/system/cpu/cpufreq/*/scaling_available_governors

[Luckyparty]

@brada4 The scaling_governor was already set to performance. Using one of the other governors results in even less upstream throughput. So this does not solve the issue. Between 22.03.5 and 23.05 I realized that the scaling_driver has changed from krait-cpufreq to cpufreq-dt.

[brada4]

(Not related to op thus)

[sapphonie]

Tested with latest 23.05 snapshot of today. No packages installed except iperf3. Getting only 250 MBit/s in upstream direction. Downstream looks good (930 MBit/s). I have two R7800. One is still running 19.07.10. I did the same test with this router again and I am getting 930 MBit/s in both directions. So there is still a performance issue with 23.05. All routers are in the same network. The iperf server to test with is an Archer C2600 (ipq806x). Also tested with a Windows iperf server but the result is nearly identical.

Went back to 22.03.5 and did the test again. 930 MBit/s in both directions. So the problem is clearly in between kernel 5.10.176 and 5.15

Can repro similar results with regard to 22.03.5 and latest dev snapshot (r23684-881235c713 / LuCI Master git-23.158.78004-23a246e)

[sapphonie]

Perhaps related to the two removed patches in https://git.openwrt.org/?p=openwrt/openwrt.git;a=commit;h=32f134fbdf027afc342caea17200728033747333 ? Maybe they were upstreamed poorly or incorrectly?

[motolav]

@sapphonie those patches have nothing to do with ipq806x, the reduced upstream bandwidth is probably from upstream changes in the kernel that got backported.

[motolav]

There is one difference on target/linux/generic/files/drivers/net/phy/ar8327.c(the switch driver) between 22.03 and main/23.05
But I have no idea if that'd cause any issues

[olemmela]

@robodogspok could you test if #13323 fixes your performance issues

[motolav]

@Luckyparty are you able to test if olemmela's patch fixes the upstream bandwidth issue #13323

[Luckyparty]

@motolav at the moment I don't have a build environment available. So unfortunately no. But I am really curious if this fixes the upstream.

[Luckyparty]

Thanks to #13323 the upload bandwidth problem is fixed. Tested with latest 23.05.0 on a R7800.