[RFC] package: add Shortcut Forwarding Engine driver

[dissent1]

Add an opensource Qualcomm Shortcut FE driver.

The driver consists of:

• sfe (core modules): shortcut-fe (ipv4) and shortcut-fe-ipv6 - modules that actually do the offloading
• shortcut-fe-cm - simple shortcut-fe connection manager - a module that creates offloading rules and passes to sfe
• fast-classifier - advanced connection manager - a module that creates offloading rules and passes to sfe. Comparing to sfe-cm has additional checks, connection statistics and creates an offloading rule only when a connection hits 128 packets flowed (can be adjusted through /sys/fast_classifier/offload_at_pkts)
• userspace-fast-classifier - a fast-classifier user-space tool to manually add connections one want to offload
  Both sfe-cm and fast-classifier get information about connections through netfilter. And after some checks it creates an offloading rule that is passed to sfe. After that sfe marks a connection as fast-forwarded and routes the flow.

Kernel parses this mark and bypasses its network stack taps for a specific connection in favor of sfe.

There are 3 patches added with this commit:
950 - adds a hook into network stack, sets no tcp window check to 0 (according to sfe readme).
951 - because fast-classifier also supports bridge offloading (echo 1 > /sys/fast_classifier/skip_to_bridge_ingress) and offloaded connections do not naturally update bridge statistics, fast-classifier will do that.
952 - allow multiple listeners for conntrack events, it is requied for netlink and sfe coexistance (for example)

All these patches are filled with ifdefs so the sfe code path is engaged only when these modules are compiled. It allows to keep kernel code untouched in case of no sfe compiled in.

Because sfe-cm and fast-classifier are competing modules, It's recommended to keep loaded only one of them. If both are loaded then in a race condition sfe-cm always comes first.

The driver is a merger of https://source.codeaurora.org/quic/qsdk/oss/lklm/shortcut-fe/log/?h=release/endive_cc with https://chromium.googlesource.com/chromiumos/third_party/kernel/+/chromeos-3.18/drivers/net/ethernet/qualcomm/sfe/

I've also added missing function to update udp statistics by fast-classifier and some more small fixes.

Patches are taken from QSDK and that chromeos-3.18 repo and a bit adjusted.

[dissent1]

Forgot to mention, SQM ingress is is not supported.
Benchmarking can be found in this thread https://forum.lede-project.org/t/qualcomm-fast-path-for-lede/4582

[dissent1]

To resolve the confusion it should be noted, that this driver is architecture agnostic and does not rely on hardware engines.

[dissent1]

Fixed a typo that pops when compiling with highest debug level
dissent1@bd2a683

[dissent1]

SFE does not break sqm anymore. Put patches into hack-4.x dir.
Update the commit with following changes:
dissent1@0468857
dissent1@bc187a8
dissent1@e5e150a

[philjohn]

Since those commits the build fails, due to the patches being in hack-4.* instead of patches-4.*

[dissent1]

That's actually correct. If you are compiling for stable 17.01 branch then you should manually copy kernel patches into patches-4.4 dir

[philjohn]

Yeah, that's what I ended up doing, thanks for confirming though!

I've got the fastpath patches applied to two different routers running custom builds of the 17.01 branch, both showing the different routes using fastpath in /sys/fashpath/debug_info so it looks like it's working as intended, and hasn't broken anything else (not currently using SQM)

[ypjalt]

MWAN3 doesn't work

[thagabe]

Noob here! Can I just pull this into a branch, commit, then build for my router (ARM)? Or do i need to go through the menuconfig and enable stuff? Right now I have the branch with the patches merged and building but nothing on my menuconfig changed. Please let me know if i can help in anyway (tests or otherwise)

[MartB]

👍 Nice work

[tapper82]

Any news on this? Is it working with SQM now?

[davidzodelin]

@tapper82 yes, working with SQM.

[montvid]

Would be nice to have it merged because of speed, cpu usage improvements. I would like to test this commit but first I have to learn how to build it for tp-link 1043nd v2.
dissent1 could you maybe do compiled ipk packages so more people could test?

[philjohn]

@thagabe I've got it running successfully on an R7800, which is ARM without issues. I'll try and paste some enabled/disabled iperf3 benches when I get home.

Has been stable for over a week now, so not seeing any instability here.

[philjohn]

@montvid you could always just take gwlim's precompiled fastpath build. I've got a 1043ND V1 in my network and have built a custom image with this patch applied, so it's not particularly difficult - contact me offline and I can provide my build env.

[montvid]

Thanks for the offer philjohn. I learned how to use git and patch LEDE so I tried to test the patch on 1043nd v2 connected two computers via lan. I did not want to reflash firmware so I just removed the packages, rebooted and tested. It seems either the removal of packages does not remove the patch or the performance is the same with or without the patch/packages. I guess more tests are needed with other hardware.
iperf3 -c 192.168.1.142 -i 1 -t 100 -V
Packages removed:
Test Complete. Summary Results:
[ ID] Interval Transfer Bandwidth Retr
[ 4] 0.00-100.00 sec 10.9 GBytes 933 Mbits/sec 0 sender
[ 4] 0.00-100.00 sec 10.9 GBytes 933 Mbits/sec receiver
CPU Utilization: local/sender 5.2% (0.1%u/5.1%s), remote/receiver 5.3% (0.4%u/4.9%s)
Packages installed:
Test Complete. Summary Results:
[ ID] Interval Transfer Bandwidth Retr
[ 4] 0.00-100.00 sec 10.9 GBytes 934 Mbits/sec 0 sender
[ 4] 0.00-100.00 sec 10.9 GBytes 933 Mbits/sec receiver
CPU Utilization: local/sender 5.2% (0.1%u/5.1%s), remote/receiver 3.4% (0.2%u/3.1%s)

[haruhitamako]

I put the patches into the patches-4.4 when I make 17.01 ,Will the SQM be broken again？

[ezar]

Someone can help me to add this PR to current environment?

[danielpetre]

20 days and this awesome patchset still isn't commited to LEDE..?

[ezar]

Commit please!

[jeolives]

SQM (cake and layer_cake.qos) with additional flags "mpu 64 nat dual-dst/srchost" seems to works alongside this. Also checked if it was even working by cat /sys/fast_classifier/debug_info.

[Mushoz]

This is probably one of the most sought-after enhancements of LEDE. Is there anybody working on reviewing the code? Would love to see this committed.

[commodo]

the copyright here is wrong; it should be

#
# Copyright (C) 2017 LEDE Project
#
# This is free software, licensed under the GNU General Public License v2.
# See /LICENSE for more information.

the copyright in this Makefile refers to LEDE Copyright

[Misiek304]

I've had a few serious problems with SQM (cake) and SFE. The SQM itself worked fine but more advanced options of cake didn't. I believe that SFE cripples cake's potential.

[thagabe]

@Misiek304 I have noticed this too, however (as my use case is a small network) i opted to disable cake in favor of SFE. Maybe SFE should disable cake if it is selected. At lease until a proper fix, if there is one, is hammered out.

[Misiek304]

Unfortunately I can't afford to disable SQM and cake in my case use. I depend on it too much so I opted out to get rid of SFE. I hope that in the future releases this issue will get resolved.

For now I will satisfy on a clear warning notice that SQM and cake don't work properly with SFE active. I've seen many claims to the contrary on the forum and I think this needs to be addressed to at least avoid flood of issue threads about latency and bufferbload. I also think that not having a clear statement on this subject hurts entire SQM and cake projects since it greatly affects peoples reception about cake effectiveness.

The best case scenario would be to have ISP's implement cake in their cable modems.

[davidzodelin]

What are the advanced options of SQM Cake that in your case does not work? I've been using SFE and SQM Cake for 2 months in a real environment and the only problem I've noticed is that SFE does not work well with long-distance wireless links.

  Please, I need to know what are those advanced options that do not work for you because in my case I also depend on them.

[Misiek304]

option iqdisc_opts 'nat dual-dsthost diffserv4 rtt 300ms mpu 64'
option eqdisc_opts 'nat dual-srchost diffserv4 rtt 300ms mpu 64'

Those settings did not work for sure. As soon as I removed SFE those settings were activated.

[MOZGIII]

Afaik traffic that goes fast path is not affected by SQM. That would explain why it doesn't work. Am I right?

[davidzodelin]

It's strange. I use the following features:

diffserv4 triple-isolate nat rtt 100.0ms noatm overhead 44 via-ethernet mpu 64

  And before my ISP removed the access points nearby that were 100 meters away everything worked perfectly. Now I have noticed that SFE goes wrong when I connect to an access point that is approximately 500 meters away. Are you sure you are using the SFE with the fixes that Dissent did for the ingress when using tc qdisc?

[Misiek304]

The nat dual-srchost didn't work for sure. I've tested this. It behaved like those options were non existent. As soon as I removed SFE, cake behaved well and I saw big improvement. SFE affected mostly upload performance.

[lucize]

something is strange, I have a cisco ip phone 7960 behind a router linked to the Call Manager by wireguard, and after adding the fastpath option to the router the calls can't be made by the first attempt, and is strange because I'm doing routing not NAT to other devices in VPN

[lucize]

@dissent1 this is really needed ?
can't it be compiled without IPV6 ?
maybe change a bit the logic about shortcut-fe-ipv6.ko ?

[blogic]

With the remerge in progress, all PRs on the lede-project organisation will be closed. Please help getting this merged or rebase/post it on the openwrt project page (https://github.com/openwrt/openwrt/pulls). All remaining PRs will be closed in 30 days.

[Priyantha]

Hi @blogic
How can we help, as users, to encourage the LEDE/OpenWRT developers to include this fantastic PR?

[Priyantha]

@blogic I see you have closed this pull request, does this mean this PR is not going to be merged? Or will there be a new PR?

[blogic]

@Priyantha read the comments ... "With the remerge in progress, all PRs on the lede-project organisation will be closed. Please help getting this merged or rebase/post it on the openwrt project page (https://github.com/openwrt/openwrt/pulls). All remaining PRs will be closed in 30 days."

[jow-]

This particular change however is unlikely to land in OpenWrt as the project opts for the upstream fast path implementation instead.

[Priyantha]

Ah okay, there is an upstream fast path implementation? I wasn't aware of that 👍 when there is one, that would be great. It's a big improvement. So I am hoping it will be part of the new OpenWRT trunk in the near future.

[LipkeGu]

Shortcut is an in-Linux-kernel IP packet forwarding engine. It's designed
to offer very high speed IP packet forwarding based on IP connection tracking.
It's dramatically faster than the standard netfilter-based NAT forwarding path
but is designed to synchronise state back to netfilter/conntrack so that it
doesn't need to deal with all of the complexities of special cases.

[nbd168]

FYI, I've backported the upstream flow offload code to Linux 4.14 in my staging tree. I've also written an extra patch that integrates it with iptables (so no dependency on nftables anymore).
You can test it by building my staging tree, enabling kmod-ipt-flow, and adding a rule to the FORWARD chain with -j FLOWOFFLOAD

[lucize]

@nbd168 sorry for asking, can you give an example to see how it would need to be ?

[Mushoz]

I cannot find the mentioned kmod-ipt-flow in menuconfig. I do find these 3 modules, however:

kmod-ipt-offload
kmod-nf-flow
kmod-nft-offload

Which ones do I need?

Edit: Felix clarified on the LEDE forum. You only need kmod-ipt-offload which will automatically select kmod-nf-flow as well. kmod-nft-offload is not needed.

[LuKePicci]

Just for the records, the SFE code in this PR had also issues over IPSec VPNs. The issue troubleshooting is simple: as soon as a connection passing through the tunnel is picked for shortcutting it breaks, so you see the TCP layer often doing retransmission and, for instance, relatively large web pages load only partially.