New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[RFC] package: add Shortcut Forwarding Engine driver #1269

Closed
wants to merge 1 commit into
base: master
from

Conversation

Projects
None yet
@dissent1
Contributor

dissent1 commented Aug 2, 2017

Add an opensource Qualcomm Shortcut FE driver.

The driver consists of:

  • sfe (core modules): shortcut-fe (ipv4) and shortcut-fe-ipv6 - modules that actually do the offloading
  • shortcut-fe-cm - simple shortcut-fe connection manager - a module that creates offloading rules and passes to sfe
  • fast-classifier - advanced connection manager - a module that creates offloading rules and passes to sfe. Comparing to sfe-cm has additional checks, connection statistics and creates an offloading rule only when a connection hits 128 packets flowed (can be adjusted through /sys/fast_classifier/offload_at_pkts)
  • userspace-fast-classifier - a fast-classifier user-space tool to manually add connections one want to offload
    Both sfe-cm and fast-classifier get information about connections through netfilter. And after some checks it creates an offloading rule that is passed to sfe. After that sfe marks a connection as fast-forwarded and routes the flow.

Kernel parses this mark and bypasses its network stack taps for a specific connection in favor of sfe.

There are 3 patches added with this commit:
950 - adds a hook into network stack, sets no tcp window check to 0 (according to sfe readme).
951 - because fast-classifier also supports bridge offloading (echo 1 > /sys/fast_classifier/skip_to_bridge_ingress) and offloaded connections do not naturally update bridge statistics, fast-classifier will do that.
952 - allow multiple listeners for conntrack events, it is requied for netlink and sfe coexistance (for example)

All these patches are filled with ifdefs so the sfe code path is engaged only when these modules are compiled. It allows to keep kernel code untouched in case of no sfe compiled in.

Because sfe-cm and fast-classifier are competing modules, It's recommended to keep loaded only one of them. If both are loaded then in a race condition sfe-cm always comes first.

The driver is a merger of https://source.codeaurora.org/quic/qsdk/oss/lklm/shortcut-fe/log/?h=release/endive_cc with https://chromium.googlesource.com/chromiumos/third_party/kernel/+/chromeos-3.18/drivers/net/ethernet/qualcomm/sfe/

I've also added missing function to update udp statistics by fast-classifier and some more small fixes.

Patches are taken from QSDK and that chromeos-3.18 repo and a bit adjusted.

@dissent1

This comment has been minimized.

Show comment
Hide comment
@dissent1

dissent1 Aug 2, 2017

Contributor

Forgot to mention, SQM ingress is is not supported.
Benchmarking can be found in this thread https://forum.lede-project.org/t/qualcomm-fast-path-for-lede/4582

Contributor

dissent1 commented Aug 2, 2017

Forgot to mention, SQM ingress is is not supported.
Benchmarking can be found in this thread https://forum.lede-project.org/t/qualcomm-fast-path-for-lede/4582

@dissent1 dissent1 changed the title from package: add Shortcut Forwarding Engine driver to [RFC] package: add Shortcut Forwarding Engine driver Aug 2, 2017

@dissent1

This comment has been minimized.

Show comment
Hide comment
@dissent1

dissent1 Aug 2, 2017

Contributor

To resolve the confusion it should be noted, that this driver is architecture agnostic and does not rely on hardware engines.

Contributor

dissent1 commented Aug 2, 2017

To resolve the confusion it should be noted, that this driver is architecture agnostic and does not rely on hardware engines.

@dissent1

This comment has been minimized.

Show comment
Hide comment
@dissent1

dissent1 Aug 3, 2017

Contributor

Fixed a typo that pops when compiling with highest debug level
dissent1@bd2a683

Contributor

dissent1 commented Aug 3, 2017

Fixed a typo that pops when compiling with highest debug level
dissent1@bd2a683

package: add Shortcut Forwarding Engine driver
Add an opensource Qualcomm Shortcut FE driver.

The driver consists of:
- sfe (core modules): shortcut-fe (ipv4) and shortcut-fe-ipv6 -
modules that actually do the offloading
- shortcut-fe-cm - simple shortcut-fe connection manager - a
module that creates offloading rules and passes to sfe
- fast-classifier - advanced connection manager - a module that
creates offloading rules and passes to sfe. Comparing to sfe-cm
has additional checks, connection statistics and creates an
offloading rule only when a connection hits 128 packets flowed
(can be adjusted through /sys/fast_classifier/offload_at_pkts)
- userspace-fast-classifier - a fast-classifier user-space tool
to manually add connections one want to offload
Both sfe-cm and fast-classifier get information about connections
through netfilter. And after some checks it creates an offloading
rule that is passed to sfe. After that sfe marks a connection as
fast-forwarded and routes the flow.

Kernel parses this mark and bypasses its network stack taps for
a specific connection in favor of sfe.

There are 3 patches added with this commit:
950 - adds a hook into network stack, sets no tcp window check
to 0 (according to sfe readme).
951 - because fast-classifier also supports bridge offloading
(echo 1 > /sys/fast_classifier/skip_to_bridge_ingress) and
offloaded connections do not naturally update bridge statistics,
fast-classifier will do that.
952 - allow multiple listeners for conntrack events, it is
requied for netlink and sfe coexistance (for example)

All these patches are filled with ifdefs so the sfe code path
is engaged only when these modules are compiled. It allows to
keep kernel code untouched in case of no sfe compiled in.

Because sfe-cm and fast-classifier are competing modules, It's
recommended to keep loaded only one of them. If both are loaded
then in a race condition sfe-cm always comes first.

The driver is a merger of
https://source.codeaurora.org/quic/qsdk/oss/lklm/shortcut-fe/log/?h=release/endive_cc
with
https://chromium.googlesource.com/chromiumos/third_party/kernel/+/chromeos-3.18/drivers/net/ethernet/qualcomm/sfe/

I've also added missing function to update udp statistics by
fast-classifier and some more small fixes.

Patches are taken from QSDK and that chromeos-3.18 repo and a bit
adjusted.

Signed-off-by: Pavel Kubelun <be.dissent@gmail.com>
@dissent1

This comment has been minimized.

Show comment
Hide comment
@dissent1

dissent1 Aug 8, 2017

Contributor

SFE does not break sqm anymore. Put patches into hack-4.x dir.
Update the commit with following changes:
dissent1@0468857
dissent1@bc187a8
dissent1@e5e150a

Contributor

dissent1 commented Aug 8, 2017

SFE does not break sqm anymore. Put patches into hack-4.x dir.
Update the commit with following changes:
dissent1@0468857
dissent1@bc187a8
dissent1@e5e150a

@philjohn

This comment has been minimized.

Show comment
Hide comment
@philjohn

philjohn Aug 9, 2017

Since those commits the build fails, due to the patches being in hack-4.* instead of patches-4.*

philjohn commented Aug 9, 2017

Since those commits the build fails, due to the patches being in hack-4.* instead of patches-4.*

@dissent1

This comment has been minimized.

Show comment
Hide comment
@dissent1

dissent1 Aug 9, 2017

Contributor

That's actually correct. If you are compiling for stable 17.01 branch then you should manually copy kernel patches into patches-4.4 dir

Contributor

dissent1 commented Aug 9, 2017

That's actually correct. If you are compiling for stable 17.01 branch then you should manually copy kernel patches into patches-4.4 dir

@philjohn

This comment has been minimized.

Show comment
Hide comment
@philjohn

philjohn Aug 9, 2017

Yeah, that's what I ended up doing, thanks for confirming though!

I've got the fastpath patches applied to two different routers running custom builds of the 17.01 branch, both showing the different routes using fastpath in /sys/fashpath/debug_info so it looks like it's working as intended, and hasn't broken anything else (not currently using SQM)

philjohn commented Aug 9, 2017

Yeah, that's what I ended up doing, thanks for confirming though!

I've got the fastpath patches applied to two different routers running custom builds of the 17.01 branch, both showing the different routes using fastpath in /sys/fashpath/debug_info so it looks like it's working as intended, and hasn't broken anything else (not currently using SQM)

@ypjalt

This comment has been minimized.

Show comment
Hide comment
@ypjalt

ypjalt Aug 11, 2017

MWAN3 doesn't work

ypjalt commented Aug 11, 2017

MWAN3 doesn't work

@thagabe

This comment has been minimized.

Show comment
Hide comment
@thagabe

thagabe Aug 11, 2017

Contributor

Noob here! Can I just pull this into a branch, commit, then build for my router (ARM)? Or do i need to go through the menuconfig and enable stuff? Right now I have the branch with the patches merged and building but nothing on my menuconfig changed. Please let me know if i can help in anyway (tests or otherwise)

Contributor

thagabe commented Aug 11, 2017

Noob here! Can I just pull this into a branch, commit, then build for my router (ARM)? Or do i need to go through the menuconfig and enable stuff? Right now I have the branch with the patches merged and building but nothing on my menuconfig changed. Please let me know if i can help in anyway (tests or otherwise)

@MartB

This comment has been minimized.

Show comment
Hide comment
@MartB

MartB Aug 12, 2017

👍 Nice work

MartB commented Aug 12, 2017

👍 Nice work

@tapper82

This comment has been minimized.

Show comment
Hide comment
@tapper82

tapper82 Aug 13, 2017

Contributor

Any news on this? Is it working with SQM now?

Contributor

tapper82 commented Aug 13, 2017

Any news on this? Is it working with SQM now?

@davidzodelin

This comment has been minimized.

Show comment
Hide comment
@davidzodelin

davidzodelin Aug 14, 2017

@tapper82 yes, working with SQM.

davidzodelin commented Aug 14, 2017

@tapper82 yes, working with SQM.

@montvid

This comment has been minimized.

Show comment
Hide comment
@montvid

montvid Aug 17, 2017

Would be nice to have it merged because of speed, cpu usage improvements. I would like to test this commit but first I have to learn how to build it for tp-link 1043nd v2.
dissent1 could you maybe do compiled ipk packages so more people could test?

montvid commented Aug 17, 2017

Would be nice to have it merged because of speed, cpu usage improvements. I would like to test this commit but first I have to learn how to build it for tp-link 1043nd v2.
dissent1 could you maybe do compiled ipk packages so more people could test?

@philjohn

This comment has been minimized.

Show comment
Hide comment
@philjohn

philjohn Aug 18, 2017

@thagabe I've got it running successfully on an R7800, which is ARM without issues. I'll try and paste some enabled/disabled iperf3 benches when I get home.

Has been stable for over a week now, so not seeing any instability here.

philjohn commented Aug 18, 2017

@thagabe I've got it running successfully on an R7800, which is ARM without issues. I'll try and paste some enabled/disabled iperf3 benches when I get home.

Has been stable for over a week now, so not seeing any instability here.

@philjohn

This comment has been minimized.

Show comment
Hide comment
@philjohn

philjohn Aug 18, 2017

@montvid you could always just take gwlim's precompiled fastpath build. I've got a 1043ND V1 in my network and have built a custom image with this patch applied, so it's not particularly difficult - contact me offline and I can provide my build env.

philjohn commented Aug 18, 2017

@montvid you could always just take gwlim's precompiled fastpath build. I've got a 1043ND V1 in my network and have built a custom image with this patch applied, so it's not particularly difficult - contact me offline and I can provide my build env.

@montvid

This comment has been minimized.

Show comment
Hide comment
@montvid

montvid Aug 19, 2017

Thanks for the offer philjohn. I learned how to use git and patch LEDE so I tried to test the patch on 1043nd v2 connected two computers via lan. I did not want to reflash firmware so I just removed the packages, rebooted and tested. It seems either the removal of packages does not remove the patch or the performance is the same with or without the patch/packages. I guess more tests are needed with other hardware.
iperf3 -c 192.168.1.142 -i 1 -t 100 -V
Packages removed:
Test Complete. Summary Results:
[ ID] Interval Transfer Bandwidth Retr
[ 4] 0.00-100.00 sec 10.9 GBytes 933 Mbits/sec 0 sender
[ 4] 0.00-100.00 sec 10.9 GBytes 933 Mbits/sec receiver
CPU Utilization: local/sender 5.2% (0.1%u/5.1%s), remote/receiver 5.3% (0.4%u/4.9%s)
Packages installed:
Test Complete. Summary Results:
[ ID] Interval Transfer Bandwidth Retr
[ 4] 0.00-100.00 sec 10.9 GBytes 934 Mbits/sec 0 sender
[ 4] 0.00-100.00 sec 10.9 GBytes 933 Mbits/sec receiver
CPU Utilization: local/sender 5.2% (0.1%u/5.1%s), remote/receiver 3.4% (0.2%u/3.1%s)

montvid commented Aug 19, 2017

Thanks for the offer philjohn. I learned how to use git and patch LEDE so I tried to test the patch on 1043nd v2 connected two computers via lan. I did not want to reflash firmware so I just removed the packages, rebooted and tested. It seems either the removal of packages does not remove the patch or the performance is the same with or without the patch/packages. I guess more tests are needed with other hardware.
iperf3 -c 192.168.1.142 -i 1 -t 100 -V
Packages removed:
Test Complete. Summary Results:
[ ID] Interval Transfer Bandwidth Retr
[ 4] 0.00-100.00 sec 10.9 GBytes 933 Mbits/sec 0 sender
[ 4] 0.00-100.00 sec 10.9 GBytes 933 Mbits/sec receiver
CPU Utilization: local/sender 5.2% (0.1%u/5.1%s), remote/receiver 5.3% (0.4%u/4.9%s)
Packages installed:
Test Complete. Summary Results:
[ ID] Interval Transfer Bandwidth Retr
[ 4] 0.00-100.00 sec 10.9 GBytes 934 Mbits/sec 0 sender
[ 4] 0.00-100.00 sec 10.9 GBytes 933 Mbits/sec receiver
CPU Utilization: local/sender 5.2% (0.1%u/5.1%s), remote/receiver 3.4% (0.2%u/3.1%s)

@haruhitamako

This comment has been minimized.

Show comment
Hide comment
@haruhitamako

haruhitamako Aug 20, 2017

I put the patches into the patches-4.4 when I make 17.01 ,Will the SQM be broken again?

haruhitamako commented Aug 20, 2017

I put the patches into the patches-4.4 when I make 17.01 ,Will the SQM be broken again?

@ezar

This comment has been minimized.

Show comment
Hide comment
@ezar

ezar Aug 21, 2017

Someone can help me to add this PR to current environment?

ezar commented Aug 21, 2017

Someone can help me to add this PR to current environment?

@danielpetre

This comment has been minimized.

Show comment
Hide comment
@danielpetre

danielpetre Aug 22, 2017

20 days and this awesome patchset still isn't commited to LEDE..?

danielpetre commented Aug 22, 2017

20 days and this awesome patchset still isn't commited to LEDE..?

@ezar

This comment has been minimized.

Show comment
Hide comment
@ezar

ezar Aug 22, 2017

Commit please!

ezar commented Aug 22, 2017

Commit please!

@jeolivera

This comment has been minimized.

Show comment
Hide comment
@jeolivera

jeolivera Aug 27, 2017

SQM (cake and layer_cake.qos) with additional flags "mpu 64 nat dual-dst/srchost" seems to works alongside this. Also checked if it was even working by cat /sys/fast_classifier/debug_info.

jeolivera commented Aug 27, 2017

SQM (cake and layer_cake.qos) with additional flags "mpu 64 nat dual-dst/srchost" seems to works alongside this. Also checked if it was even working by cat /sys/fast_classifier/debug_info.

@Mushoz

This comment has been minimized.

Show comment
Hide comment
@Mushoz

Mushoz Aug 27, 2017

Contributor

This is probably one of the most sought-after enhancements of LEDE. Is there anybody working on reviewing the code? Would love to see this committed.

Contributor

Mushoz commented Aug 27, 2017

This is probably one of the most sought-after enhancements of LEDE. Is there anybody working on reviewing the code? Would love to see this committed.

@commodo

This comment has been minimized.

Show comment
Hide comment
@commodo

commodo Aug 27, 2017

the copyright here is wrong; it should be

#
# Copyright (C) 2017 LEDE Project
#
# This is free software, licensed under the GNU General Public License v2.
# See /LICENSE for more information.

the copyright in this Makefile refers to LEDE Copyright

commodo commented on package/kernel/shortcut-fe/Makefile in b87b473 Aug 27, 2017

the copyright here is wrong; it should be

#
# Copyright (C) 2017 LEDE Project
#
# This is free software, licensed under the GNU General Public License v2.
# See /LICENSE for more information.

the copyright in this Makefile refers to LEDE Copyright

@commodo

This comment has been minimized.

Show comment
Hide comment
@commodo

commodo Aug 27, 2017

Contributor

@dissent1
So, I'll add a bit of initial set of thoughts [ that could count as an initial review ].

Can we change from adding all the source files into the repo and reference the git repo from codeaurora.org ?
If possible, what I would do is define a section in the main Makefile like:

PKG_SOURCE_URL:=https://source.codeaurora.org/quic/qsdk/oss/lklm/shortcut-fe
PKG_SOURCE_PROTO:=git
PKG_SOURCE_DATE:=2017-06-14
PKG_SOURCE_VERSION:=25e41dabed2cda9a95dc95e3383cf2a617f88de7
PKG_MAINTAINER:=Pavel Kubelun <be.dissent@gmail.com>
PKG_LICENSE:=<license used here>

FWIW: I checked that these values are correct (based on your link):

PKG_SOURCE_DATE:=2017-06-14
PKG_SOURCE_VERSION:=25e41dabed2cda9a95dc95e3383cf2a617f88de7

And then all patches from https://chromium.googlesource.com/chromiumos/third_party/kernel/+/chromeos-3.18/drivers/net/ethernet/qualcomm/sfe/
could be turned into file-patches and put into the patches folder.
If possible, I would make each patch from this repo, a singular commit, referencing a web-link to the commit in that repo.

Since the kernel patches are only 3, what I would do, is make each kernel patch, it's own commit, and reference a web-link to each commit from where they were obtained.
And then the config-4.4 & config-4.9 could be a single commit.

From a reviewer stand-point, this PR/patch, is very difficult to review [I mostly do reviews at work and it takes some energy to do].
But if you manage to remove all the source files, and just download from the source from codeaurora.org, and add patch files [in the patches folder], it would simplify things a great deal.
Also, because it reduces the chance/risk that any of these files were copied/patched wrongly [ the assumption here is that human error is very possible ]

PKG_LICENSE [ and PKG_LICENSE_FILES fields ] are very much needed;
The legal aspects are important for this kind of project [as LEDE & OpenWrt are].
I hope that there aren't any legal aspects preventing the SFE engine to be packaged into LEDE :)

PKG_MAINTAINER is preferred, since you are implicitly taking ownership of this module [in LEDE], by committing this change.

If you need clarification or help in organizing this, feel free to ping me.
I can probably alloc some time this week to work with you into splitting this into a PR that is more manage-able for review.
That is the best that I can do, since I'm just an occasional contributor.

Contributor

commodo commented Aug 27, 2017

@dissent1
So, I'll add a bit of initial set of thoughts [ that could count as an initial review ].

Can we change from adding all the source files into the repo and reference the git repo from codeaurora.org ?
If possible, what I would do is define a section in the main Makefile like:

PKG_SOURCE_URL:=https://source.codeaurora.org/quic/qsdk/oss/lklm/shortcut-fe
PKG_SOURCE_PROTO:=git
PKG_SOURCE_DATE:=2017-06-14
PKG_SOURCE_VERSION:=25e41dabed2cda9a95dc95e3383cf2a617f88de7
PKG_MAINTAINER:=Pavel Kubelun <be.dissent@gmail.com>
PKG_LICENSE:=<license used here>

FWIW: I checked that these values are correct (based on your link):

PKG_SOURCE_DATE:=2017-06-14
PKG_SOURCE_VERSION:=25e41dabed2cda9a95dc95e3383cf2a617f88de7

And then all patches from https://chromium.googlesource.com/chromiumos/third_party/kernel/+/chromeos-3.18/drivers/net/ethernet/qualcomm/sfe/
could be turned into file-patches and put into the patches folder.
If possible, I would make each patch from this repo, a singular commit, referencing a web-link to the commit in that repo.

Since the kernel patches are only 3, what I would do, is make each kernel patch, it's own commit, and reference a web-link to each commit from where they were obtained.
And then the config-4.4 & config-4.9 could be a single commit.

From a reviewer stand-point, this PR/patch, is very difficult to review [I mostly do reviews at work and it takes some energy to do].
But if you manage to remove all the source files, and just download from the source from codeaurora.org, and add patch files [in the patches folder], it would simplify things a great deal.
Also, because it reduces the chance/risk that any of these files were copied/patched wrongly [ the assumption here is that human error is very possible ]

PKG_LICENSE [ and PKG_LICENSE_FILES fields ] are very much needed;
The legal aspects are important for this kind of project [as LEDE & OpenWrt are].
I hope that there aren't any legal aspects preventing the SFE engine to be packaged into LEDE :)

PKG_MAINTAINER is preferred, since you are implicitly taking ownership of this module [in LEDE], by committing this change.

If you need clarification or help in organizing this, feel free to ping me.
I can probably alloc some time this week to work with you into splitting this into a PR that is more manage-able for review.
That is the best that I can do, since I'm just an occasional contributor.

@tapper82

This comment has been minimized.

Show comment
Hide comment
@tapper82

tapper82 Aug 27, 2017

Contributor

I am happy to see that people are still talking about this as I thought it was a ded duck! @commodo is rite about this being the biggest thing for LEDE /OpenWRT for a long time.

Contributor

tapper82 commented Aug 27, 2017

I am happy to see that people are still talking about this as I thought it was a ded duck! @commodo is rite about this being the biggest thing for LEDE /OpenWRT for a long time.

@Misiek304

This comment has been minimized.

Show comment
Hide comment
@Misiek304

Misiek304 Nov 11, 2017

I've had a few serious problems with SQM (cake) and SFE. The SQM itself worked fine but more advanced options of cake didn't. I believe that SFE cripples cake's potential.

Misiek304 commented Nov 11, 2017

I've had a few serious problems with SQM (cake) and SFE. The SQM itself worked fine but more advanced options of cake didn't. I believe that SFE cripples cake's potential.

@thagabe

This comment has been minimized.

Show comment
Hide comment
@thagabe

thagabe Nov 12, 2017

Contributor

@Misiek304 I have noticed this too, however (as my use case is a small network) i opted to disable cake in favor of SFE. Maybe SFE should disable cake if it is selected. At lease until a proper fix, if there is one, is hammered out.

Contributor

thagabe commented Nov 12, 2017

@Misiek304 I have noticed this too, however (as my use case is a small network) i opted to disable cake in favor of SFE. Maybe SFE should disable cake if it is selected. At lease until a proper fix, if there is one, is hammered out.

@Misiek304

This comment has been minimized.

Show comment
Hide comment
@Misiek304

Misiek304 Nov 12, 2017

Unfortunately I can't afford to disable SQM and cake in my case use. I depend on it too much so I opted out to get rid of SFE. I hope that in the future releases this issue will get resolved.

For now I will satisfy on a clear warning notice that SQM and cake don't work properly with SFE active. I've seen many claims to the contrary on the forum and I think this needs to be addressed to at least avoid flood of issue threads about latency and bufferbload. I also think that not having a clear statement on this subject hurts entire SQM and cake projects since it greatly affects peoples reception about cake effectiveness.

The best case scenario would be to have ISP's implement cake in their cable modems.

Misiek304 commented Nov 12, 2017

Unfortunately I can't afford to disable SQM and cake in my case use. I depend on it too much so I opted out to get rid of SFE. I hope that in the future releases this issue will get resolved.

For now I will satisfy on a clear warning notice that SQM and cake don't work properly with SFE active. I've seen many claims to the contrary on the forum and I think this needs to be addressed to at least avoid flood of issue threads about latency and bufferbload. I also think that not having a clear statement on this subject hurts entire SQM and cake projects since it greatly affects peoples reception about cake effectiveness.

The best case scenario would be to have ISP's implement cake in their cable modems.

@davidzodelin

This comment has been minimized.

Show comment
Hide comment
@davidzodelin

davidzodelin Nov 12, 2017

What are the advanced options of SQM Cake that in your case does not work? I've been using SFE and SQM Cake for 2 months in a real environment and the only problem I've noticed is that SFE does not work well with long-distance wireless links.

  Please, I need to know what are those advanced options that do not work for you because in my case I also depend on them.

davidzodelin commented Nov 12, 2017

What are the advanced options of SQM Cake that in your case does not work? I've been using SFE and SQM Cake for 2 months in a real environment and the only problem I've noticed is that SFE does not work well with long-distance wireless links.

  Please, I need to know what are those advanced options that do not work for you because in my case I also depend on them.

@Misiek304

This comment has been minimized.

Show comment
Hide comment
@Misiek304

Misiek304 Nov 12, 2017

option iqdisc_opts 'nat dual-dsthost diffserv4 rtt 300ms mpu 64'
option eqdisc_opts 'nat dual-srchost diffserv4 rtt 300ms mpu 64'

Those settings did not work for sure. As soon as I removed SFE those settings were activated.

Misiek304 commented Nov 12, 2017

option iqdisc_opts 'nat dual-dsthost diffserv4 rtt 300ms mpu 64'
option eqdisc_opts 'nat dual-srchost diffserv4 rtt 300ms mpu 64'

Those settings did not work for sure. As soon as I removed SFE those settings were activated.

@MOZGIII

This comment has been minimized.

Show comment
Hide comment
@MOZGIII

MOZGIII Nov 12, 2017

Afaik traffic that goes fast path is not affected by SQM. That would explain why it doesn't work. Am I right?

MOZGIII commented Nov 12, 2017

Afaik traffic that goes fast path is not affected by SQM. That would explain why it doesn't work. Am I right?

@davidzodelin

This comment has been minimized.

Show comment
Hide comment
@davidzodelin

davidzodelin Nov 13, 2017

It's strange. I use the following features:

diffserv4 triple-isolate nat rtt 100.0ms noatm overhead 44 via-ethernet mpu 64

  And before my ISP removed the access points nearby that were 100 meters away everything worked perfectly. Now I have noticed that SFE goes wrong when I connect to an access point that is approximately 500 meters away. Are you sure you are using the SFE with the fixes that Dissent did for the ingress when using tc qdisc?

davidzodelin commented Nov 13, 2017

It's strange. I use the following features:

diffserv4 triple-isolate nat rtt 100.0ms noatm overhead 44 via-ethernet mpu 64

  And before my ISP removed the access points nearby that were 100 meters away everything worked perfectly. Now I have noticed that SFE goes wrong when I connect to an access point that is approximately 500 meters away. Are you sure you are using the SFE with the fixes that Dissent did for the ingress when using tc qdisc?

@Misiek304

This comment has been minimized.

Show comment
Hide comment
@Misiek304

Misiek304 Nov 19, 2017

The nat dual-srchost didn't work for sure. I've tested this. It behaved like those options were non existent. As soon as I removed SFE, cake behaved well and I saw big improvement. SFE affected mostly upload performance.

Misiek304 commented Nov 19, 2017

The nat dual-srchost didn't work for sure. I've tested this. It behaved like those options were non existent. As soon as I removed SFE, cake behaved well and I saw big improvement. SFE affected mostly upload performance.

@lucize

This comment has been minimized.

Show comment
Hide comment
@lucize

lucize Nov 29, 2017

Contributor

something is strange, I have a cisco ip phone 7960 behind a router linked to the Call Manager by wireguard, and after adding the fastpath option to the router the calls can't be made by the first attempt, and is strange because I'm doing routing not NAT to other devices in VPN

Contributor

lucize commented Nov 29, 2017

something is strange, I have a cisco ip phone 7960 behind a router linked to the Call Manager by wireguard, and after adding the fastpath option to the router the calls can't be made by the first attempt, and is strange because I'm doing routing not NAT to other devices in VPN

SECTION:=kernel
CATEGORY:=Kernel modules
SUBMENU:=Network Support
DEPENDS:=@IPV6

This comment has been minimized.

@lucize

lucize Dec 8, 2017

Contributor

@dissent1 this is really needed ?
can't it be compiled without IPV6 ?
maybe change a bit the logic about shortcut-fe-ipv6.ko ?

@lucize

lucize Dec 8, 2017

Contributor

@dissent1 this is really needed ?
can't it be compiled without IPV6 ?
maybe change a bit the logic about shortcut-fe-ipv6.ko ?

@blogic

This comment has been minimized.

Show comment
Hide comment
@blogic

blogic Jan 2, 2018

Contributor

With the remerge in progress, all PRs on the lede-project organisation will be closed. Please help getting this merged or rebase/post it on the openwrt project page (https://github.com/openwrt/openwrt/pulls). All remaining PRs will be closed in 30 days.

Contributor

blogic commented Jan 2, 2018

With the remerge in progress, all PRs on the lede-project organisation will be closed. Please help getting this merged or rebase/post it on the openwrt project page (https://github.com/openwrt/openwrt/pulls). All remaining PRs will be closed in 30 days.

@Priyantha

This comment has been minimized.

Show comment
Hide comment
@Priyantha

Priyantha Jan 12, 2018

Hi @blogic
How can we help, as users, to encourage the LEDE/OpenWRT developers to include this fantastic PR?

Priyantha commented Jan 12, 2018

Hi @blogic
How can we help, as users, to encourage the LEDE/OpenWRT developers to include this fantastic PR?

@blogic blogic closed this Feb 11, 2018

@Priyantha

This comment has been minimized.

Show comment
Hide comment
@Priyantha

Priyantha Feb 11, 2018

@blogic I see you have closed this pull request, does this mean this PR is not going to be merged? Or will there be a new PR?

Priyantha commented Feb 11, 2018

@blogic I see you have closed this pull request, does this mean this PR is not going to be merged? Or will there be a new PR?

@blogic

This comment has been minimized.

Show comment
Hide comment
@blogic

blogic Feb 11, 2018

Contributor

@Priyantha read the comments ... "With the remerge in progress, all PRs on the lede-project organisation will be closed. Please help getting this merged or rebase/post it on the openwrt project page (https://github.com/openwrt/openwrt/pulls). All remaining PRs will be closed in 30 days."

Contributor

blogic commented Feb 11, 2018

@Priyantha read the comments ... "With the remerge in progress, all PRs on the lede-project organisation will be closed. Please help getting this merged or rebase/post it on the openwrt project page (https://github.com/openwrt/openwrt/pulls). All remaining PRs will be closed in 30 days."

@jow-

This comment has been minimized.

Show comment
Hide comment
@jow-

jow- Feb 11, 2018

Member

This particular change however is unlikely to land in OpenWrt as the project opts for the upstream fast path implementation instead.

Member

jow- commented Feb 11, 2018

This particular change however is unlikely to land in OpenWrt as the project opts for the upstream fast path implementation instead.

@Priyantha

This comment has been minimized.

Show comment
Hide comment
@Priyantha

Priyantha Feb 11, 2018

Ah okay, there is an upstream fast path implementation? I wasn't aware of that 👍 when there is one, that would be great. It's a big improvement. So I am hoping it will be part of the new OpenWRT trunk in the near future.

Priyantha commented Feb 11, 2018

Ah okay, there is an upstream fast path implementation? I wasn't aware of that 👍 when there is one, that would be great. It's a big improvement. So I am hoping it will be part of the new OpenWRT trunk in the near future.

@LipkeGu

This comment has been minimized.

Show comment
Hide comment
@LipkeGu

LipkeGu Feb 15, 2018

Shortcut is an in-Linux-kernel IP packet forwarding engine. It's designed
to offer very high speed IP packet forwarding based on IP connection tracking.
It's dramatically faster than the standard netfilter-based NAT forwarding path
but is designed to synchronise state back to netfilter/conntrack so that it
doesn't need to deal with all of the complexities of special cases.

LipkeGu commented on package/kernel/shortcut-fe/Makefile in b87b473 Feb 15, 2018

Shortcut is an in-Linux-kernel IP packet forwarding engine. It's designed
to offer very high speed IP packet forwarding based on IP connection tracking.
It's dramatically faster than the standard netfilter-based NAT forwarding path
but is designed to synchronise state back to netfilter/conntrack so that it
doesn't need to deal with all of the complexities of special cases.

@nbd168

This comment has been minimized.

Show comment
Hide comment
@nbd168

nbd168 Feb 20, 2018

Contributor

FYI, I've backported the upstream flow offload code to Linux 4.14 in my staging tree. I've also written an extra patch that integrates it with iptables (so no dependency on nftables anymore).
You can test it by building my staging tree, enabling kmod-ipt-flow, and adding a rule to the FORWARD chain with -j FLOWOFFLOAD

Contributor

nbd168 commented Feb 20, 2018

FYI, I've backported the upstream flow offload code to Linux 4.14 in my staging tree. I've also written an extra patch that integrates it with iptables (so no dependency on nftables anymore).
You can test it by building my staging tree, enabling kmod-ipt-flow, and adding a rule to the FORWARD chain with -j FLOWOFFLOAD

@lucize

This comment has been minimized.

Show comment
Hide comment
@lucize

lucize Feb 22, 2018

Contributor

@nbd168 sorry for asking, can you give an example to see how it would need to be ?

Contributor

lucize commented Feb 22, 2018

@nbd168 sorry for asking, can you give an example to see how it would need to be ?

@Mushoz

This comment has been minimized.

Show comment
Hide comment
@Mushoz

Mushoz Feb 22, 2018

Contributor

I cannot find the mentioned kmod-ipt-flow in menuconfig. I do find these 3 modules, however:

kmod-ipt-offload
kmod-nf-flow
kmod-nft-offload

Which ones do I need?

Edit: Felix clarified on the LEDE forum. You only need kmod-ipt-offload which will automatically select kmod-nf-flow as well. kmod-nft-offload is not needed.

Contributor

Mushoz commented Feb 22, 2018

I cannot find the mentioned kmod-ipt-flow in menuconfig. I do find these 3 modules, however:

kmod-ipt-offload
kmod-nf-flow
kmod-nft-offload

Which ones do I need?

Edit: Felix clarified on the LEDE forum. You only need kmod-ipt-offload which will automatically select kmod-nf-flow as well. kmod-nft-offload is not needed.

@pexcn pexcn referenced this pull request Feb 25, 2018

Open

Hack OpenWrt #1

@LuKePicci

This comment has been minimized.

Show comment
Hide comment
@LuKePicci

LuKePicci Mar 25, 2018

Just for the records, the SFE code in this PR had also issues over IPSec VPNs. The issue troubleshooting is simple: as soon as a connection passing through the tunnel is picked for shortcutting it breaks, so you see the TCP layer often doing retransmission and, for instance, relatively large web pages load only partially.

LuKePicci commented Mar 25, 2018

Just for the records, the SFE code in this PR had also issues over IPSec VPNs. The issue troubleshooting is simple: as soon as a connection passing through the tunnel is picked for shortcutting it breaks, so you see the TCP layer often doing retransmission and, for instance, relatively large web pages load only partially.

kuoruan added a commit to kuoruan/lede that referenced this pull request Apr 22, 2018

pexcn added a commit to openwrt-dev/openwrt that referenced this pull request Sep 23, 2018

package: add Shortcut Forwarding Engine driver
Reference: lede-project/source#1269

Add an opensource Qualcomm Shortcut FE driver.

The driver consists of:
- sfe (core modules): shortcut-fe (ipv4) and shortcut-fe-ipv6 -
modules that actually do the offloading
- shortcut-fe-cm - simple shortcut-fe connection manager - a
module that creates offloading rules and passes to sfe
- fast-classifier - advanced connection manager - a module that
creates offloading rules and passes to sfe. Comparing to sfe-cm
has additional checks, connection statistics and creates an
offloading rule only when a connection hits 128 packets flowed
(can be adjusted through /sys/fast_classifier/offload_at_pkts)
- userspace-fast-classifier - a fast-classifier user-space tool
to manually add connections one want to offload
Both sfe-cm and fast-classifier get information about connections
through netfilter. And after some checks it creates an offloading
rule that is passed to sfe. After that sfe marks a connection as
fast-forwarded and routes the flow.

Kernel parses this mark and bypasses its network stack taps for
a specific connection in favor of sfe.

There are 3 patches added with this commit:
950 - adds a hook into network stack, sets no tcp window check
to 0 (according to sfe readme).
951 - because fast-classifier also supports bridge offloading
(echo 1 > /sys/fast_classifier/skip_to_bridge_ingress) and
offloaded connections do not naturally update bridge statistics,
fast-classifier will do that.
952 - allow multiple listeners for conntrack events, it is
requied for netlink and sfe coexistance (for example)

All these patches are filled with ifdefs so the sfe code path
is engaged only when these modules are compiled. It allows to
keep kernel code untouched in case of no sfe compiled in.

Because sfe-cm and fast-classifier are competing modules, It's
recommended to keep loaded only one of them. If both are loaded
then in a race condition sfe-cm always comes first.

The driver is a merger of
https://source.codeaurora.org/quic/qsdk/oss/lklm/shortcut-fe/log/?h=release/endive_cc
with
https://chromium.googlesource.com/chromiumos/third_party/kernel/+/chromeos-3.18/drivers/net/ethernet/qualcomm/sfe/

I've also added missing function to update udp statistics by
fast-classifier and some more small fixes.

Patches are taken from QSDK and that chromeos-3.18 repo and a bit
adjusted.

Signed-off-by: Pavel Kubelun <be.dissent@gmail.com>

pexcn added a commit to openwrt-dev/openwrt that referenced this pull request Sep 23, 2018

package: add Shortcut Forwarding Engine driver
Reference: lede-project/source#1269

Add an opensource Qualcomm Shortcut FE driver.

The driver consists of:
- sfe (core modules): shortcut-fe (ipv4) and shortcut-fe-ipv6 -
modules that actually do the offloading
- shortcut-fe-cm - simple shortcut-fe connection manager - a
module that creates offloading rules and passes to sfe
- fast-classifier - advanced connection manager - a module that
creates offloading rules and passes to sfe. Comparing to sfe-cm
has additional checks, connection statistics and creates an
offloading rule only when a connection hits 128 packets flowed
(can be adjusted through /sys/fast_classifier/offload_at_pkts)
- userspace-fast-classifier - a fast-classifier user-space tool
to manually add connections one want to offload
Both sfe-cm and fast-classifier get information about connections
through netfilter. And after some checks it creates an offloading
rule that is passed to sfe. After that sfe marks a connection as
fast-forwarded and routes the flow.

Kernel parses this mark and bypasses its network stack taps for
a specific connection in favor of sfe.

There are 3 patches added with this commit:
950 - adds a hook into network stack, sets no tcp window check
to 0 (according to sfe readme).
951 - because fast-classifier also supports bridge offloading
(echo 1 > /sys/fast_classifier/skip_to_bridge_ingress) and
offloaded connections do not naturally update bridge statistics,
fast-classifier will do that.
952 - allow multiple listeners for conntrack events, it is
requied for netlink and sfe coexistance (for example)

All these patches are filled with ifdefs so the sfe code path
is engaged only when these modules are compiled. It allows to
keep kernel code untouched in case of no sfe compiled in.

Because sfe-cm and fast-classifier are competing modules, It's
recommended to keep loaded only one of them. If both are loaded
then in a race condition sfe-cm always comes first.

The driver is a merger of
https://source.codeaurora.org/quic/qsdk/oss/lklm/shortcut-fe/log/?h=release/endive_cc
with
https://chromium.googlesource.com/chromiumos/third_party/kernel/+/chromeos-3.18/drivers/net/ethernet/qualcomm/sfe/

I've also added missing function to update udp statistics by
fast-classifier and some more small fixes.

Patches are taken from QSDK and that chromeos-3.18 repo and a bit
adjusted.

Signed-off-by: Pavel Kubelun <be.dissent@gmail.com>

pexcn added a commit to openwrt-dev/openwrt that referenced this pull request Sep 23, 2018

package: add Shortcut Forwarding Engine driver
Reference: lede-project/source#1269

Add an opensource Qualcomm Shortcut FE driver.

The driver consists of:
- sfe (core modules): shortcut-fe (ipv4) and shortcut-fe-ipv6 -
modules that actually do the offloading
- shortcut-fe-cm - simple shortcut-fe connection manager - a
module that creates offloading rules and passes to sfe
- fast-classifier - advanced connection manager - a module that
creates offloading rules and passes to sfe. Comparing to sfe-cm
has additional checks, connection statistics and creates an
offloading rule only when a connection hits 128 packets flowed
(can be adjusted through /sys/fast_classifier/offload_at_pkts)
- userspace-fast-classifier - a fast-classifier user-space tool
to manually add connections one want to offload
Both sfe-cm and fast-classifier get information about connections
through netfilter. And after some checks it creates an offloading
rule that is passed to sfe. After that sfe marks a connection as
fast-forwarded and routes the flow.

Kernel parses this mark and bypasses its network stack taps for
a specific connection in favor of sfe.

There are 3 patches added with this commit:
950 - adds a hook into network stack, sets no tcp window check
to 0 (according to sfe readme).
951 - because fast-classifier also supports bridge offloading
(echo 1 > /sys/fast_classifier/skip_to_bridge_ingress) and
offloaded connections do not naturally update bridge statistics,
fast-classifier will do that.
952 - allow multiple listeners for conntrack events, it is
requied for netlink and sfe coexistance (for example)

All these patches are filled with ifdefs so the sfe code path
is engaged only when these modules are compiled. It allows to
keep kernel code untouched in case of no sfe compiled in.

Because sfe-cm and fast-classifier are competing modules, It's
recommended to keep loaded only one of them. If both are loaded
then in a race condition sfe-cm always comes first.

The driver is a merger of
https://source.codeaurora.org/quic/qsdk/oss/lklm/shortcut-fe/log/?h=release/endive_cc
with
https://chromium.googlesource.com/chromiumos/third_party/kernel/+/chromeos-3.18/drivers/net/ethernet/qualcomm/sfe/

I've also added missing function to update udp statistics by
fast-classifier and some more small fixes.

Patches are taken from QSDK and that chromeos-3.18 repo and a bit
adjusted.

Signed-off-by: Pavel Kubelun <be.dissent@gmail.com>

pexcn added a commit to openwrt-dev/openwrt that referenced this pull request Sep 23, 2018

package: add Shortcut Forwarding Engine driver
Reference: lede-project/source#1269

Add an opensource Qualcomm Shortcut FE driver.

The driver consists of:
- sfe (core modules): shortcut-fe (ipv4) and shortcut-fe-ipv6 -
modules that actually do the offloading
- shortcut-fe-cm - simple shortcut-fe connection manager - a
module that creates offloading rules and passes to sfe
- fast-classifier - advanced connection manager - a module that
creates offloading rules and passes to sfe. Comparing to sfe-cm
has additional checks, connection statistics and creates an
offloading rule only when a connection hits 128 packets flowed
(can be adjusted through /sys/fast_classifier/offload_at_pkts)
- userspace-fast-classifier - a fast-classifier user-space tool
to manually add connections one want to offload
Both sfe-cm and fast-classifier get information about connections
through netfilter. And after some checks it creates an offloading
rule that is passed to sfe. After that sfe marks a connection as
fast-forwarded and routes the flow.

Kernel parses this mark and bypasses its network stack taps for
a specific connection in favor of sfe.

There are 3 patches added with this commit:
950 - adds a hook into network stack, sets no tcp window check
to 0 (according to sfe readme).
951 - because fast-classifier also supports bridge offloading
(echo 1 > /sys/fast_classifier/skip_to_bridge_ingress) and
offloaded connections do not naturally update bridge statistics,
fast-classifier will do that.
952 - allow multiple listeners for conntrack events, it is
requied for netlink and sfe coexistance (for example)

All these patches are filled with ifdefs so the sfe code path
is engaged only when these modules are compiled. It allows to
keep kernel code untouched in case of no sfe compiled in.

Because sfe-cm and fast-classifier are competing modules, It's
recommended to keep loaded only one of them. If both are loaded
then in a race condition sfe-cm always comes first.

The driver is a merger of
https://source.codeaurora.org/quic/qsdk/oss/lklm/shortcut-fe/log/?h=release/endive_cc
with
https://chromium.googlesource.com/chromiumos/third_party/kernel/+/chromeos-3.18/drivers/net/ethernet/qualcomm/sfe/

I've also added missing function to update udp statistics by
fast-classifier and some more small fixes.

Patches are taken from QSDK and that chromeos-3.18 repo and a bit
adjusted.

Signed-off-by: Pavel Kubelun <be.dissent@gmail.com>

pexcn added a commit to openwrt-dev/openwrt that referenced this pull request Sep 23, 2018

package: add Shortcut Forwarding Engine driver
Reference: lede-project/source#1269

Add an opensource Qualcomm Shortcut FE driver.

The driver consists of:
- sfe (core modules): shortcut-fe (ipv4) and shortcut-fe-ipv6 -
modules that actually do the offloading
- shortcut-fe-cm - simple shortcut-fe connection manager - a
module that creates offloading rules and passes to sfe
- fast-classifier - advanced connection manager - a module that
creates offloading rules and passes to sfe. Comparing to sfe-cm
has additional checks, connection statistics and creates an
offloading rule only when a connection hits 128 packets flowed
(can be adjusted through /sys/fast_classifier/offload_at_pkts)
- userspace-fast-classifier - a fast-classifier user-space tool
to manually add connections one want to offload
Both sfe-cm and fast-classifier get information about connections
through netfilter. And after some checks it creates an offloading
rule that is passed to sfe. After that sfe marks a connection as
fast-forwarded and routes the flow.

Kernel parses this mark and bypasses its network stack taps for
a specific connection in favor of sfe.

There are 3 patches added with this commit:
950 - adds a hook into network stack, sets no tcp window check
to 0 (according to sfe readme).
951 - because fast-classifier also supports bridge offloading
(echo 1 > /sys/fast_classifier/skip_to_bridge_ingress) and
offloaded connections do not naturally update bridge statistics,
fast-classifier will do that.
952 - allow multiple listeners for conntrack events, it is
requied for netlink and sfe coexistance (for example)

All these patches are filled with ifdefs so the sfe code path
is engaged only when these modules are compiled. It allows to
keep kernel code untouched in case of no sfe compiled in.

Because sfe-cm and fast-classifier are competing modules, It's
recommended to keep loaded only one of them. If both are loaded
then in a race condition sfe-cm always comes first.

The driver is a merger of
https://source.codeaurora.org/quic/qsdk/oss/lklm/shortcut-fe/log/?h=release/endive_cc
with
https://chromium.googlesource.com/chromiumos/third_party/kernel/+/chromeos-3.18/drivers/net/ethernet/qualcomm/sfe/

I've also added missing function to update udp statistics by
fast-classifier and some more small fixes.

Patches are taken from QSDK and that chromeos-3.18 repo and a bit
adjusted.

Signed-off-by: Pavel Kubelun <be.dissent@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment