multi Interface shaping using queues leads to halved bandwidth

[marunjar]

Important notices

Before you add a new report, we ask you kindly to acknowledge the following:

• I have read the contributing guide lines at https://github.com/opnsense/core/blob/master/CONTRIBUTING.md
• I am convinced that my issue is new after having checked both open and closed issues at https://github.com/opnsense/core/issues?q=is%3Aissue

Describe the bug

I was trying to limit bandwith networks using pipes with given bandwidth and also some queues so bandwith is shared by weight between different clients.
I tried to defined rules using 2 interfaces, so they work independent of network as ip ranges are meant to change and also ipv6 should be added in near future.
When doing this, bandwith for clients is only halve of the bandwith as defined in pipes.

To Reproduce

Steps to reproduce the behavior, for detailed settings look at screenshots:

1. create 2 new pipes with given bandwith (for down- and upload)
2. create 2 new queues with weight targeting the pipes
3. create 2 new rules for download and upload targeting the queues, that matches 2 interfaces (WAN, GUEST)

All in all setup is similar to https://docs.opnsense.org/manual/how-tos/shaper_guestnet.html, but with queues.

Then test download/upload bandwith for a single client behind GUEST interface with iperf3 or simple speed test.
One can observe, that only half of bandwith of pipes from step 1 is available.

Expected behavior

Bandwith should not be halved, if there is only one client in GUEST net, it should get full bandwith given by pipe.

Describe alternatives you considered

As a workaround rules based on source and destination may be possible, but these networks are meant to change.

Screenshots
pipe up

pipe down

queue up

queue down

rule up

rule down

Additional context

output of ipfw show lists these rules:

60009     0       0 queue 10002 ip from any to any recv igb0 xmit igb1_vlan20 // da1cda49-cfd1-44a5-9509-8381855800ba wan -> opt4: LOW Prio Down Queue
60011     0       0 queue 10006 ip from any to any recv igb1_vlan20 xmit igb0 // 21372daf-d801-4fe6-89c2-3286b60cd58c opt4 -> wan: LOW Prio Up Queue

According to https://www.freebsd.org/cgi/man.cgi?ipfw(8) this should apply:

The recv interface can be tested on either incoming or outgoing
packets, while the xmit interface can only be tested on outgoing
packets. So out is required (and in is invalid) whenever xmit is
used.

We can use pipes to artificially limit bandwidth, e.g. on a machine act-
ing as a router, if we want to limit traffic from local clients on
192.168.2.0/24 we do:

ipfw add pipe 1 ip from 192.168.2.0/24 to any out
ipfw pipe 1 config bw 300Kbit/s queue 50KBytes

note that we use the out modifier so that the rule is not used twice.
Remember in fact that ipfw rules are checked both on incoming and outgo-
ing packets.

It looks like the out modifier is missing so packets are processed twice which leads to halved bandwidth because of using queues.
Also changing direction of rule doesn't add any modifier, but only changes interfaces within the rule.
So maybe it's enough to simply add out modifier after destination in rule if second interface is selected.

Relates to https://forum.opnsense.org/index.php?topic=26913.0

Environment

Software version used and hardware type if relevant, e.g.:

OPNsense 22.1 on X11SBA-LN4F (https://www.supermicro.com/en/products/motherboard/X11SBA-LN4F)

[fichtner]

It's sort of the same as https://docs.opnsense.org/manual/how-tos/shaper_limit_per_user.html#limit-maximum-internet-bandwidth-users-can-consume but without the masks set in pipes since you don't need user-based limits. What stands out is that you use bi-directional rules which don't work as it should and the correct way to get the limit is to set source / destination for each rule as the documentation lays out.

Cheers,
Franco

[marunjar]

Thanks for fast reply.

I have to add something here, didn't think this may be relevant, but i'm using queues so i can add rules for my "internet of things" network later. So bandwith should be shared between users, depending on weight.

I was thinking about using source and destination for my use case too, but unfortunately i wasn't able to select e.g. "GUEST net" like it's possible for firewall rules.
I can add "GUEST net" for source/destination but then i get only "please specify a valid network segment or address (IPv4/IPv6)"
So if network changes i have to adapt these values manually which i want to avoid.

So i tried to adapt https://docs.opnsense.org/manual/how-tos/shaper_guestnet.html which seems to work, except the halved bandwith :(

Also these rules doesn't seem to be bidirectional to me:
Is this because of missing in/out modifier or something else?
I also tried to change direction of rule (where only both means bidirectional to me) but this option seems to work different as soon as second interface is selected because then in/out isn't added to rules any more.

When i'm testing download, packet counter for one rule is increasing as expected and during upload the other rule matches.

[AdSchellevis]

It might be best to fiddle a bit with the rules and see which change you need, maybe we miss something, maybe we don't, I'm not 100% sure to be honest.

The rules are stored in /usr/local/etc/ipfw.rules and can be reloaded/flished using ipfw -f /usr/local/etc/ipfw.rules

[marunjar]

Thanks for this hint, i'll try to change ipfw.rules directly on weekend.
Will this file be reverted to my config if i apply in gui? Just in case i break something.

[fichtner]

Will this file be reverted to my config if i apply in gui?

Yes.

[marunjar]

I was able to find the real problem.
My rule using 2 interfaces somehow clashed with an already existing "default queue" from previous tries.
My bad, i blamed the new rule only for this problem.

It is no good idea to mix up rules matching interfaces (without src/destination) and rules matching addresses only.
It was already working with src/dst address before, but when i changed the rule to match interfaces there are suddenly multiple rules matching:

60001    13     2833 queue 10000 ip from any to any recv igb0 xmit igb1_vlan10 // 47a9f075-3071-485a-af77-10bf40b890bf wan -> opt3: HIGH Prio Down Queue
60002    13     2387 queue 10004 ip from any to any xmit igb0 recv igb1_vlan10 // 6fd0200e-67ec-4cd3-8e28-48b09b2b7295 opt3 -> wan: HIGH Prio Up Queue
60003    26     5798 queue 10003 ip from any to any in via igb0 // a08111ee-8f5a-4d94-996c-3c77f22fb6e2 wan: NO Prio Down Queue
60004    11     3506 queue 10007 ip from any to any out via igb0 // f63b5459-06e8-4e3f-950a-72e7acc652d8 wan: NO Prio Up Queue

So for download both, rule 60001 and 60003 matched and i think because of this, the packet count twice for pipe --> the bandwith is reduced.

Though, the example here is wrong https://docs.opnsense.org/manual/how-tos/shaper_guestnet.html
For upload/download either the interfaces must be swapped or the direction must be changed.
If both, direction and interfaces, are changed this will result in same rule, e.g.

queue 10000 ip from any to any recv igb0 xmit igb1_vlan10
queue 10004 ip from any to any xmit igb1_vlan10 recv igb0

So only traffic in one direction is shaped when following current example.

[AdSchellevis]

looks like we accepted the wrong fix last year opnsense/docs@680af4f, let me change the docs and close this ticket.