Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

config_aqm Unable to configure flowset, flowset busy! #1279

Closed
KyleSanderson opened this issue Nov 26, 2016 · 16 comments
Closed

config_aqm Unable to configure flowset, flowset busy! #1279

KyleSanderson opened this issue Nov 26, 2016 · 16 comments

Comments

@KyleSanderson
Copy link

pflog0: promiscuous mode enabled
ipfw2 (+ipv6) initialized, divert loadable, nat loadable, default to accept, logging disabled
DUMMYNET 0 with IPv6 initialized (100409)
load_dn_aqm dn_aqm PIE loaded
load_dn_aqm dn_aqm CODEL loaded
load_dn_sched dn_sched FIFO loaded
load_dn_sched dn_sched QFQ loaded
load_dn_sched dn_sched RR loaded
load_dn_sched dn_sched WF2Q+ loaded
load_dn_sched dn_sched PRIO loaded
load_dn_sched dn_sched FQ_CODEL loaded
load_dn_sched dn_sched FQ_PIE loaded
ovpnc1: link state changed to UP
gif0: link state changed to DOWN
gif0: link state changed to UP
gif0: link state changed to DOWN
gif0: link state changed to UP
gif0: link state changed to DOWN
gif0: link state changed to UP
ovpnc1: link state changed to DOWN
ovpnc1: link state changed to UP
pid 27327 (python2.7), uid 0, was killed: out of swap space
Bump sched buckets to 256 (was 0)
Bump sched buckets to 256 (was 0)
config_aqm Unable to configure flowset, flowset busy!
config_aqm Unable to configure flowset, flowset busy!
config_aqm Unable to configure flowset, flowset busy!
config_aqm Unable to configure flowset, flowset busy!
config_aqm Unable to configure flowset, flowset busy!
config_aqm Unable to configure flowset, flowset busy!
config_aqm Unable to configure flowset, flowset busy!
Bump sched buckets to 256 (was 0)
Bump sched buckets to 256 (was 0)
config_aqm Unable to configure flowset, flowset busy!
config_aqm Unable to configure flowset, flowset busy!
config_aqm Unable to configure flowset, flowset busy!
config_aqm Unable to configure flowset, flowset busy!
config_aqm Unable to configure flowset, flowset busy!
config_aqm Unable to configure flowset, flowset busy!
config_aqm Unable to configure flowset, flowset busy!
Bump sched buckets to 256 (was 0)
Bump sched buckets to 256 (was 0)
config_aqm Unable to configure flowset, flowset busy!
config_aqm Unable to configure flowset, flowset busy!
config_aqm Unable to configure flowset, flowset busy!
config_aqm Unable to configure flowset, flowset busy!
config_aqm Unable to configure flowset, flowset busy!
config_aqm Unable to configure flowset, flowset busy!
config_aqm Unable to configure flowset, flowset busy!
root@OPNsense:/var/netflow #
@AdSchellevis
Copy link
Member

@KyleSanderson please read https://github.com/opnsense/core/blob/master/CONTRIBUTING.md Without background of the specific setup, features that are used and a way to reproduce the issue its highly unlikely that anyone will try to help out.

@fichtner
Copy link
Member 

# pid 27327 (python2.7), uid 0, was killed: out of swap space

This is weird and likely the issue: RAM is depleted? Do you have swap space? And how much memory is consumed in general? What hardware specs are this?

@KyleSanderson
Copy link
Author

4G. flowd sqlite DBs consumed 2G... rm'd them.

@fichtner
Copy link
Member

CC'ing @RasoolAlSaadi he wrote the AQM code.

@RasoolAlSaadi
Copy link

@KyleSanderson
I guess you are trying to use CoDel/PIE (not FQ_CoDel/FQ_PIE), is that right?
It seems that the pipe/queue is currently busy (a traffic is passing through pipe/queue ipfw rule) while you are trying to reconfigure AQM.
Due to the fact that the internal variables of each AQM instance is allocated and freed dynamically, re-configuring AQM (e.g. select CoDel for a pipe that currently uses PIE) can lead the AQM code to access unallocated memory (freed during AQM re-configuration) and kernel panic could happen. Therefore, I tried to avoid any potential kernel panic by preventing re-configuring AQM for busy pipe/queue.

Try one of the following solutions:

  • Temporarily block the traffic (using IPFW rules) from passing through AQM PIE/queue until you finish your configuration.
  • Delete the old pipes/queues that need to be re-configured with AQM and create them again with all the desired settings.
  • Reboot the firewall

If the above solutions do not work, please provide me more details about in which cases the error message appears so I can provide you a suitable solution.

Regards,
Rasool

@KyleSanderson
Copy link
Author

Thanks for following up @RasoolAlSaadi

I'm trying to use FQ_Codel on OPNsense however this is implemented via a checkbox so I'm not certain if it's actually applying?

My status is the following, however considering there's 0 flows I'd assume this is not working at all?

Limiters:
10000: 150.000 Mbit/s    0 ms burst 0 
q75536  50 sl. 0 flows (1 buckets) sched 10000 weight 0 lmax 0 pri 0 droptail
 sched 75536 type FIFO flags 0x1 256 buckets 0 active
    mask:  0x00 0x00000000/0x0000 -> 0xffffffff/0x0000
10001: 150.000 Mbit/s    0 ms burst 0 
q75537  50 sl. 0 flows (1 buckets) sched 10001 weight 0 lmax 0 pri 0 droptail
 sched 75537 type FIFO flags 0x1 256 buckets 0 active
    mask:  0x00 0x00000000/0x0000 -> 0xffffffff/0x0000


Queues:
q10006  50 sl. 0 flows (1 buckets) sched 10000 weight 60 lmax 0 pri 0 
q10007  50 sl. 0 flows (1 buckets) sched 10001 weight 60 lmax 0 pri 0 
q10005  50 sl. 0 flows (1 buckets) sched 10001 weight 20 lmax 0 pri 0 
q10002  50 sl. 0 flows (1 buckets) sched 10001 weight 5 lmax 0 pri 0 
q10003  50 sl. 0 flows (1 buckets) sched 10000 weight 5 lmax 0 pri 0 
q10000  50 sl. 0 flows (1 buckets) sched 10000 weight 10 lmax 0 pri 0 
q10001  50 sl. 0 flows (1 buckets) sched 10001 weight 10 lmax 0 pri 0

I have two 150 Mbit pipes
Queues: https://i.imgur.com/GPs12mY.png
Rules: https://i.imgur.com/ra1mdfD.png

I don't want to say (accuse, rather) it isn't working at all, however performing a speedtest exceeds the pipe; so I'm not all too convinced traffic shaping is actually working on the platform... Thoughts?

@RasoolAlSaadi
Copy link

@KyleSanderson
To be honest, I don't have any experience with OPNsense. However, the traffic shaping should work fine if the traffic passes through the limiter.
I don't see any output related to AQM in the status you posted. So, I hope @AdSchellevis or @fichtner can help you to configure the firewall with AQM as they have good experience with that.

If "config_aqm Unable to configure flowset, flowset busy!" still appears, please provide me a way to reproduce the issue so I can see what happens.

@KyleSanderson
Copy link
Author

@fichtner any update on enabling FQ-PIE in the GUI? It looks like support was merged 6(!) months ago.

@Animosity022
Copy link

Not sure if it is helpful or not, but I got the same errors when I tried to apply.

I got pretty mixed results when testing afterwards as it wasn't quite working as I had expected, but it did do something.

@Animosity022
Copy link

Specifically, if I can configure a PIPE with no mask, I get no errors.

If I configure a "Upload" Pipe Source Masked and a "Download" Pipe Destination Masked and hit apply, I get the flowset error.

@fichtner
Copy link
Member

The only clean way here seems to not reuse pipes or unload IPFW, but considering the low frequency of reports and no actual problems we're closing this. A special thanks to @RasoolAlSaadi for the authors explanation of how AQM works and what the error message means exactly. :)

@KyleSanderson
Copy link
Author

@fichtner I appreciate you're cleaning up issues, but the bug report is this doesn't work at all. Was this actually fixed? I had to switch to LEDE many moons ago for hw offloading with AQM, but I'm not certain this changed?

@fichtner
Copy link
Member

I'm reading raw ipfw dump, flowd bug (not related), then report of shaping not working, then feature request for fq-pie. It's hard to follow... I went by title, expecting the intention there was clear.

17.1 added "shared forwarding option", because up until 16.7 the policy routing (firewall rules with gateway) would blackhole the traffic and not provide them to ipfw/aqm. I could have likely been your issue.

@Animosity022
Copy link

I've played around quite a bit and I can conclude from my previous report, it's working fine. I haven't been able to reproduce the error messages from quite some time.

I can see traffic matching my proper queues and it's definitely working:

root@phoenix:~ # ipfw -a list
00100   0     0 allow pfsync from any to any
00110   0     0 allow carp from any to any
00120   0     0 allow ip from any to any layer2 mac-type 0x0806,0x8035
00130   0     0 allow ip from any to any layer2 mac-type 0x888e,0x88c7
00140   0     0 allow ip from any to any layer2 mac-type 0x8863,0x8864
00150   0     0 deny ip from any to any layer2 not mac-type 0x0800,0x86dd
00200   0     0 skipto 60000 ip6 from ::1 to any
00201   0     0 skipto 60000 ip4 from 127.0.0.0/8 to any
00202   0     0 skipto 60000 ip6 from any to ::1
00203   0     0 skipto 60000 ip4 from any to 127.0.0.0/8
01002   0     0 skipto 60000 udp from any to 192.168.1.1 dst-port 53 keep-state
01002  85  7194 skipto 60000 ip from any to { 255.255.255.255 or 192.168.1.1 } in
01002 138 34499 skipto 60000 ip from { 255.255.255.255 or 192.168.1.1 } to any out
01002   0     0 skipto 60000 icmp from { 255.255.255.255 or 192.168.1.1 } to any out icmptypes 0
01002   0     0 skipto 60000 icmp from any to { 255.255.255.255 or 192.168.1.1 } in icmptypes 8
06000  81  8897 skipto 60000 tcp from any to any out
06199 166 15550 skipto 60000 ip from any to any
30000   0     0 count ip from any to any
60000   0     0 return ip from any to any
60001   2   154 queue 10000 ip from any to 192.168.1.90 in via igb0
60002   1   121 queue 10000 ip from any to 192.168.1.55 in via igb0
60003   0     0 queue 10000 ip from any to 192.168.1.50 in via igb0
60004   0     0 queue 10000 ip from any to 192.168.1.51 in via igb0
60005   3   231 queue 10003 ip from 192.168.1.90 to any out via igb0
60006   2   173 queue 10003 ip from 192.168.1.55 to any out via igb0
60007   0     0 queue 10003 ip from 192.168.1.50 to any out via igb0
60008   0     0 queue 10003 ip from 192.168.1.51 to any out via igb0
60009   1    52 queue 10002 ip from 192.99.13.124 to 192.168.1.30 in via igb0
60010   0     0 queue 10002 ip from any to 192.168.1.30 src-port 563 in via igb0
60011   1    81 queue 10005 ip from 192.168.1.30 to 192.99.13.124 out via igb0
60012   0     0 queue 10005 ip from 192.168.1.30 to any dst-port 563 out via igb0
60013  71  5688 queue 10001 ip from any to any in via igb0
60014  63  6691 queue 10004 ip from any to any out via igb0
65533 326 52949 allow ip from any to any
65534   0     0 deny ip from any to any
65535   7   426 allow ip from any to any

I'm basically configured 2 a pipe for upload/download/a high, medium, low queue for up/down and than rules that match "in" traffic and "out" traffic to drop them in the respective queues with FQ_Codel setup at the pipes:

http://www.dslreports.com/speedtest/18260817

@fichtner
Copy link
Member

@animosity22 thanks for the update :)

@NMartin354
Copy link

Can we run like "add 59999 skipto 70000 ip from any to any" before running /usr/local/etc/ipfw.rules and pull it out once it's completed?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Milestone
No milestone
Development

No branches or pull requests
6 participants
@KyleSanderson @fichtner @NMartin354 @Animosity022 @AdSchellevis @RasoolAlSaadi