Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Server freezing #50

Open
melicherm opened this issue Feb 27, 2017 · 16 comments
Open

Server freezing #50

melicherm opened this issue Feb 27, 2017 · 16 comments

Comments

@melicherm
Copy link

melicherm commented Feb 27, 2017 •
edited
Loading

Hi Guys,
got 2 servers in the role of an firewall (mainly FORWARD chain filtering)

server 1:
debian jessie 8.6 - kernel 3.16.0-4-amd64

server 2:
debian stretch 9.0 - kernel 4.8.0-1-amd64

The server 1 is an production server, server 2 is for testing purposes.

The thing is, both the servers insta freez (sometimes within 5min, sometimes it lasts some hours -> applying NDPI rules on 20:00 server dead on 04:00) in random intervals (mainly the production one, where the traffic is 200Mbit/s 5min average to 500Mbit/s 5min average). Through the production server goes the traffic of around 500 - 700 client PC's.

1Gbit uplink

The server 1 has much more complicated iptables rules ( ca 1170 in the FORWARD, PREROUTING chain)..., but the server 2 has only few, so the debug is here possible (37 rules in the FORWARD chain, no PREROUTING, some INPUT chain rules just for security reasons).

The test server is much more stable, because the minimal traffic, but freezes are also happening.

If i apply the dpi_check rules:
iptables -t mangle -A PREROUTING -m ndpi --dpi_check
iptables -t mangle -A POSTROUTING -m ndpi --dpi_check

the servers are insta dead.... nothing in the console, the server is unresponsive, nothing on the screen, nothing in the logs. The only possible thing is to reboot the servers.

Any idea how to debug this behavior? Got the NDPI kernel module build from this GIT.

ii iptables 1.6.0+snapshot20161117-5
ii iptables-dev 1.6.0+snapshot20161117-4
ii iptables-persistent 1.0.4+nmu1
ii xtables-addons-common 2.11-3
ii xtables-addons-dkms 2.11-3

the rules are:

-A FORWARD -m ndpi --applejuice --directconnect --gnutella --edonkey --bittorrent --soulseek -m comment --comment "ndpi checker - dropper" -j DROP

The loopback is enabled:
-A INPUT -i lo -j ACCEPT
@rightkick
Copy link

rightkick commented Feb 27, 2017 via email

@melicherm
Copy link
Author

melicherm commented Feb 28, 2017 •
edited
Loading

Going to try it out.

rigth now:

iptables -t filter -S
-P INPUT ACCEPT
-P FORWARD ACCEPT
-P OUTPUT ACCEPT

INPUT (default policy is allow):
-A INPUT -i lo -j ACCEPT
...
--> some allowing rules
...
-A INPUT -j DROP (on the end drop)

FORWARD (default policy is allow):
--> some allowing rules
...
-A FORWARD -j DROP (on the end drop)

OUTPUT (default policy is allow):
nothing here...

iptables -t mangle -S
-P PREROUTING ACCEPT
-P INPUT ACCEPT
-P FORWARD ACCEPT
-P OUTPUT ACCEPT
-P POSTROUTING ACCEPT
-A PREROUTING -i lo -j ACCEPT
-A PREROUTING -m ndpi --dpi_check
-A POSTROUTING -o lo -j ACCEPT
-A POSTROUTING -m ndpi --dpi_check

@melicherm
Copy link
Author

melicherm commented Feb 28, 2017 •
edited
Loading

@rightkick Alex, but the default policy is accept (the whole mangle table)

Also if i just had the ndpi on the production server (just counting the packets like this):

iptables -I FORWARD -m ndpi --applejuice --directconnect --gnutella --edonkey --bittorrent --soulseek -m comment --comment "ndpi checker"

The server also freezes ( the -m ndpi --dpi_check rules are not applied... just this one rule )

@melicherm
Copy link
Author

melicherm commented Feb 28, 2017 •
edited
Loading

Ok, tried right now to speed it up:

added:

iptables -I FORWARD -m ndpi --youtube -j DROP

watched youtube maybe 2 minutes and the server crashes over ipmi the login screen was there, no response to keys, going to check the logs after the reboot...

It's the testing server 2 2 posts up are the iptables rules .... dpi_check applied, also the loopback allowed, but default policy on mangle is allow.

Only droping is happening on the FORWARD and INPUT chain in the filter table.

A.D.: Nothing in the logs

@elico
Copy link

elico commented Feb 28, 2017

@melicherm This is the same issue I was having since I tried using this repo.
I tried to use i but everytime the machine froze so eventually I started other options.

@melicherm
Copy link
Author

@elico - what were your other options? Curious about it.

@elico
Copy link

elico commented Feb 28, 2017

@melicherm zeroshell FW, Ubiquiti EdgeMAX router, and https://github.com/vel21ripn/nDPI.
All of the options are not perfect but I didn't managed to make the server iresponsive.
I have also seen that ClearOS and\or NethServer have nDPI support but yet to try it.

@rightkick
Copy link

rightkick commented Feb 28, 2017 via email

@melicherm
Copy link
Author

What do you mean with that line:

Amend mangle chains to exlude that traffic from ndpi checks. I had encountered similar freezes and was able to resolve them with checking only forwarded traffic.

what i want to have in the -t filter table in the FORWARD chain this rule:
iptables -I FORWARD -m ndpi --applejuice --directconnect --gnutella --edonkey --bittorrent --soulseek -m comment --comment "ndpi checker" -j DROP

That is my only wish :).. without freezes, what are you suggesting. Please be specific.

Thank you,
Markus.

@rightkick
Copy link

rightkick commented Mar 1, 2017 via email

@melicherm
Copy link
Author

melicherm commented Mar 5, 2017 •
edited
Loading

New info...
tried right now:

@rightkick - separating rules like you wrote
-t filter
iptables -A FORWARD -s XXX.XXX.XXX.XXX/21 -j ndpi_filter
iptables -A FORWARD -d XXX.XXX.XXX.XXX/21 -j ndpi_filter

iptables -A FORWARD -s XXX.XXX.XXX.XXX/21 -j ndpi_dropper
iptables -A FORWARD -d XXX.XXX.XXX.XXX/21 -j ndpi_dropper

-A ndpi_dropper -m ndpi --gnutella -j DROP
-A ndpi_dropper -m ndpi --edonkey -j DROP
-A ndpi_dropper -m ndpi --bittorrent -j DROP
-A ndpi_filter -m ndpi --gnutella -j SET --add-set torrents src,dst --timeout 7200
-A ndpi_filter -m ndpi --edonkey -j SET --add-set torrents src,dst --timeout 7200
-A ndpi_filter -m ndpi --bittorrent -j SET --add-set torrents src,dst --timeout 7200

-t mangle
iptables -A PREROUTING -m ndpi --dpi_check
iptables -A POSTROUTING -m ndpi --dpi_check

also specifiying the network in the rules?:

iptables -A PREROUTING -s XXX.XXX.XXX.XXX/21 -m ndpi --dpi_check
iptables -A PREROUTING -d XXX.XXX.XXX.XXX/21 -m ndpi --dpi_check
iptables -A POSTROUTING -s XXX.XXX.XXX.XXX/21 -m ndpi --dpi_check
iptables -A POSTROUTING -d XXX.XXX.XXX.XXX/21 -m ndpi --dpi_check

tried every of this (with DROPing or just counting), also separating the rules like only the FB rule nothing more, only YT, only bittorent, only edonkey, etc...
-A ndpi_dropper -m ndpi --gnutella -j DROP
-A ndpi_dropper -m ndpi --edonkey -j DROP
-A ndpi_dropper -m ndpi --bittorrent -j DROP
-A ndpi_dropper -m ndpi --facebook -j DROP
-A ndpi_dropper -m ndpi --youtube -j DROP
freezes just with facebook && youtube && ndpi_check in mangle

also tried none of the rules, so just the -m ndpi --dpi_check in the mangle table...

Something with the module & kernel somewhere...

the server freezes... @betolj any help from you?

@rightkick
Copy link

rightkick commented Mar 6, 2017 via email

@melicherm
Copy link
Author

melicherm commented Mar 6, 2017 •
edited
Loading

server 1:
debian jessie 8.6 - kernel 3.16.0-4-amd64

server 2: the main tests are done here, the first one is the production one
debian stretch 9.0 - kernel 4.8.0-1-amd64

The server should handle up to 1Gbps traffic, but the 5min average right now is 250Mbps

@rightkick
Copy link

rightkick commented Mar 6, 2017 via email

@melicherm
Copy link
Author

melicherm commented Mar 7, 2017 •
edited
Loading

On the production one i tested:

only filter table, forward chain, 1 rule... no mangle

iptables -I FORWARD -m ndpi --applejuice --directconnect --gnutella --edonkey --bittorrent --soulseek -m comment --comment "ndpi checker"

Just counting, not dropping or anything -> server freezes 3.16 kernel

applied rule on 20.00 server dead on 04:00 ca, after that i don't do any tests on the production one :D

Will try on the test one different kernel then.

@rightkick
Copy link

After testing with persistent high traffic (~ 100Mbit/s) I also encountered kernel panic with 3.18.36.

Using https://github.com/vel21ripn/nDPI/tree/netfilter for 30+hours with same traffic did not panic.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@elico @rightkick @melicherm