-
Notifications
You must be signed in to change notification settings - Fork 69
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
kernel panic and build issue w/ current #8
Comments
You followed the steps in "ndpi.install" file?
|
trying now, FYI, instruction #2 says to pull nDPI from svn..... |
Ok... i update the ndpi.install. Its better. |
And now, the build works? |
yes. this works. I'm only matching a fraction of some protocols though, specifically bittorrent. ntopng seems to match much more. is this to be expected? my test is downloading a torrent on the same box as ndpi-netfilter, here are my iptables rules: |
Try with: 2015-06-16 15:56 GMT-04:00 syadnom notifications@github.com:
|
What does --dpi_check do? here is my output by the way, it's not matching anything. This is with a completely flushed iptables mangle table Chain PREROUTING (policy ACCEPT 155K packets, 218M bytes) Chain INPUT (policy ACCEPT 155K packets, 218M bytes) Chain FORWARD (policy ACCEPT 0 packets, 0 bytes) Chain OUTPUT (policy ACCEPT 77243 packets, 4704K bytes) Chain POSTROUTING (policy ACCEPT 77246 packets, 4704K bytes) |
This rule will not match anything, but force ndpi check all flows for any protocol. |
ok, what can I learn from that? I see no matching etc, or is that fact that it hasn't errored what we care about? when I do something like -m ndpi --http or --ssl DROP, it does work like 90+%. Chrome doesn't immediate fail to connect because something gets through but it definitely is matching that. Bittorrent on the other hand is barely matching (20%?) vs 90%+ on ntopng's interface. |
The dpi_check type doesn't exist internally in nDPI API. When you use this rule, all packets will be analyzed internally and dpi flows will be created too (in any direction). This increases chances of inspection of the flow through the nDPI. Maybe, I can force a valid return for a packet account statistic, but dpi_check is not a protocol. Try only this rule: This will not work because you will haven't the full dpi flow and not "match" in the best moment. And, for ntopng... everything is analized. It doesn't depend on a flow or trigger rule. But, if you add "iptables -t mangle -I POSTROUTING -m ndpi --dpi_check", this rule will drop your conections - because you already have the any output flow mapped. It's a little bit confusing, but the goal was this. I will do more tests and try to improve detection. |
ok, so --dpi_check is to help increase the matches when use with other ndpi So, I start with the then add additional rule like which should make it so that I drop inbound youtube videos.. now, I've done this and I'm only matching a small number of packets and I'm Generated by iptables-save v1.4.21 on Tue Jun 16 17:20:14 2015*mangle Completed on Tue Jun 16 17:20:14 2015Generated by iptables-save v1.4.21 on Tue Jun 16 17:20:14 2015*filter On Tue, Jun 16, 2015 at 4:47 PM, Humberto Jucá notifications@github.com
|
yeah, I'm not matching much at all. Can't drop youtube or netflix. Some iptables: Generated by iptables-save v1.4.21 on Tue Jun 16 17:51:20 2015*mangle Completed on Tue Jun 16 17:51:20 2015Generated by iptables-save v1.4.21 on Tue Jun 16 17:51:20 2015*filter Completed on Tue Jun 16 17:51:20 2015iptables -L Chain INPUT (policy ACCEPT 68198 packets, 73M bytes) Chain FORWARD (policy ACCEPT 0 packets, 0 bytes) Chain OUTPUT (policy ACCEPT 63217 packets, 50M bytes) iptables -L -t mangle Chain PREROUTING (policy ACCEPT 1070K packets, 1120M bytes) Chain INPUT (policy ACCEPT 1070K packets, 1120M bytes) Chain FORWARD (policy ACCEPT 0 packets, 0 bytes) Chain OUTPUT (policy ACCEPT 960K packets, 602M bytes) Chain POSTROUTING (policy ACCEPT 960K packets, 602M bytes) On Tue, Jun 16, 2015 at 5:20 PM, dan dandenson@gmail.com wrote:
|
I made some tests with youtube and it looks like all ssl connections are not recognized. I presume that the same thing happens also with netflix and bittorent protocols for encrypted connections. Also noticed that blocking youtube comes together with unecrypted google and other destinations like github?! |
upon further testing, I think kong's guess on ssl being an issue holds up, and for many protocols. For example, I could only ID about 20% of an ubuntu ISO torrent, but 0% of a Fedora torrent. Inspecting the packets, the Fedora torrent was 100% SSL and the ubuntu torrent had a lot of unencrypted packets. This issue seems specific to ndpi-netfilter because ntopng will identify these as torrents right away. I'm wondering if the code that checks SSL certs is playing a more significant role in identifying packets and is only functional in ntopng/ndpi proper... |
"It can be detected by analyzing the SSL client certificate and checking the name that does not match to a real host in addition of begin a bit weird. As doing DNS resolution is not a task for nDPI we let applications do and then recognize SSL-tunnelled connections". It's a quote from nDPI/README.protocols. |
no change apparent... I flushed iptables first so all transfers are either the torrent, or the SSH session. 313M DL vs 800k dropped. Chain INPUT (policy ACCEPT 228K packets, 313M bytes) Chain FORWARD (policy ACCEPT 0 packets, 0 bytes) Chain OUTPUT (policy ACCEPT 117K packets, 7037K bytes) transmission: |
fyi, i did an svn co and got revision 41, and the module built and loaded fine: [65472.777851] nf_conntrack version 0.5.0 (16384 buckets, 65536 max) |
this specific match: this rule set even drops ICMP now: it drops dns lookups etc. it's matching almost everyhting.....except bittorrent which still gets through. It takes a while for the bittorrent to work around this block, but I just downloaded the ubuntu ISO in about 5 minutes with this block enabled while ever other network activity except SSH is funky. Generated by iptables-save v1.4.21 on Wed Jun 17 12:51:59 2015*filter Completed on Wed Jun 17 12:51:59 2015Generated by iptables-save v1.4.21 on Wed Jun 17 12:51:59 2015*mangle Completed on Wed Jun 17 12:51:59 2015 |
You remove xt_ndpi before? |
yes. I did this twice just to make sure. rmmod xt_ndpi (after flushing iptables). I even deleted all the source files and redownloaded, both a ZIP from github, and by svn and used the included nDPI archive. |
You don't build this with nDPI svn. |
This is a brief description of how nDPI treats SSL. Not much, because you can do it without nDPI, matching the known urls. |
I'm building with the nDPI archive included in the ndpi-netfilter sources. |
Hi, It has worked well in my tests. |
And, FYI... you dont need make firewall marks. |
Looks much better. Seems to work with youtube over SSL and also torrents... looks promising, but needs more testing. Good job by the way :) |
looks good, the matchers are basically functioning. bittorrent though, iptables -t mangle -I PREROUTING -m ndpi --dpi_check On Mon, Jun 22, 2015 at 12:43 PM, kong156 notifications@github.com wrote:
|
slight update, if I setup the block on bittorrent before starting a again, ntopng doesn't detect this data, so it's either the pull of ndpi we Thanks. On Mon, Jun 22, 2015 at 1:28 PM, dan dandenson@gmail.com wrote:
|
Tested the torrents, but they started to download only after I removed the rules from firewall. Maybe more testing is needed |
good job anyway, appreciate... |
Thanks alot |
for nothing |
Next days, I will test the shaping to verify syadnom's assertions. Right now, I don't have the right conditions to do this, but they will be. Matching and shaping rules are different things... |
I've just downloaded a 1.9GB encrypted torrent with the following rules set: Chain INPUT (policy ACCEPT 59057 packets, 64M bytes) #iptables -nvL -t mangle Chain PREROUTING (policy ACCEPT 1677K packets, 2365M bytes) Chain POSTROUTING (policy ACCEPT 35840 packets, 3065K bytes) The counters exceeds the download but it seems to be all right because we also have to take into account the extra traffic induced by proto/encryption overhead. |
kong, can you dump your iptables commands for me so I can test identical On Tue, Jun 23, 2015 at 2:49 PM, kong156 notifications@github.com wrote:
|
*mangle make abstraction of masks which are defaults |
I must be marking in the wrong spot... See my add below. In about 250MB *mangle added DSCP setting for the 0x1 mark from 2 commands up########################## Completed on Tue Jun 23 15:15:02 2015Generated by iptables-save v1.4.21 on Tue Jun 23 15:15:02 2015*filter On Tue, Jun 23, 2015 at 3:02 PM, kong156 notifications@github.com wrote:
|
DSCP doesn't preserve across the whole session, so you have to figure how to fix this issue. |
You set DSCP for egress and must fit egress only. The receiver doesn't preserve it so that you souldn't find it on ingress. |
What exactly you want to do? Are you trying to set DSCP into your firewall and shape in a border router? So, its depends on how and when you want to filter and shape it. |
You should shape your traffic with flow-based classifiers (ex. flow-based WFQ in case of a Cisco router or something similar for other vendors) or use a linux based router, brocade, vyatta, edgeos, maybe mikrotik if connmark or similar techniques are supported. |
Thanks to nDPI and betolj, now we can classify the traffic at L7 which is way better for lazy admins like me :(. You are a life saver, betolj :). Hope that my joy isn't premature. |
For the next release i'll work on IPv6 support too. |
ok, so I tested very thoroughly. It seems that nDPI is matching the I changed my iptables log file to a file and did this: then I pushed the src ip addresses from the match to my router into an I'm going to work on matching connections related to the conn-marked ones ######################## What I'm trying to do is mark torrents that are downstream of the router, I was hoping that just marking DSCP on the return packets would be enough My next path to this is to put the device upstream of the router and make On Wed, Jun 24, 2015 at 9:38 AM, Humberto Jucá notifications@github.com
|
Your DSCP approach is hopeless. It won't work. |
On Wed, Jun 24, 2015 at 12:16 PM, kong156 notifications@github.com wrote:
why? |
see the reason above. |
On Wed, Jun 24, 2015 at 12:17 PM, kong156 notifications@github.com wrote:
maybe I'm blind, but I see no reason above. |
:)) OK. You could use DSCP if you are in control of the both sides. The receiver doesn't preserve it so that you shouldn't find it on ingress. Source address is ephemeral. |
I'm worried about youtube... this one, at least, isn't functional for connmark. It's still ok for DROP rule. |
It seems that nDPI identifies it as Quic proto. That's fine for me :) Older versions find it as youtube. |
May be a true proto detection! |
If I understand right, you have a ndpi-netfilter machine as wire-bump behind a mikrotik router. In this case, I think you can restore the connmarks on mikrotik using DSCP marks from your ndpi-netfilter for both, upstream and downstream. Am I right? |
Also checked against facebook and it seems to work well. Can't tell if it catches till the last bit, but it performs very nice. |
I'm closing this as the 'issues' have basically been fixed or determined to be protocol specific oddities. Thanks. |
Fix out of bounds array indexing
I've built ndpi and ndpi-netfilter both from trunk on ubuntu 14.04 x64, kernel 3.13.0-55-generic, ndpi and ndpi-netfilter checked out from trunk 2015/6/16
I'm unable to load the module,
modprobe: ERROR: could not insert 'xt_ndpi': Unknown symbol in module, or unknown parameter (see dmesg)
@syadnom
I do see a couple warnings in the build:
WARNING: "ndpi_search_kakaotalk_voice" [/usr/src/ndpi-netfilter/trunk/src/xt_ndpi.ko] undefined!
WARNING: "ndpi_search_eaq" [/usr/src/ndpi-netfilter/trunk/src/xt_ndpi.ko] undefined!
LD [M] /usr/src/ndpi-netfilter/trunk/src/xt_ndpi.ko
make[2]: Leaving directory /usr/src/linux-headers-3.13.0-55-generic' rm -r ndpi_cpy make[1]: Leaving directory/usr/src/ndpi-netfilter/trunk/src'
here is the dmesg output:
[ 381.317982] nf_conntrack version 0.5.0 (16384 buckets, 65536 max)
[ 381.319381] xt_ndpi: module verification failed: signature and/or required key missing - tainting kernel
[ 1000.082087] xt_ndpi: Unknown symbol ndpi_search_eaq (err 0)
[ 1000.082122] xt_ndpi: Unknown symbol ndpi_search_kakaotalk_voice (err 0)
I removed kakaotalk and eaq from ndpi which allowed my to build and load xt_ndpi, but now I get a kernel panic whenever iptables loads the ndpi module.
dmesg shows this when loading the module
[ 2689.841269] xt_ndpi 2.0 (nDPI wrapper module).
[ 2689.849190] [NDPI] ndpi_string_to_automa(protoId=193): INTERNAL ERROR
[ 2689.849195] [NDPI] ndpi_string_to_automa(protoId=195): INTERNAL ERROR
[ 2689.849260] [NDPI] ndpi_string_to_automa(protoId=195): INTERNAL ERROR
[ 2689.849496] [NDPI] ndpi_init_protocol_defaults(missing protoId=194) INTERNAL ERROR: not all protocols have been initialized
now:
iptables -m ndpi --help segfaults
ndpi match options:
--ftp Match for FTP_CONTROL protocol packets.
--pop Match for MAIL_POP protocol packets.
<-- truncated list -->
--whatsapp_voice Match for WHATSAPP_VOICE protocol packets.
--dpi_check Match for CHECK protocol packets.
Segmentation fault (core dumped)
what revision of ndpi is ndpi-netfilter built against? I'd like to try again with whatever revision that is.
Thanks.
The text was updated successfully, but these errors were encountered: