netdata uses obsolete procfs for conntrack data - netfilter.plugin

[jlec]

Supply CT list in procfs (OBSOLETE)
CONFIG_NF_CONNTRACK_PROCFS:
This option enables for the list of known conntrack entries
to be shown in procfs under net/netfilter/nf_conntrack. This
is considered obsolete in favor of using the conntrack(8)
tool which uses Netlink.

In current kernels the procfs interface is obsolete. Could netdata query the conntrack data in an different way?

[ktsaou]

It does not use files in /proc/net/netfilter.

It is using /proc/net/stat/nf_conntrack. So, it does not need to read through all the sockets monitored by the connection tracker like conntrack(8) does.

It just needs the connection tracker statistics:

# cat /proc/net/stat/nf_conntrack
entries  searched found new invalid ignore delete delete_list insert insert_failed drop early_drop icmp_error  expect_new expect_create expect_delete search_restart
0000071e  0760e6ff 8c03e663 00522d9e 001e9e1a 00ac099f 006c61af 005a1da9 003fe1df 00000000 0000053a 00000000 00021251  00000029 00004b29 000069a6 00000000
0000071e  00b57258 0f2d9e94 0026027a 00010d90 007c689a 001a408f 0017d10c 00239697 00000000 00000000 00000000 00000000  00000007 0000294d 00001de8 00000000
0000071e  0044e878 057c5a59 001b46d7 0000a2ba 008c734b 00150fd0 0013a597 0019df0e 00000000 00000008 00000000 00000000  00000007 00001ae1 00001188 00000000
0000071e  004414c8 05638122 001b4c1a 0000a339 008d316b 00130a1a 0011a0a4 0019e489 00000000 00000008 00000000 00000000  00000005 00001bde 0000121b 00000000

Is /proc/net/stat/nf_conntrack enabled with CONFIG_NF_CONNTRACK_PROCFS?

[ktsaou]

Yeap! It is part of CONFIG_NF_CONNTRACK_PROCFS.

So, we are moving towards another plugin with external dependencies (libmnl) and special access rights...

[ktsaou]

There is also a pending task to move nfacct data collection outside netdata (currently it is disabled), which also depends on libmnl (#22 #63), so both could be combined to one netfilter.plugin.

[shartge]

Be careful. Directly using /proc/net/stat/nf_conntrack will stall the whole network stack if there are many connections active. See https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=565565 which is a bug filed against munin.

You may not see this problem on your home router but when you read from /proc/net/stat/nf_conntrack on a system with more than 20,000 active connections, you will generate network hangs.

[ktsaou]

@shartge they are talking about /proc/net/nf_conntrack not /proc/net/stat/nf_conntrack.
There is /stat/ in the file read by netdata. The other one is a huge list of all open sockets of the system. The /stat/ file is just a few lines with counters.

[shartge]

I see, yes. And to be sure, I just confirmed that accessing /proc/net/stat/* does not put any load on the system, while accessing /proc/net/nf_conntrack directly takes a very very long time (40 seconds on a firewall with 100,000 tracked connection).
So you really want to stay away from that.

[ktsaou]

I just confirmed that accessing /proc/net/stat/* does not put any load on the system

thanks!

[dseira]

Another way to get the current netfilter connections is through /proc/sys/net/netfilter/nf_conntrack_count variable. Where /proc/sys/net/netfilter/nf_conntrack_max is the maximum value that nf_conntrack_count can reach.

The explanation of that value is:

Number of currently allocated flow entries.

https://www.kernel.org/doc/Documentation/networking/nf_conntrack-sysctl.txt

I don't know if this value should be in the same plugin as this thread or into a separate one. What do you think @ktsaou ?

[ktsaou]

I think this is a very nice idea. So I added it with PR #1273 and I also added an alarm based on the max value. Thank you for suggesting it.

[ktsaou]

and I just merged it...

[dseira]

Thanks to you for this great tool!

[ktsaou]

I am refactoring the nfacct plugin to turn it to an external process that will work with setuid to root. I am trying to find if I can collect netfilter statistics with libmnl, to satisfy the original requirement. I can't find any information. It seems that the only way to collect this information is conntrackd.

If you have any ideas how to do it, please post...

[ktsaou]

Found it (reading the kernel is always a joy: IPCTNL_MSG_CT_GET_STATS_CPU https://github.com/torvalds/linux/search?utf8=%E2%9C%93&q=IPCTNL_MSG_CT_GET_STATS_CPU).

And how conntrack uses it: https://github.com/formorer/pkg-conntrack-tools/blob/master/src/conntrack.c

[ktsaou]

connection tracker statistics using libmnl (netlink) in PR #1963

[ktsaou]

merged it.