New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

iptables plugin not working on Ubuntu >= 11.10 #326

Closed
peteroconnor opened this Issue May 17, 2013 · 15 comments

Comments

Projects
None yet
9 participants
@peteroconnor

peteroconnor commented May 17, 2013

If the section related to iptables is removed out of collectd.conf, things are fine. However, once it is added back, collectd stopped collecting all metrics.

/var/log/collectd.log doesn't show any error messages.

I wonder what I can do to investigate more.

apt-get update -y
apt-get install -y libconfig-general-perl librrds-perl libregexp-common-perl 
apt-get install -y iptables-dev linux-libc-dev collectd

In /etc/collectd/collectd.conf,

LoadPlugin cpu
LoadPlugin load
LoadPlugin memory

LoadPlugin logfile
<Plugin "logfile">
    LogLevel "debug"
    File "/var/log/collectd.log"
    Timestamp true
</Plugin>

LoadPlugin iptables
<Plugin "iptables">
    Chain filter OUTPUT
</Plugin>
@vbulax85

This comment has been minimized.

Show comment
Hide comment
@vbulax85

vbulax85 May 20, 2013

I can confirm this one.
For V5.3 result is the same as for V4.10.

% egrep '(iptables|collectd)'               
ii  collectd                          4.10.1-2.1ubuntu7  
ii  collectd-core                     4.10.1-2.1ubuntu7  
ii  iptables                          1.4.12-1ubuntu5    
ii  iptables-dev                      1.4.12-1ubuntu5    
% uname -a
Linux ubuntu12 3.2.0-43-virtual #68-Ubuntu SMP Wed May 15 04:15:45 UTC 2013 i686 athlon i386 GNU/Linux
% ls -al /lib/libtc*
lrwxrwxrwx 1 root root    17 Mar  8 01:47 /lib/libip4tc.so -> libip4tc.so.0.0.0
lrwxrwxrwx 1 root root    17 Mar  8 01:47 /lib/libip4tc.so.0 -> libip4tc.so.0.0.0
-rw-r--r-- 1 root root 26308 Mar  8 01:47 /lib/libip4tc.so.0.0.0
lrwxrwxrwx 1 root root    17 Mar  8 01:47 /lib/libip6tc.so -> libip6tc.so.0.0.0
lrwxrwxrwx 1 root root    17 Mar  8 01:47 /lib/libip6tc.so.0 -> libip6tc.so.0.0.0
-rw-r--r-- 1 root root 26340 Mar  8 01:47 /lib/libip6tc.so.0.0.0
lrwxrwxrwx 1 root root    19 Mar  8 01:47 /lib/libipq_pic.so -> libipq_pic.so.0.0.0
lrwxrwxrwx 1 root root    19 Mar  8 01:47 /lib/libipq_pic.so.0 -> libipq_pic.so.0.0.0
-rw-r--r-- 1 root root  9740 Mar  8 01:47 /lib/libipq_pic.so.0.0.0
lrwxrwxrwx 1 root root    15 Mar  8 01:47 /lib/libipq.so -> libipq.so.0.0.0
lrwxrwxrwx 1 root root    15 Mar  8 01:47 /lib/libipq.so.0 -> libipq.so.0.0.0
-rw-r--r-- 1 root root  9736 Mar  8 01:47 /lib/libipq.so.0.0.0
lrwxrwxrwx 1 root root    16 Mar  8 01:47 /lib/libiptc.so -> libiptc.so.0.0.0
lrwxrwxrwx 1 root root    16 Mar  8 01:47 /lib/libiptc.so.0 -> libiptc.so.0.0.0
-rw-r--r-- 1 root root  5268 Mar  8 01:47 /lib/libiptc.so.0.0.0
% pkg-config --libs libiptc
-liptc  
% nm -D /usr/lib/collectd/iptables.so
         w _Jv_RegisterClasses
00003088 A __bss_start
         w __cxa_finalize
         U __errno_location
         w __gmon_start__
         U __stack_chk_fail
         U __strdup
00003088 A _edata
00003098 A _end
00001948 T _fini
000007ec T _init
         U free
         U hostname_g
         U interval_g
         U ip6tc_first_rule
         U ip6tc_free
         U ip6tc_init
         U ip6tc_next_rule
         U ip6tc_strerror
         U iptc_first_rule
         U iptc_free
         U iptc_init
         U iptc_next_rule
         U iptc_strerror
         U malloc
00001880 T module_register
         U plugin_dispatch_values
         U plugin_log
         U plugin_register_config
         U plugin_register_read
         U plugin_register_shutdown
         U realloc
         U ssnprintf
         U sstrerror
         U sstrncpy
         U strcasecmp
         U strcmp
         U strlen
         U strsplit
         U strtol
% cat ./collectd_iptables.conf
LoadPlugin syslog
LoadPlugin iptables
<Plugin iptables>
        Chain "filter" "INPUT"
</Plugin>
% sudo /usr/sbin/collectd -f -T
/usr/sbin/collectd: symbol lookup error: /usr/lib/collectd/iptables.so: undefined symbol: iptc_init

vbulax85 commented May 20, 2013

I can confirm this one.
For V5.3 result is the same as for V4.10.

% egrep '(iptables|collectd)'               
ii  collectd                          4.10.1-2.1ubuntu7  
ii  collectd-core                     4.10.1-2.1ubuntu7  
ii  iptables                          1.4.12-1ubuntu5    
ii  iptables-dev                      1.4.12-1ubuntu5    
% uname -a
Linux ubuntu12 3.2.0-43-virtual #68-Ubuntu SMP Wed May 15 04:15:45 UTC 2013 i686 athlon i386 GNU/Linux
% ls -al /lib/libtc*
lrwxrwxrwx 1 root root    17 Mar  8 01:47 /lib/libip4tc.so -> libip4tc.so.0.0.0
lrwxrwxrwx 1 root root    17 Mar  8 01:47 /lib/libip4tc.so.0 -> libip4tc.so.0.0.0
-rw-r--r-- 1 root root 26308 Mar  8 01:47 /lib/libip4tc.so.0.0.0
lrwxrwxrwx 1 root root    17 Mar  8 01:47 /lib/libip6tc.so -> libip6tc.so.0.0.0
lrwxrwxrwx 1 root root    17 Mar  8 01:47 /lib/libip6tc.so.0 -> libip6tc.so.0.0.0
-rw-r--r-- 1 root root 26340 Mar  8 01:47 /lib/libip6tc.so.0.0.0
lrwxrwxrwx 1 root root    19 Mar  8 01:47 /lib/libipq_pic.so -> libipq_pic.so.0.0.0
lrwxrwxrwx 1 root root    19 Mar  8 01:47 /lib/libipq_pic.so.0 -> libipq_pic.so.0.0.0
-rw-r--r-- 1 root root  9740 Mar  8 01:47 /lib/libipq_pic.so.0.0.0
lrwxrwxrwx 1 root root    15 Mar  8 01:47 /lib/libipq.so -> libipq.so.0.0.0
lrwxrwxrwx 1 root root    15 Mar  8 01:47 /lib/libipq.so.0 -> libipq.so.0.0.0
-rw-r--r-- 1 root root  9736 Mar  8 01:47 /lib/libipq.so.0.0.0
lrwxrwxrwx 1 root root    16 Mar  8 01:47 /lib/libiptc.so -> libiptc.so.0.0.0
lrwxrwxrwx 1 root root    16 Mar  8 01:47 /lib/libiptc.so.0 -> libiptc.so.0.0.0
-rw-r--r-- 1 root root  5268 Mar  8 01:47 /lib/libiptc.so.0.0.0
% pkg-config --libs libiptc
-liptc  
% nm -D /usr/lib/collectd/iptables.so
         w _Jv_RegisterClasses
00003088 A __bss_start
         w __cxa_finalize
         U __errno_location
         w __gmon_start__
         U __stack_chk_fail
         U __strdup
00003088 A _edata
00003098 A _end
00001948 T _fini
000007ec T _init
         U free
         U hostname_g
         U interval_g
         U ip6tc_first_rule
         U ip6tc_free
         U ip6tc_init
         U ip6tc_next_rule
         U ip6tc_strerror
         U iptc_first_rule
         U iptc_free
         U iptc_init
         U iptc_next_rule
         U iptc_strerror
         U malloc
00001880 T module_register
         U plugin_dispatch_values
         U plugin_log
         U plugin_register_config
         U plugin_register_read
         U plugin_register_shutdown
         U realloc
         U ssnprintf
         U sstrerror
         U sstrncpy
         U strcasecmp
         U strcmp
         U strlen
         U strsplit
         U strtol
% cat ./collectd_iptables.conf
LoadPlugin syslog
LoadPlugin iptables
<Plugin iptables>
        Chain "filter" "INPUT"
</Plugin>
% sudo /usr/sbin/collectd -f -T
/usr/sbin/collectd: symbol lookup error: /usr/lib/collectd/iptables.so: undefined symbol: iptc_init
@shredder12

This comment has been minimized.

Show comment
Hide comment
@shredder12

shredder12 May 29, 2013

Any fix for this issue?

shredder12 commented May 29, 2013

Any fix for this issue?

@l13t

This comment has been minimized.

Show comment
Hide comment
@l13t

l13t Jun 12, 2013

good day. any ideas about fix this error?

l13t commented Jun 12, 2013

good day. any ideas about fix this error?

@ser

This comment has been minimized.

Show comment
Hide comment
@ser

ser commented Aug 30, 2013

I've just submitted it into ubuntu bugs:

https://bugs.launchpad.net/ubuntu/+source/collectd/+bug/1219080

@anitakrueger

This comment has been minimized.

Show comment
Hide comment
@anitakrueger

anitakrueger Jan 24, 2014

+1
Was so looking forward to use this plugin :(

anitakrueger commented Jan 24, 2014

+1
Was so looking forward to use this plugin :(

@fessyfoo

This comment has been minimized.

Show comment
Hide comment
@fessyfoo

fessyfoo Feb 3, 2014

I hacked around this problem like this:

install iptables-dev:

aptitude install iptables-dev

make a hacked wrapper script that uses LD_PRELOAD to load the missing library called /usr/sbin/collectd.wrap:

cat > /usr/sbin/collectd.wrap <<EOM
#!/bin/sh

export LD_PRELOAD=/lib/libip4tc.so 
exec collectd "$@"
EOM

chmod +x /etc/init.d# cat /usr/sbin/collectd.wrap

then update the init script to use this wrapper. illustrated here with a patch:

patch /etc/init.d/collectd <<EOM
--- /etc/init.d/collectd.orig   2014-02-02 16:53:34.033511601 -0800
+++ /etc/init.d/collectd        2014-02-02 16:54:37.290688460 -0800
@@ -26,7 +26,7 @@

 DESC="statistics collection and monitoring daemon"
 NAME=collectd
-DAEMON=/usr/sbin/collectd
+DAEMON=/usr/sbin/collectd.wrap

 CONFIGFILE=/etc/collectd/collectd.conf
 PIDFILE=/var/run/collectd.pid
@@ -90,7 +90,7 @@

        if test "$USE_COLLECTDMON" == 1; then
                start-stop-daemon --start --quiet --oknodo --pidfile "$_PIDFILE" \
-                       --exec $COLLECTDMON_DAEMON -- -P "$_PIDFILE" -- -C "$CONFIGFILE"
+                       --exec $COLLECTDMON_DAEMON -- -c "$DAEMON" -P "$_PIDFILE" -- -C "$CONFIGFILE"
        else
                start-stop-daemon --start --quiet --oknodo --pidfile "$_PIDFILE" \
EOM

fessyfoo commented Feb 3, 2014

I hacked around this problem like this:

install iptables-dev:

aptitude install iptables-dev

make a hacked wrapper script that uses LD_PRELOAD to load the missing library called /usr/sbin/collectd.wrap:

cat > /usr/sbin/collectd.wrap <<EOM
#!/bin/sh

export LD_PRELOAD=/lib/libip4tc.so 
exec collectd "$@"
EOM

chmod +x /etc/init.d# cat /usr/sbin/collectd.wrap

then update the init script to use this wrapper. illustrated here with a patch:

patch /etc/init.d/collectd <<EOM
--- /etc/init.d/collectd.orig   2014-02-02 16:53:34.033511601 -0800
+++ /etc/init.d/collectd        2014-02-02 16:54:37.290688460 -0800
@@ -26,7 +26,7 @@

 DESC="statistics collection and monitoring daemon"
 NAME=collectd
-DAEMON=/usr/sbin/collectd
+DAEMON=/usr/sbin/collectd.wrap

 CONFIGFILE=/etc/collectd/collectd.conf
 PIDFILE=/var/run/collectd.pid
@@ -90,7 +90,7 @@

        if test "$USE_COLLECTDMON" == 1; then
                start-stop-daemon --start --quiet --oknodo --pidfile "$_PIDFILE" \
-                       --exec $COLLECTDMON_DAEMON -- -P "$_PIDFILE" -- -C "$CONFIGFILE"
+                       --exec $COLLECTDMON_DAEMON -- -c "$DAEMON" -P "$_PIDFILE" -- -C "$CONFIGFILE"
        else
                start-stop-daemon --start --quiet --oknodo --pidfile "$_PIDFILE" \
EOM
@fessyfoo

This comment has been minimized.

Show comment
Hide comment
@fessyfoo

fessyfoo Feb 3, 2014

not really an elegant fix, and I don't fully understand what the right fix is. I think something like the shared libraries in the iptables-dev library need to be linked statically into collectds iptables.so or they need to be available in a separate package that collectd depends on and linked to dynamically in collectds iptables.so

fessyfoo commented Feb 3, 2014

not really an elegant fix, and I don't fully understand what the right fix is. I think something like the shared libraries in the iptables-dev library need to be linked statically into collectds iptables.so or they need to be available in a separate package that collectd depends on and linked to dynamically in collectds iptables.so

@davewongillies

This comment has been minimized.

Show comment
Hide comment
@davewongillies

davewongillies Feb 17, 2014

So in src/Makefile.in and src/Makefile.am I changed:

@BUILD_PLUGIN_IPTABLES_TRUE@iptables_la_LIBADD = -liptc

to

@BUILD_PLUGIN_IPTABLES_TRUE@iptables_la_LIBADD = -liptc -lip4tc -lip6tc

It doesn't appear to be sending metrics right now, but perhaps its my iptables or collectd config:

<Plugin iptables>
    Chain "filter" "FORWARD"
    Chain "filter" "OUTPUT"
    Chain "filter" "INPUT"
</Plugin>

davewongillies commented Feb 17, 2014

So in src/Makefile.in and src/Makefile.am I changed:

@BUILD_PLUGIN_IPTABLES_TRUE@iptables_la_LIBADD = -liptc

to

@BUILD_PLUGIN_IPTABLES_TRUE@iptables_la_LIBADD = -liptc -lip4tc -lip6tc

It doesn't appear to be sending metrics right now, but perhaps its my iptables or collectd config:

<Plugin iptables>
    Chain "filter" "FORWARD"
    Chain "filter" "OUTPUT"
    Chain "filter" "INPUT"
</Plugin>
@davewongillies

This comment has been minimized.

Show comment
Hide comment
@davewongillies

davewongillies Feb 18, 2014

So my collectd.conf was incomplete and also my iptables rules that I'm testing with don't have any comments. Using this minimal config, I got some metrics out:

<Plugin iptables>
  Chain "filter" "INPUT" 5
</Plugin>

Out the other side in graphite:

my.hostname.iptables.filter-INPUT.ipt_packets.5 76934 1392681616
my.hostname.iptables.filter-INPUT.ipt_bytes.5 176003659 1392681616

davewongillies commented Feb 18, 2014

So my collectd.conf was incomplete and also my iptables rules that I'm testing with don't have any comments. Using this minimal config, I got some metrics out:

<Plugin iptables>
  Chain "filter" "INPUT" 5
</Plugin>

Out the other side in graphite:

my.hostname.iptables.filter-INPUT.ipt_packets.5 76934 1392681616
my.hostname.iptables.filter-INPUT.ipt_bytes.5 176003659 1392681616
@mfournier

This comment has been minimized.

Show comment
Hide comment
@mfournier

mfournier Feb 18, 2014

Contributor

Hey, nice catch @davewongillies & @fessyfoo !

So was this with collectd 4.10.1 ? Or one of the latest 5.x ?

Here, only fixes in 5.3.x and 5.4.x will happen, as they are the current supported releases.

For those who'd like to make it hit 4.10.1 in ubuntu 12.04, you'll have to work with the ubuntu maintainers over at https://bugs.launchpad.net/ubuntu/+source/collectd/+bug/1219080

If the problem is indeed on all the versions of collectd from 4.10 to master, it should probably be pretty easy to figure out a proper solution on current versions, and submit the fix to ubuntu 12.04 as a dpatch.

Contributor

mfournier commented Feb 18, 2014

Hey, nice catch @davewongillies & @fessyfoo !

So was this with collectd 4.10.1 ? Or one of the latest 5.x ?

Here, only fixes in 5.3.x and 5.4.x will happen, as they are the current supported releases.

For those who'd like to make it hit 4.10.1 in ubuntu 12.04, you'll have to work with the ubuntu maintainers over at https://bugs.launchpad.net/ubuntu/+source/collectd/+bug/1219080

If the problem is indeed on all the versions of collectd from 4.10 to master, it should probably be pretty easy to figure out a proper solution on current versions, and submit the fix to ubuntu 12.04 as a dpatch.

@davewongillies

This comment has been minimized.

Show comment
Hide comment
@davewongillies

davewongillies Feb 18, 2014

Hi @mfournier, my testing has been on Ubuntu 12.04 with collectd 5.4.0

davewongillies commented Feb 18, 2014

Hi @mfournier, my testing has been on Ubuntu 12.04 with collectd 5.4.0

@fessyfoo

This comment has been minimized.

Show comment
Hide comment
@fessyfoo

fessyfoo Feb 18, 2014

I was working with Ubuntu 12.04 with collectd deb version 4.10.1-2.1ubuntu7

fessyfoo commented Feb 18, 2014

I was working with Ubuntu 12.04 with collectd deb version 4.10.1-2.1ubuntu7

@mfournier

This comment has been minimized.

Show comment
Hide comment
@mfournier

mfournier Mar 8, 2014

Contributor

What happens in this case is that libiptc.so is mostly empty and it's sole role seems to be to depend on 2 other libs: libip4tc.so and libip6tc.so. Comparing the output of objdump -T /lib/libiptc.so with objdump -T /lib/libip4tc.so make that pretty obvious.

By default, gcc on Ubuntu only links libraries which are actually used (cf https://wiki.ubuntu.com/ToolChain/CompilerFlags#A-Wl.2C--as-needed). But as pkg-config --libs libiptc returns -liptc and not -lip4tc -lip6tc, the linker drops libiptc, as no code refers to it.

So both workarounds described above are perfectly safe to use, imho: using LD_PRELOAD or changing the Makefile to explicitly link libip4tc and libip6tc.

I think the problem is now nailed down. Now what's the best fix, the question remains open. What do you think ? @tokkee's opinion could be valuable here.

Contributor

mfournier commented Mar 8, 2014

What happens in this case is that libiptc.so is mostly empty and it's sole role seems to be to depend on 2 other libs: libip4tc.so and libip6tc.so. Comparing the output of objdump -T /lib/libiptc.so with objdump -T /lib/libip4tc.so make that pretty obvious.

By default, gcc on Ubuntu only links libraries which are actually used (cf https://wiki.ubuntu.com/ToolChain/CompilerFlags#A-Wl.2C--as-needed). But as pkg-config --libs libiptc returns -liptc and not -lip4tc -lip6tc, the linker drops libiptc, as no code refers to it.

So both workarounds described above are perfectly safe to use, imho: using LD_PRELOAD or changing the Makefile to explicitly link libip4tc and libip6tc.

I think the problem is now nailed down. Now what's the best fix, the question remains open. What do you think ? @tokkee's opinion could be valuable here.

@mfournier

This comment has been minimized.

Show comment
Hide comment
@mfournier

mfournier Mar 10, 2014

Contributor

Quick census:

precise:

# pkg-config --libs libiptc
-liptc  
# pkg-config --libs --static libiptc
-liptc -lip4tc -lip6tc  

quantal:

# pkg-config --libs libiptc
-liptc
# pkg-config --libs --static libiptc
-liptc -lip4tc -lip6tc

raring:

# pkg-config --libs libiptc
-liptc  
# pkg-config --libs --static lbiptc
-liptc -lip4tc -lip6tc

saucy:

# pkg-config --libs libiptc
-lip4tc -lip6tc
# pkg-config --libs --static libiptc
-lip4tc -lip6tc

trusty:

# pkg-config --libs libiptc
-lip4tc -lip6tc
# pkg-config --libs --static libiptc
-lip4tc -lip6tc

squeeze:

# pkg-config --libs libiptc
-L/lib -liptc
# pkg-config --libs --static libiptc
-L/lib -liptc

wheezy:

# pkg-config --libs libiptc
-lip4tc -lip6tc  
# pkg-config --libs --static libiptc
-lip4tc -lip6tc

jessie:

# pkg-config --libs libiptc
-lip4tc -lip6tc  
# pkg-config --libs --static libiptc
-lip4tc -lip6tc

centos 5.8: no libiptc.pc

centos 6.4:

# pkg-config --libs libiptc
-L/lib64 -liptc  
# pkg-config --libs --static libiptc
-L/lib64 -liptc
Contributor

mfournier commented Mar 10, 2014

Quick census:

precise:

# pkg-config --libs libiptc
-liptc  
# pkg-config --libs --static libiptc
-liptc -lip4tc -lip6tc  

quantal:

# pkg-config --libs libiptc
-liptc
# pkg-config --libs --static libiptc
-liptc -lip4tc -lip6tc

raring:

# pkg-config --libs libiptc
-liptc  
# pkg-config --libs --static lbiptc
-liptc -lip4tc -lip6tc

saucy:

# pkg-config --libs libiptc
-lip4tc -lip6tc
# pkg-config --libs --static libiptc
-lip4tc -lip6tc

trusty:

# pkg-config --libs libiptc
-lip4tc -lip6tc
# pkg-config --libs --static libiptc
-lip4tc -lip6tc

squeeze:

# pkg-config --libs libiptc
-L/lib -liptc
# pkg-config --libs --static libiptc
-L/lib -liptc

wheezy:

# pkg-config --libs libiptc
-lip4tc -lip6tc  
# pkg-config --libs --static libiptc
-lip4tc -lip6tc

jessie:

# pkg-config --libs libiptc
-lip4tc -lip6tc  
# pkg-config --libs --static libiptc
-lip4tc -lip6tc

centos 5.8: no libiptc.pc

centos 6.4:

# pkg-config --libs libiptc
-L/lib64 -liptc  
# pkg-config --libs --static libiptc
-L/lib64 -liptc

@mfournier mfournier self-assigned this Mar 10, 2014

mfournier added a commit to mfournier/collectd that referenced this issue Mar 10, 2014

link iptables plugin against flags from pkg-config
This solves issue #326 for ubuntu versions >= 13.10
@mfournier

This comment has been minimized.

Show comment
Hide comment
@mfournier

mfournier Jul 29, 2014

Contributor

Closing as the bugfix has been merged. This issue should not appear anymore in the next 5.3.x, 5.4.x and newer versions of collectd.

NB: for ubuntu versions prior to saucy, the support library is broken in the distro, and there's nothing we can do about it. The LD_PRELOAD trick mentioned above is a valid workaround though.

Thanks to all for your help figuring out the problem !

Contributor

mfournier commented Jul 29, 2014

Closing as the bugfix has been merged. This issue should not appear anymore in the next 5.3.x, 5.4.x and newer versions of collectd.

NB: for ubuntu versions prior to saucy, the support library is broken in the distro, and there's nothing we can do about it. The LD_PRELOAD trick mentioned above is a valid workaround though.

Thanks to all for your help figuring out the problem !

@mfournier mfournier closed this Jul 29, 2014

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment