mlogc with 100% cpu load and apache not serving requests

[chrissaffer]

Hello everybody,

we are running several apache 2.2 and modsecurity instances.
After running a while a random server "crashes" without any reason and doesn't serve any requests...

Here are the versions:

:# dpkg -l |grep apache
ii apache2 2.2.16-6+squeeze11 Apache HTTP Server metapackage
ii apache2-mpm-worker 2.2.16-6+squeeze11 Apache HTTP Server - high speed threaded model
ii apache2-threaded-dev 2.2.16-6+squeeze11 Apache development headers - threaded MPM
ii apache2-utils 2.2.16-6+squeeze11 utility programs for webservers
ii apache2.2-bin 2.2.16-6+squeeze11 Apache HTTP Server common binary files
ii apache2.2-common 2.2.16-6+squeeze11 Apache HTTP Server common files
ii apachetop 0.12.6-12 Realtime Apache monitoring tool
ii libapache2-mod-auth-plain 2.0.50 Module for Apache2 which provides plaintext authentication
ii libapache2-mod-fcgid 1:2.3.6-1+squeeze1 an alternative module compat with mod_fastcgi
ii libapache2-mod-geoip 1.2.5-2 GeoIP support for apache2
ii libapache2-mod-perl2 2.0.4-7+squeeze1 Integration of perl with the Apache2 web server
ii libapache2-mod-rpaf 0.6-7bpo60+1 module for Apache2 which takes the last IP from the 'X-Forwarded-For' header
ii libapache2-redirtoservname 0.1.2-3 Apache 2 module to redirect users to the canonical hostname
ii libapache2-reload-perl 0.10-2 Reload Perl modules when changed on disk

:~# /usr/local/modsecurity/bin/mlogc -v
ModSecurity Log Collector (mlogc) v2.7.7
APR: compiled="1.4.2"; loaded="1.4.2"
PCRE: compiled="8.2"; loaded="8.02 2010-03-19"
cURL: compiled="7.21.0"; loaded="libcurl/7.21.0 OpenSSL/0.9.8o zlib/1.2.3.4 libidn/1.15 libssh2/1.2.6"

A strace on the process does nothing, here are the gdb output:

(gdb) attach 39324
Attaching to process 39324
Reading symbols from /usr/local/modsecurity/bin/mlogc...done.
Reading symbols from /usr/lib/libapr-1.so.0...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/libapr-1.so.0
Reading symbols from /usr/lib/libcurl.so.4...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/libcurl.so.4
Reading symbols from /lib/libpcre.so.3...(no debugging symbols found)...done.
Loaded symbols for /lib/libpcre.so.3
Reading symbols from /lib/libpthread.so.0...(no debugging symbols found)...done.
[Thread debugging using libthread_db enabled]
[New Thread 0x7f46d1927700 (LWP 39571)]
[New Thread 0x7f46d2128700 (LWP 39570)]
Loaded symbols for /lib/libpthread.so.0
Reading symbols from /lib/libc.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib/libc.so.6
Reading symbols from /lib/libuuid.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib/libuuid.so.1
Reading symbols from /lib/librt.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib/librt.so.1
Reading symbols from /lib/libcrypt.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib/libcrypt.so.1
Reading symbols from /lib/libdl.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib/libdl.so.2
Reading symbols from /usr/lib/libidn.so.11...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/libidn.so.11
Reading symbols from /usr/lib/libssh2.so.1...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/libssh2.so.1
Reading symbols from /usr/lib/liblber-2.4.so.2...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/liblber-2.4.so.2
Reading symbols from /usr/lib/libldap_r-2.4.so.2...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/libldap_r-2.4.so.2
Reading symbols from /usr/lib/libgssapi_krb5.so.2...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/libgssapi_krb5.so.2
Reading symbols from /usr/lib/libssl.so.0.9.8...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/libssl.so.0.9.8
Reading symbols from /usr/lib/libcrypto.so.0.9.8...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/libcrypto.so.0.9.8
Reading symbols from /usr/lib/libz.so.1...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/libz.so.1
Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
Reading symbols from /usr/lib/libgcrypt.so.11...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/libgcrypt.so.11
Reading symbols from /lib/libresolv.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib/libresolv.so.2
Reading symbols from /usr/lib/libsasl2.so.2...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/libsasl2.so.2
Reading symbols from /usr/lib/libgnutls.so.26...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/libgnutls.so.26
Reading symbols from /usr/lib/libkrb5.so.3...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/libkrb5.so.3
Reading symbols from /usr/lib/libk5crypto.so.3...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/libk5crypto.so.3
Reading symbols from /lib/libcom_err.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib/libcom_err.so.2
Reading symbols from /usr/lib/libkrb5support.so.0...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/libkrb5support.so.0
Reading symbols from /lib/libkeyutils.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib/libkeyutils.so.1
Reading symbols from /usr/lib/libgpg-error.so.0...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/libgpg-error.so.0
Reading symbols from /usr/lib/libtasn1.so.3...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/libtasn1.so.3
Reading symbols from /lib/libnss_files.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib/libnss_files.so.2
Reading symbols from /lib/libnss_dns.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib/libnss_dns.so.2
Reading symbols from /lib/libgcc_s.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib/libgcc_s.so.1
0x00007f46d5ee749d in apr_pool_destroy () from /usr/lib/libapr-1.so.0

(gdb) bt
#0 0x00007f46d5ee749d in apr_pool_destroy () from /usr/lib/libapr-1.so.0
#1 0x00007f46d5ee7448 in apr_pool_destroy () from /usr/lib/libapr-1.so.0
#2 0x0000000000405029 in create_new_worker (lock=0) at mlogc.c:1796
#3 0x0000000000405187 in add_entry (data=, start_worker=1) at mlogc.c:409
#4 0x0000000000405afa in receive_loop () at mlogc.c:2065
#5 0x000000000040614f in main (argc=, argv=0x7fff380923d8) at mlogc.c:2306

Does anybody know what to do here or where the problem is?

Regards,
Christian

[zimmerle]

Most likely a consequence of PCRE version mismatch.