Spontaneously high CPU usage

[celesteking]

This bug is hard to replicate, but I'll try to describe it here.
We've activated modsec on our servers, but with some we're noticing extremely high cpu usage. It manifests itself after running for some time in detection mode.
We have mlogc logging enabled.

It might be caused by graceful restart, although I'm not sure -- no way to replicate reliably.

Here's what happens with a process that's consuming 100% CPU:

(gdb) bt
#0  0x00002b36cebad969 in run_child_cleanups () from /usr/local/apache/lib/libapr-1.so.0
#1  0x00002b36cebad9af in cleanup_pool_for_exec () from /usr/local/apache/lib/libapr-1.so.0
#2  0x00002b36cebad9c6 in cleanup_pool_for_exec () from /usr/local/apache/lib/libapr-1.so.0
#3  0x00002b36cebad9c6 in cleanup_pool_for_exec () from /usr/local/apache/lib/libapr-1.so.0
#4  0x00002b36cebad9eb in apr_pool_cleanup_for_exec () from /usr/local/apache/lib/libapr-1.so.0
#5  0x00002b36cebba4e9 in apr_proc_create () from /usr/local/apache/lib/libapr-1.so.0
#6  0x00002b36cf1e86c2 in suphp_script_handler (r=0x1122ceb8) at mod_suphp.c:953
#7  0x00002b36cf1e8ef9 in suphp_handler (r=0x1122ceb8) at mod_suphp.c:569
#8  0x000000000044ac13 in ap_run_handler ()
#9  0x000000000044b4dc in ap_invoke_handler ()
#10 0x00000000004b9ddd in ap_internal_redirect ()
#11 0x00000000004e1f11 in handler_redirect ()
#12 0x000000000044ac13 in ap_run_handler ()
#13 0x000000000044b4dc in ap_invoke_handler ()
#14 0x00000000004b9ddd in ap_internal_redirect ()
#15 0x00000000004b90d3 in ap_die ()
#16 0x00000000004b92ca in ap_process_request ()
#17 0x00000000004b5abd in ap_process_http_connection ()
#18 0x00000000004546cf in ap_run_process_connection ()
#19 0x0000000000454b33 in ap_process_connection ()
#20 0x00000000004e3157 in process_socket ()
#21 0x00000000004e3a82 in worker_thread ()
#22 0x00002b36cebbb3a1 in dummy_worker () from /usr/local/apache/lib/libapr-1.so.0
#23 0x00000037c180683d in start_thread () from /lib64/libpthread.so.0
#24 0x00000037c10d4fcd in clone () from /lib64/libc.so.6

strace tells nothing (as far as I tried). apache running perfectly fine without modsec (SecRuleEngine Off).

# httpd  -V
Server version: Apache/2.2.29 (Unix)
Server built:   May 14 2015 10:34:22
Cpanel::Easy::Apache v3.28.8 rev9999
Server's Module Magic Number: 20051115:36
Server loaded:  APR 1.5.1, APR-Util 1.5.4
Compiled using: APR 1.5.1, APR-Util 1.5.4
Architecture:   64-bit
Server MPM:     Worker
  threaded:     yes (fixed thread count)
    forked:     yes (variable process count)
Server compiled with....
 -D APACHE_MPM_DIR="server/mpm/worker"
 -D APR_HAS_SENDFILE
 -D APR_HAS_MMAP
 -D APR_HAVE_IPV6 (IPv4-mapped addresses disabled)
 -D APR_USE_SYSVSEM_SERIALIZE
 -D APR_USE_PTHREAD_SERIALIZE
 -D SINGLE_LISTEN_UNSERIALIZED_ACCEPT
 -D APR_HAS_OTHER_CHILD
 -D AP_HAVE_RELIABLE_PIPED_LOGS
 -D DYNAMIC_MODULE_LIMIT=128
 -D HTTPD_ROOT="/usr/local/apache"
 -D SUEXEC_BIN="/usr/local/apache/bin/suexec"
 -D DEFAULT_SCOREBOARD="logs/apache_runtime_status"
 -D DEFAULT_ERRORLOG="logs/error_log"
 -D AP_TYPES_CONFIG_FILE="conf/mime.types"
 -D SERVER_CONFIG_FILE="conf/httpd.conf"

Hopefully, will have more info as we gather stats.

[celesteking]

Alright, happening on another server now, with same versions. Take a look at that --

  PID USER      PR  NI  VIRT  RES  SHR SWAP S %CPU %MEM    TIME+  COMMAND                                                                                                                                                                    
 8248 nobody    20   0 2035m  47m  368    0 R 25.1  0.3   3:42.35 /usr/local/apache/bin/httpd -k start -DSSL                                                                                                                                 
 7573 nobody    20   0 2035m  47m  368    0 R 24.8  0.3   4:14.22 /usr/local/apache/bin/httpd -k start -DSSL                                                                                                                                 
 7921 nobody    20   0 2035m  47m  368    0 R 24.8  0.3   3:58.99 /usr/local/apache/bin/httpd -k start -DSSL                                                                                                                                 
 7984 nobody    20   0 2035m  47m  368    0 R 24.8  0.3   3:43.85 /usr/local/apache/bin/httpd -k start -DSSL                                                                                                                                 
 8012 nobody    20   0 2035m  47m  368    0 R 24.8  0.3   3:44.34 /usr/local/apache/bin/httpd -k start -DSSL                                                                                                                                 
 8177 nobody    20   0 2035m  47m  368    0 R 24.8  0.3   3:39.23 /usr/local/apache/bin/httpd -k start -DSSL                                                                                                                                 
 7985 nobody    20   0 2035m  47m  368    0 R 24.4  0.3   3:44.45 /usr/local/apache/bin/httpd -k start -DSSL                                                                                                                                 
 8104 nobody    20   0 2035m  47m  368    0 R 24.4  0.3   3:49.61 /usr/local/apache/bin/httpd -k start -DSSL                                                                                                                                 
 8132 nobody    20   0 2035m  47m  368    0 R 24.4  0.3   3:45.99 /usr/local/apache/bin/httpd -k start -DSSL                                                                                                                                 
 7596 nobody    20   0 2035m  47m  368    0 R 23.5  0.3   4:40.06 /usr/local/apache/bin/httpd -k start -DSSL                                                                                                                                 
 7947 nobody    20   0 2035m  47m  368    0 R 23.5  0.3   4:05.32 /usr/local/apache/bin/httpd -k start -DSSL                                                                                                                                 
 8019 nobody    20   0 2035m  47m  368    0 R 23.5  0.3   3:44.38 /usr/local/apache/bin/httpd -k start -DSSL                                                                                                                                 
 8159 nobody    20   0 2035m  47m  368    0 R 23.5  0.3   3:36.48 /usr/local/apache/bin/httpd -k start -DSSL                                                                                                                                 
 8360 nobody    20   0 2035m  47m  368    0 R 23.5  0.3   3:36.19 /usr/local/apache/bin/httpd -k start -DSSL                                                                                                                                 
 8409 nobody    20   0 2035m  47m  368    0 R 23.5  0.3   3:29.32 /usr/local/apache/bin/httpd -k start -DSSL                                                                                                                                 
 8447 nobody    20   0 2035m  47m  368    0 R 23.5  0.3   3:39.56 /usr/local/apache/bin/httpd -k start -DSSL                                                                                                                                 
 7971 nobody    20   0 2035m  47m  368    0 R 23.1  0.3   4:00.09 /usr/local/apache/bin/httpd -k start -DSSL                                         

SecData dir:

-rw-r-----  1 nobody nobody    0 May  6 07:35 global.dir
-rw-r-----  1 nobody nobody    0 May  6 07:35 global.pag
-rw-r-----  1 nobody nobody 4.0K May 23 16:54 ip.dir
-rw-r-----  1 nobody nobody  24M May 23 16:54 ip.pag

And again, nothing in strace output.. But CPU continues to be consumed!

[celesteking]

Compiled apr with debug on.

gdb) bt
#0  0x00002ad3759cd969 in run_child_cleanups (cref=0x1606f728) at memory/unix/apr_pools.c:2364
#1  0x00002ad3759cd9af in cleanup_pool_for_exec (p=0x1606f708) at memory/unix/apr_pools.c:2372
#2  0x00002ad3759cd9c6 in cleanup_pool_for_exec (p=0x1606f708) at memory/unix/apr_pools.c:2375
#3  0x00002ad3759cd9c6 in cleanup_pool_for_exec (p=0x1606d6f8) at memory/unix/apr_pools.c:2375
#4  0x00002ad3759cd9eb in apr_pool_cleanup_for_exec () at memory/unix/apr_pools.c:2380
#5  0x00002ad3759da4e9 in apr_proc_create (new=0x17b1a770, progname=0x2ad376010769 "/opt/suphp/sbin/suphp", args=0x17b19170, env=0x17b19da0, attr=0x17b1a320, pool=0x17aec408) at threadproc/unix/proc.c:425
#6  0x00002ad37600f6c2 in suphp_script_handler (r=0x17afe1c0) at mod_suphp.c:953
#7  0x00002ad37600fef9 in suphp_handler (r=0x17afe1c0) at mod_suphp.c:569
(gdb) list
2359    static void run_child_cleanups(cleanup_t **cref)
2360    {
2361        cleanup_t *c = *cref;
2362
2363        while (c) {
2364            *cref = c->next;
2365            (*c->child_cleanup_fn)((void *)c->data);
2366            c = *cref;
2367        }
2368    }

(gdb) display **cref
1: **cref = {next = 0x161a37c8, data = 0x179b4290, plain_cleanup_fn = 0x2ad3765ba680 <msc_pcre_cleanup>, child_cleanup_fn = 0x42c3b8 <apr_pool_cleanup_null@plt>}
(gdb) next
2366            c = *cref;
1: **cref = {next = 0x161a37c8, data = 0x179b4290, plain_cleanup_fn = 0x2ad3765ba680 <msc_pcre_cleanup>, child_cleanup_fn = 0x42c3b8 <apr_pool_cleanup_null@plt>}
(gdb) 
2363        while (c) {
1: **cref = {next = 0x161a37c8, data = 0x179b4290, plain_cleanup_fn = 0x2ad3765ba680 <msc_pcre_cleanup>, child_cleanup_fn = 0x42c3b8 <apr_pool_cleanup_null@plt>}
(gdb) 
2364            *cref = c->next;
1: **cref = {next = 0x161a37c8, data = 0x179b4290, plain_cleanup_fn = 0x2ad3765ba680 <msc_pcre_cleanup>, child_cleanup_fn = 0x42c3b8 <apr_pool_cleanup_null@plt>}
(gdb) 
2365            (*c->child_cleanup_fn)((void *)c->data);
1: **cref = {next = 0x161a37c8, data = 0x179b4290, plain_cleanup_fn = 0x2ad3765ba680 <msc_pcre_cleanup>, child_cleanup_fn = 0x42c3b8 <apr_pool_cleanup_null@plt>}
(gdb) 
2366            c = *cref;
1: **cref = {next = 0x161a37c8, data = 0x179b4290, plain_cleanup_fn = 0x2ad3765ba680 <msc_pcre_cleanup>, child_cleanup_fn = 0x42c3b8 <apr_pool_cleanup_null@plt>}
(gdb) 
2363        while (c) {
1: **cref = {next = 0x161a37c8, data = 0x179b4290, plain_cleanup_fn = 0x2ad3765ba680 <msc_pcre_cleanup>, child_cleanup_fn = 0x42c3b8 <apr_pool_cleanup_null@plt>}
(gdb) 
2364            *cref = c->next;
1: **cref = {next = 0x161a37c8, data = 0x179b4290, plain_cleanup_fn = 0x2ad3765ba680 <msc_pcre_cleanup>, child_cleanup_fn = 0x42c3b8 <apr_pool_cleanup_null@plt>}
(gdb) 
2365            (*c->child_cleanup_fn)((void *)c->data);
1: **cref = {next = 0x161a37c8, data = 0x179b4290, plain_cleanup_fn = 0x2ad3765ba680 <msc_pcre_cleanup>, child_cleanup_fn = 0x42c3b8 <apr_pool_cleanup_null@plt>}
(gdb) 

(gdb) info threads
* 1 Thread 0x2ad37b11f940 (LWP 13772)  run_child_cleanups (cref=0x1606f728) at memory/unix/apr_pools.c:2365

(gdb) p *c->child_cleanup_fn
$10 = {apr_status_t (void *)} 0x42c3b8 <apr_pool_cleanup_null@plt>

gdb) 
apr_pool_cleanup_null (data=0x179b4290) at memory/unix/apr_pools.c:2405
2405        return APR_SUCCESS;
(gdb) list
2400    #endif /* !defined(WIN32) && !defined(OS2) */
2401
2402    APR_DECLARE_NONSTD(apr_status_t) apr_pool_cleanup_null(void *data)
2403    {
2404        /* do nothing cleanup routine */
2405        return APR_SUCCESS;
2406    }
2407
2408    /* Subprocesses don't use the generic cleanup interface because
2409     * we don't want multiple subprocesses to result in multiple

I should also note this all is happening when running in detectiononly mode.

[zimmerle]

Hi @celesteking,

thank you for the detailed description. What is your ModSecurity version? there are any public rules set loaded? any custom rules?

The version of the apr library which was dynamic loaded is the same that ModSecurity was compiled with ?

[celesteking]

2.8.0, apache 2.2 mpm worker fixed thread count, suphp. Yes, same version of apr library.

open crs ruleset with a couple of rules disabled and some minor adjustments:

# Disable GeoLookup
SecRuleRemoveById 900050
SecRuleRemoveById 900051

# Disable httprbl lookups
SecRuleRemoveById 981138
SecRuleRemoveById 981139

[celesteking]

Getting same behaviour with modsec 2.9.0 , same apache version (2.2) & config.

[celesteking]

Anyone having similar problems? Basically, the module is unusable at current state.

[cp0815]

we are currently experiencing the same behavior on a CentOS 7.1 System:
httpd-2.4.6-31.el7.centos.x86_64
mod_security_crs-2.2.6-6.el7.noarch
mod_security-2.7.3-5.el7.x86_64
It only happens in detectiononly mode and there are increasing amount of pid that are consuming cpu, but strace doesn't show up anything for them

[calebdelnayatgfs]

I ran into a similar problem as well. Processes consuming 100% CPU due to threads becoming stuck.

This is on a CentOS 7.1 CPanel server with Apache 2.4.16 running event MPM, mod_security version 2.9.0 with the OWASP ModSecurity Core Rule Set installed. Running in detection only mode.

Looking at the thread stacks with gdb revealed the stacks pasted below. I didn't do any further digging.

#0  0x00007fb7339f38c2 in parse_operator2 (sf=0x7fb7287f75c0) at libinjection/libinjection_sqli.c:562
#1  0x00007fb7339f5188 in libinjection_sqli_tokenize (sf=sf@entry=0x7fb7287f75c0) at libinjection/libinjection_sqli.c:1268
#2  0x00007fb7339f5b15 in libinjection_sqli_fold (sf=sf@entry=0x7fb7287f75c0) at libinjection/libinjection_sqli.c:1702
#3  0x00007fb7339f65a6 in libinjection_sqli_fingerprint (sql_state=0x7fb7287f75c0, flags=flags@entry=9)
    at libinjection/libinjection_sqli.c:1903
#4  0x00007fb7339f6b91 in libinjection_is_sqli (sql_state=sql_state@entry=0x7fb7287f75c0) at libinjection/libinjection_sqli.c:2255
#5  0x00007fb7339f6d26 in libinjection_sqli (input=<optimized out>, slen=<optimized out>, fingerprint=fingerprint@entry=0x7fb7287f7840 "")
    at libinjection/libinjection_sqli.c:2313
#6  0x00007fb733a1eda9 in msre_op_detectSQLi_execute (msr=0x7fb670022aa0, rule=0x1536af8, var=0x7fb6a809ebd0, error_msg=0x7fb7287f7910)
    at re_operators.c:2134
#7  0x00007fb733a19140 in execute_operator (var=var@entry=0x7fb6a809ebd0, rule=rule@entry=0x1536af8, msr=msr@entry=0x7fb670022aa0,
    acting_actionset=acting_actionset@entry=0xebfba0, mptmp=mptmp@entry=0x7fb6a809e488) at re.c:2650
#8  0x00007fb733a1a7a9 in msre_rule_process_normal (msr=<optimized out>, rule=0x1536af8) at re.c:3138
#9  msre_rule_process (msr=<optimized out>, rule=0x1536af8) at re.c:3353
#10 msre_ruleset_process_phase (ruleset=<optimized out>, msr=msr@entry=0x7fb670022aa0) at re.c:1773
#11 0x00007fb7339faa80 in modsecurity_process_phase_request_body (msr=0x7fb670022aa0) at modsecurity.c:569
#12 modsecurity_process_phase (msr=msr@entry=0x7fb670022aa0, phase=phase@entry=2) at modsecurity.c:815
#13 0x00007fb7339f9915 in hook_request_late (r=0x7fb670021340) at mod_security2.c:1055
#14 0x00000000004551fd in ap_run_fixups ()
#15 0x0000000000456ec9 in ap_process_request_internal ()
#16 0x00000000004a2936 in ap_process_async_request ()
#17 0x000000000049e281 in ap_process_http_async_connection ()
#18 0x000000000049e499 in ap_process_http_connection ()
#19 0x000000000047153a in ap_run_process_connection ()
#20 0x000000000052c578 in process_socket ()
#21 0x000000000052ee6a in worker_thread ()
#22 0x00007fb7375b6627 in dummy_worker () from /usr/local/apache/lib/libapr-1.so.0
#23 0x00007fb736d2edf5 in start_thread () from /lib64/libpthread.so.0
#24 0x00007fb7368581ad in clone () from /lib64/libc.so.6

#0  0x00007fb7339f3f68 in strlencspn (
    s=s@entry=0x7fb6e00de0a6 "member><name>methodName</name><value><string>wp.getCategories</string></value></member><member><name>params</name><value><array><data><value><string></string></value><value><string>REDACTED</strin"..., len=<optimized out>,
    accept=accept@entry=0x7fb733a580b0 " []{}<>:\\?=@!#~+-*/&|^%(),';\t\n\v\f\r\"\240") at libinjection/libinjection_sqli.c:189
#1  0x00007fb7339f4586 in parse_word (sf=0x7fb7197d95c0) at libinjection/libinjection_sqli.c:898
#2  0x00007fb7339f5188 in libinjection_sqli_tokenize (sf=sf@entry=0x7fb7197d95c0) at libinjection/libinjection_sqli.c:1268
#3  0x00007fb7339f5b15 in libinjection_sqli_fold (sf=sf@entry=0x7fb7197d95c0) at libinjection/libinjection_sqli.c:1702
#4  0x00007fb7339f65a6 in libinjection_sqli_fingerprint (sql_state=0x7fb7197d95c0, flags=flags@entry=9)
    at libinjection/libinjection_sqli.c:1903
#5  0x00007fb7339f6b91 in libinjection_is_sqli (sql_state=sql_state@entry=0x7fb7197d95c0) at libinjection/libinjection_sqli.c:2255
#6  0x00007fb7339f6d26 in libinjection_sqli (input=<optimized out>, slen=<optimized out>, fingerprint=fingerprint@entry=0x7fb7197d9840 "")
    at libinjection/libinjection_sqli.c:2313
#7  0x00007fb733a1eda9 in msre_op_detectSQLi_execute (msr=0x7fb6e804ab18, rule=0xb1e648, var=0x7fb6e0058a40, error_msg=0x7fb7197d9910)
    at re_operators.c:2134
#8  0x00007fb733a19140 in execute_operator (var=var@entry=0x7fb6e0058a40, rule=rule@entry=0xb1e648, msr=msr@entry=0x7fb6e804ab18,
    acting_actionset=acting_actionset@entry=0x12b6b80, mptmp=mptmp@entry=0x7fb6e00582f8) at re.c:2650
#9  0x00007fb733a1a7a9 in msre_rule_process_normal (msr=<optimized out>, rule=0xb1e648) at re.c:3138
#10 msre_rule_process (msr=<optimized out>, rule=0xb1e648) at re.c:3353
#11 msre_ruleset_process_phase (ruleset=<optimized out>, msr=msr@entry=0x7fb6e804ab18) at re.c:1773
#12 0x00007fb7339faa80 in modsecurity_process_phase_request_body (msr=0x7fb6e804ab18) at modsecurity.c:569
#13 modsecurity_process_phase (msr=msr@entry=0x7fb6e804ab18, phase=phase@entry=2) at modsecurity.c:815
#14 0x00007fb7339f9915 in hook_request_late (r=0x7fb6e80493c0) at mod_security2.c:1055
#15 0x00000000004551fd in ap_run_fixups ()
#16 0x0000000000456ec9 in ap_process_request_internal ()
#17 0x00000000004a2936 in ap_process_async_request ()
#18 0x000000000049e281 in ap_process_http_async_connection ()
#19 0x000000000049e499 in ap_process_http_connection ()
#20 0x000000000047153a in ap_run_process_connection ()
#21 0x000000000052c578 in process_socket ()
#22 0x000000000052ee6a in worker_thread ()
#23 0x00007fb7375b6627 in dummy_worker () from /usr/local/apache/lib/libapr-1.so.0
#24 0x00007fb736d2edf5 in start_thread () from /lib64/libpthread.so.0
#25 0x00007fb7368581ad in clone () from /lib64/libc.so.6

[hcanning2014]

I get strange CPU activity too. #991

[victorhora]

Please give it a try with release 2.9.2.

If the problem persists let us know.

Thanks.

[celesteking]

Problem persists.

EL6 latest, stock kernel.
ea-apache24-mod_security2-2.9.2-10.10.1.src.rpm

httpd  -V
Server version: Apache/2.4.37 (cPanel)
Server built:   Dec  3 2018 16:16:44
Server's Module Magic Number: 20120211:83
Server loaded:  APR 1.6.3, APR-UTIL 1.6.1
Compiled using: APR 1.6.3, APR-UTIL 1.6.1
Architecture:   64-bit
Server MPM:     worker
  threaded:     yes (fixed thread count)
    forked:     yes (variable process count)
Server compiled with....
 -D APR_HAS_SENDFILE
 -D APR_HAS_MMAP
 -D APR_HAVE_IPV6 (IPv4-mapped addresses disabled)
 -D APR_USE_SYSVSEM_SERIALIZE
 -D APR_USE_PTHREAD_SERIALIZE
 -D SINGLE_LISTEN_UNSERIALIZED_ACCEPT
 -D APR_HAS_OTHER_CHILD
 -D AP_HAVE_RELIABLE_PIPED_LOGS
 -D DYNAMIC_MODULE_LIMIT=256
 -D HTTPD_ROOT="/etc/apache2"
 -D SUEXEC_BIN="/usr/sbin/suexec"
 -D DEFAULT_PIDLOG="/var/run/apache2/httpd.pid"
 -D DEFAULT_SCOREBOARD="logs/apache_runtime_status"
 -D DEFAULT_ERRORLOG="logs/error_log"
 -D AP_TYPES_CONFIG_FILE="conf/mime.types"
 -D SERVER_CONFIG_FILE="conf/httpd.conf"

(gdb) bt
#0  run_child_cleanups (p=0x55b2a39c1138) at memory/unix/apr_pools.c:2642
#1  cleanup_pool_for_exec (p=0x55b2a39c1138) at memory/unix/apr_pools.c:2649
#2  0x00002ba5109da448 in cleanup_pool_for_exec (p=0x55b2a39c1138) at memory/unix/apr_pools.c:2652
#3  0x00002ba5109da448 in cleanup_pool_for_exec (p=0x55b2a39bf128) at memory/unix/apr_pools.c:2652
#4  0x00002ba5109e5227 in apr_proc_create (new=<value optimized out>, progname=0x2ba51b87dc59 "/usr/sbin/suphp", args=0x2ba5744155d8, env=0x2ba574402440, attr=0x2ba574402968, pool=0x2ba574417c98) at threadproc/unix/proc.c:425
#5  0x00002ba51b87c857 in suphp_script_handler (r=0x2ba574417d10) at mod_suphp.c:953
#6  0x00002ba51b87d3b0 in suphp_handler (r=0x2ba574417d10) at mod_suphp.c:569
#7  0x000055b2a18b50d0 in ap_run_handler (r=0x2ba574417d10) at config.c:171
#8  0x000055b2a18b96be in ap_invoke_handler (r=0x2ba574417d10) at config.c:444
#9  0x000055b2a18ceafa in ap_process_async_request (r=0x2ba574417d10) at http_request.c:453
#10 0x000055b2a18cec4e in ap_process_request (r=<value optimized out>) at http_request.c:488
#11 0x000055b2a18caa45 in ap_process_http_sync_connection (c=0x2ba524030078) at http_core.c:210
#12 ap_process_http_connection (c=0x2ba524030078) at http_core.c:251
#13 0x000055b2a18c09b0 in ap_run_process_connection (c=0x2ba524030078) at connection.c:42
#14 0x00002ba5117b7842 in process_socket (thd=0x55b2a65e2b98, dummy=<value optimized out>) at worker.c:479
#15 worker_thread (thd=0x55b2a65e2b98, dummy=<value optimized out>) at worker.c:808
#16 0x00002ba510bfaaa1 in start_thread () from /lib64/libpthread.so.0
#17 0x00002ba510ef8c4d in clone () from /lib64/libc.so.6
(gdb) 

# ps -ww -o cmd,wchan,rss,pcpu -p 4474
CMD                         WCHAN    RSS %CPU
/usr/sbin/httpd -k start    -      77620 10.1

[celesteking]

This might be related to mlogc being specified via SecAuditLog "|/usr/bin/mlogc /etc/apache2/conf/mlogc.conf". It looks like this is happening after graceful restart has been invoked. Maybe somehow modsec is impeding the pool cleanup? Or maybe there's a stack corruption of some sort?