Nginx 1.5.10 + ModSecurity 2.7.7: Process killed by segmentation fault at 10 #658

ThraaxSession · 2014-02-07T07:51:42Z

The process is killed by Signal 11. The software tries to allocate unassigned memory area.
kernel: [6062665.616763] nginx[8497]: segfault at 10 ip 00007fab654c2581 sp 00007fff401c4db0 error 4 in libapr-1.so.0.4.6[7fab654aa000+30000]

ThraaxSession · 2014-04-28T13:08:21Z

Hey guys! :)
I'd like to know if something new about the issue. Any feedback please.

Regards
Gino

zimmerle · 2014-04-28T14:04:28Z

Hi @GinoHereIam we are working on a refactoring of our nginx versions. Beta code is available at:

https://github.com/SpiderLabs/ModSecurity/tree/nginx_refactoring

Do you mind to test it?

ThraaxSession · 2014-04-28T14:10:36Z

I'll give feedback in the near future.
Hopefully it works.

DOSarrest · 2014-05-15T21:01:52Z

We are also seeing this problem. We have tested the nginx_refactoring branch and still observe the same problem...
kernel: nginx[7045]: segfault at 10 ip 00007f6aebb9f1c1 sp 00007fffe1b0f130 error 4 in libapr-1.so.0.4.8[7f6aebb86000+2f000]
This also causes problems with nginx...
[alert] 21527#0: worker process 5999 exited on signal 11
Making modsecurity for nginx VERY unstable. Our current testing is with nginx/1.4.7

zimmerle · 2014-05-15T21:08:38Z

Hi @DOSarrest, it would be great if you can share more details about this segfault.

Can you send me GDB output? while running the nginx_refactoring branch?

There are instructions on how to do it here:
https://github.com/SpiderLabs/ModSecurity/wiki/Reference-Manual

This GDB output will be very valuable to fix the problem.

Br.,
F.

DOSarrest · 2014-05-15T21:14:55Z

Hello @zimmerle, Yes, we are willing to do what ever we can to help resolve this.

We located the information on how to do it here...

https://github.com/SpiderLabs/ModSecurity/wiki/Debugging-ModSecurity

and we are going to go try and get you the information right now and will reply shortly.

zimmerle · 2014-05-15T21:17:01Z

Ops, Sorry I have placed the wrong link.

Thanks this GDB information will be very helpful.

DOSarrest · 2014-05-15T21:19:39Z

Np problem, Glad to help. If you don't mind, could you also check into #713 we have not had a response on that yet. It's related to nginx refactorying.

DOSarrest · 2014-05-15T22:16:39Z

Here is what I am getting currently...

# gdb nginx
GNU gdb (Gentoo 7.6.2 p1) 7.6.2
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-pc-linux-gnu".
For bug reporting instructions, please see:
<http://bugs.gentoo.org/>...
Reading symbols from /usr/sbin/nginx...(no debugging symbols found)...done.
(gdb) run
Starting program: /usr/sbin/nginx 
warning: Could not load shared library symbols for linux-vdso.so.1.
Do you need "set solib-search-path" or "set sysroot"?
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
nginx: [warn] duplicate MIME type "text/html" in /etc/nginx/nginx.conf:56

Program received signal SIGSEGV, Segmentation fault.
0x00007ffff703b1c1 in apr_global_mutex_lock () from /usr/lib64/libapr-1.so.0
(gdb) bt full
#0  0x00007ffff703b1c1 in apr_global_mutex_lock () from /usr/lib64/libapr-1.so.0
No symbol table info available.
#1  0x000000000056403e in ?? ()
No symbol table info available.
#2  0x0000000000562a0d in ?? ()
No symbol table info available.
#3  0x00000000005607a2 in ?? ()
No symbol table info available.
#4  0x000000000055b17a in modsecFinishRequest ()
No symbol table info available.
#5  0x0000000000532839 in ?? ()
No symbol table info available.
#6  0x0000000000423d09 in ngx_destroy_pool ()
No symbol table info available.
#7  0x0000000000476b31 in ngx_http_free_request ()
No symbol table info available.
#8  0x00000000004756a3 in ?? ()
No symbol table info available.
#9  0x0000000000474bc7 in ?? ()
No symbol table info available.
#10 0x000000000047482f in ngx_http_finalize_request ()
No symbol table info available.
#11 0x00000000004922e8 in ?? ()
No symbol table info available.
#12 0x000000000049153a in ?? ()
No symbol table info available.
#13 0x00000000004912a4 in ?? ()
No symbol table info available.
#14 0x000000000048fc63 in ?? ()
No symbol table info available.
#15 0x000000000048e133 in ?? ()
No symbol table info available.
#16 0x000000000048c9fb in ?? ()
No symbol table info available.
#17 0x0000000000455e9e in ?? ()
No symbol table info available.
#18 0x000000000044653c in ngx_process_events_and_timers ()
No symbol table info available.
#19 0x00000000004529c1 in ngx_single_process_cycle ()
No symbol table info available.
#20 0x0000000000420b20 in main ()
No symbol table info available.

DOSarrest · 2014-05-15T22:30:24Z

I'm trying to figure out how to get debugging symbols working

zimmerle · 2014-05-15T22:30:43Z

Did you had a change to re-compile, both, ModSec and nginx with its respective "debug" enablers?

If so, can you try a "make clean" (or start from a fresh directory) before: "./configure" ... "make install"

DOSarrest · 2014-05-15T22:49:16Z

I am using gentoo package management to handle the compilation process, and the main problem was that gentoo was stripping the binaries before installing them. I had modified the build process to stop stripping the binaries and have a better dump for you...

# gdb nginx
GNU gdb (Gentoo 7.6.2 p1) 7.6.2
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-pc-linux-gnu".
For bug reporting instructions, please see:
<http://bugs.gentoo.org/>...
Reading symbols from /usr/sbin/nginx...done.
(gdb) run
Starting program: /usr/sbin/nginx 
warning: Could not load shared library symbols for linux-vdso.so.1.
Do you need "set solib-search-path" or "set sysroot"?
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
nginx: [warn] duplicate MIME type "text/html" in /etc/nginx/nginx.conf:58

Program received signal SIGSEGV, Segmentation fault.
0x00007ffff703b1c1 in apr_global_mutex_lock () from /usr/lib64/libapr-1.so.0
(gdb) bt full
#0  0x00007ffff703b1c1 in apr_global_mutex_lock () from /usr/lib64/libapr-1.so.0
No symbol table info available.
#1  0x00000000005688cf in sec_audit_logger (msr=0x98ba8d0) at ../apache2/msc_logging.c:587
        arr = 0x0
        te = 0x0
        tarr_pattern = 0x0
        telts_pattern = 0x0
        str1 = 0x0
        str2 = 0x0
        text = 0x0
        rule = 0x0
        next_rule = 0x0
        nbytes = 160309048
        nbytes_written = 6053746
        md5hash = "\000\000\000\000\000\000\000\000\240\"\216\t\000\000\000"
        was_limited = 0
        present = 0
        wrote_response_body = 0
        entry_filename = 0x7fffffffcc00 "\020\004B"
        entry_basename = 0x7fffffffe250 "\001"
        rc = 0
        i = 0
        limit = 558
        k = 32767
        sanitized_partial = -173670503
        j = 32767
        buf = 0x0
        pat = 0x0
        mparm = 0x0
        arg_min = 32767
        arg_max = -170072608
        sanitize_matched = 32767
#2  0x0000000000565de1 in modsecurity_process_phase_logging (msr=0x98ba8d0) at ../apache2/modsecurity.c:701
        time_before = 1400194077820059
        time_after = 1400194077820082
#3  0x000000000056604c in modsecurity_process_phase (msr=0x98ba8d0, phase=5) at ../apache2/modsecurity.c:807
No locals.
#4  0x0000000000563be2 in hook_log_transaction (r=0x98b9b30) at ../apache2/mod_security2.c:1239
        arr = 0x98ba420
        origr = 0x98b9b30
        msr = 0x98ba8d0
#5  0x000000000055b8e5 in modsecFinishRequest (r=0x98b9b30) at api.c:656
No locals.
#6  0x00000000005327d9 in ngx_http_modsecurity_cleanup (data=0x93e1820) at /var/tmp/portage/www-servers/nginx-1.4.7/work/ModSecurity-nginx_refactoring/nginx/modsecurity/ngx_http_modsecurity.c:1535
        ctx = 0x93e1820
#7  0x0000000000423ca9 in ngx_destroy_pool (pool=0x93dfbf0) at src/core/ngx_palloc.c:55
        p = 0x2
        n = 0x8914c8
        l = 0x9914528
        c = 0x93e0bd8
#8  0x0000000000476ad1 in ngx_http_free_request (r=0x93dfc40, rc=0) at src/http/ngx_http_request.c:3391
        log = 0x98303d0
        pool = 0x93dfbf0
        linger = {l_onoff = 0, l_linger = 0}
        cln = 0x0
        ctx = 0x98304a0
        clcf = 0x0
#9  0x000000000047683c in ngx_http_close_request (r=0x93dfc40, rc=0) at src/http/ngx_http_request.c:3307
        c = 0x7ff9fef17390
#10 0x00000000004763a7 in ngx_http_lingering_close_handler (rev=0x7ff9fe922aa0) at src/http/ngx_http_request.c:3167
        n = 0
        timer = 30
        c = 0x7ff9fef17390
        r = 0x93dfc40
        clcf = 0x85e4b8 <cached_time+696>
        buffer = '\000' <repeats 64 times>, "\200\317\377\377\377\177", '\000' <repeats 14 times>, "\002\000\000\000\022\311\213\t", '\000' <repeats 396 times>...
#11 0x0000000000455e3e in ngx_epoll_process_events (cycle=0x88fac0, timer=425, flags=1) at src/event/modules/ngx_epoll_module.c:683
        events = 1
        revents = 21
        instance = 0
        i = 0
        level = 9021472
        err = 0
        rev = 0x7ff9fe922aa0
        wev = 0x7ff9fe32eaa0
        queue = 0x860340 <ngx_event_timer_sentinel>
        c = 0x7ff9fef17390
#12 0x00000000004464dc in ngx_process_events_and_timers (cycle=0x88fac0) at src/event/ngx_event.c:249
        flags = 1
        timer = 425
        delta = 1400194077819
#13 0x0000000000452961 in ngx_single_process_cycle (cycle=0x88fac0) at src/os/unix/ngx_process_cycle.c:315
        i = 70
#14 0x0000000000420ac0 in main (argc=1, argv=0x7fffffffe258) at src/core/nginx.c:409
        i = 70
        log = 0x85e120 <ngx_log>
        cycle = 0x88fac0
        init_cycle = {conf_ctx = 0x0, pool = 0x88eb10, log = 0x85e120 <ngx_log>, new_log = {log_level = 0, file = 0x0, connection = 0, handler = 0x0, data = 0x0, action = 0x0, priority = 0, facility = 0, syslog_on = 0, syslog_set = 0}, files = 0x0, 
          free_connections = 0x0, free_connection_n = 0, reusable_connections_queue = {prev = 0x0, next = 0x0}, listening = {elts = 0x0, nelts = 0, size = 0, nalloc = 0, pool = 0x0}, paths = {elts = 0x0, nelts = 0, size = 0, nalloc = 0, pool = 0x0}, open_files = {
            last = 0x0, part = {elts = 0x0, nelts = 0, next = 0x0}, size = 0, nalloc = 0, pool = 0x0}, shared_memory = {last = 0x0, part = {elts = 0x0, nelts = 0, next = 0x0}, size = 0, nalloc = 0, pool = 0x0}, connection_n = 0, files_n = 0, connections = 0x0, 
          read_events = 0x0, write_events = 0x0, old_cycle = 0x0, conf_file = {len = 21, data = 0x5bcc9a "/etc/nginx/nginx.conf"}, conf_param = {len = 0, data = 0x0}, conf_prefix = {len = 11, data = 0x5bcc9a "/etc/nginx/nginx.conf"}, prefix = {len = 5, 
            data = 0x5bcc94 "/usr/"}, lock_file = {len = 0, data = 0x0}, hostname = {len = 0, data = 0x0}}
        ccf = 0x890b70
(gdb)

zimmerle · 2014-05-15T23:18:13Z

I believe that "msr->modsecurity->auditlog_lock" is - for some reason - null or invalid at the moment that "sec_audit_logger" is called (Here: https://github.com/SpiderLabs/ModSecurity/blob/nginx_refactoring/apache2/msc_logging.c#L587).

It is declared here: https://github.com/SpiderLabs/ModSecurity/blob/nginx_refactoring/apache2/modsecurity.c#L122
as you can see it should return a error in case it does not able to create the mutex, but it is not being well handled at: https://github.com/SpiderLabs/ModSecurity/blob/nginx_refactoring/apache2/mod_security2.c#L663

Meanwhile, you can check the semaphores that you have running, by:

sudo ipcs -s

if you see too many, you can try to disabled it:

for i in `ipcs -s | awk '/some-user-name/ {print $2}'`; do echo "Removing $i..."; ipcrm -s $i; done

(Replace some-user-name with the user name displayed by "ipcs -s").

After that restart your nginx and let me know if this temporarily solve the problem.

DOSarrest · 2014-05-15T23:34:30Z

Nginx is started by root then switches to the nginx user/group. I cleared the semaphores for the root account and received two segfaults when I first ran nginx in gdb both looking like the following...

(gdb) run
Starting program: /usr/sbin/nginx
warning: Could not load shared library symbols for linux-vdso.so.1.
Do you need "set solib-search-path" or "set sysroot"?
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
nginx: [warn] duplicate MIME type "text/html" in /etc/nginx/nginx.conf:58

Program received signal SIGSEGV, Segmentation fault.
0x000000000042a807 in ngx_atoi (line=0x0, n=9038512) at src/core/ngx_string.c:865
865     src/core/ngx_string.c: No such file or directory.
(gdb) bt full
#0  0x000000000042a807 in ngx_atoi (line=0x0, n=9038512) at src/core/ngx_string.c:865
        value = 0
#1  0x000000000052f6ce in ngx_http_modsecurity_load_request (r=0x98b9ae0) at /var/tmp/portage/www-servers/nginx-1.4.7/work/ModSecurity-nginx_refactoring/nginx/modsecurity/ngx_http_modsecurity.c:329
        ctx = 0x98bb6d8
        req = 0x99be3b0
        str = {len = 9038513, data = 0x0}
        root = 9
        path = {len = 33, data = 0x98bb890 "/usr/html/ClientAccessPolicy.xml"}
#2  0x0000000000530c6c in ngx_http_modsecurity_process_request (r=0x98b9ae0) at /var/tmp/portage/www-servers/nginx-1.4.7/work/ModSecurity-nginx_refactoring/nginx/modsecurity/ngx_http_modsecurity.c:872
        rc = 0
        ctx = 0x98bb6d8
        load_request_body = 0
#3  0x0000000000531902 in ngx_http_modsecurity_handler (r=0x98b9ae0) at /var/tmp/portage/www-servers/nginx-1.4.7/work/ModSecurity-nginx_refactoring/nginx/modsecurity/ngx_http_modsecurity.c:1179
        rc = 0
        ctx = 0x98bb6d8
        cf = 0x173c858
#4  0x00000000004642e3 in ngx_http_core_generic_phase (r=0x98b9ae0, ph=0x9914630) at src/http/ngx_http_core_module.c:910
        rc = -5
#5  0x0000000000464232 in ngx_http_core_run_phases (r=0x98b9ae0) at src/http/ngx_http_core_module.c:888
        rc = -2
        ph = 0x99145a0
        cmcf = 0x8914c8
#6  0x00000000004641a0 in ngx_http_handler (r=0x98b9ae0) at src/http/ngx_http_core_module.c:871
        cmcf = 0x4f0000000089eafd
#7  0x0000000000473696 in ngx_http_process_request (r=0x98b9ae0) at src/http/ngx_http_request.c:1834
        c = 0x7ff9fef17690
#8  0x0000000000471f70 in ngx_http_process_request_headers (rev=0x7ff9fe922c40) at src/http/ngx_http_request.c:1265
        p = 0x98b9a90 "\215\252\213\t"
        len = 20
        n = 184
        rc = 0
        rv = 160147472
        h = 0x98bab80
        c = 0x7ff9fef17690
        hh = 0x0
        r = 0x98b9ae0
        cscf = 0xe4cba0
        cmcf = 0x8914c8
#9  0x000000000047130b in ngx_http_process_request_line (rev=0x7ff9fe922c40) at src/http/ngx_http_request.c:946
        n = 251
        rc = 0
        rv = 140711694576848
        host = {len = 17, data = 0x89ea9b "www.dosarrest.com:5555/ClientAccessPolicy.xml HTTP/1.1\r\nHost"}
        c = 0x7ff9fef17690
        r = 0x98b9ae0
#10 0x00000000004760ae in ngx_http_keepalive_handler (rev=0x7ff9fe922c40) at src/http/ngx_http_request.c:3085
        size = 1024
        n = 251
        b = 0x927b668
        c = 0x7ff9fef17690
#11 0x0000000000455e3e in ngx_epoll_process_events (cycle=0x88fac0, timer=181, flags=1) at src/event/modules/ngx_epoll_module.c:683
        events = 1
        revents = 5
        instance = 0
        i = 0
---Type <return> to continue, or q <return> to quit---
        level = 9021032
        err = 0
        rev = 0x7ff9fe922c40
        wev = 0x7ff9fef17510
        queue = 0x860340 <ngx_event_timer_sentinel>
        c = 0x7ff9fef17690
#12 0x00000000004464dc in ngx_process_events_and_timers (cycle=0x88fac0) at src/event/ngx_event.c:249
        flags = 1
        timer = 181
        delta = 1400196270759
#13 0x0000000000452961 in ngx_single_process_cycle (cycle=0x88fac0) at src/os/unix/ngx_process_cycle.c:315
        i = 70
#14 0x0000000000420ac0 in main (argc=1, argv=0x7fffffffe258) at src/core/nginx.c:409
        i = 70
        log = 0x85e120 <ngx_log>
        cycle = 0x88fac0
        init_cycle = {conf_ctx = 0x0, pool = 0x88eb10, log = 0x85e120 <ngx_log>, new_log = {log_level = 0, file = 0x0, connection = 0, handler = 0x0, data = 0x0, action = 0x0, priority = 0, facility = 0, syslog_on = 0, syslog_set = 0}, files = 0x0,
          free_connections = 0x0, free_connection_n = 0, reusable_connections_queue = {prev = 0x0, next = 0x0}, listening = {elts = 0x0, nelts = 0, size = 0, nalloc = 0, pool = 0x0}, paths = {elts = 0x0, nelts = 0, size = 0, nalloc = 0, pool = 0x0}, open_files = {
            last = 0x0, part = {elts = 0x0, nelts = 0, next = 0x0}, size = 0, nalloc = 0, pool = 0x0}, shared_memory = {last = 0x0, part = {elts = 0x0, nelts = 0, next = 0x0}, size = 0, nalloc = 0, pool = 0x0}, connection_n = 0, files_n = 0, connections = 0x0,
          read_events = 0x0, write_events = 0x0, old_cycle = 0x0, conf_file = {len = 21, data = 0x5bcc9a "/etc/nginx/nginx.conf"}, conf_param = {len = 0, data = 0x0}, conf_prefix = {len = 11, data = 0x5bcc9a "/etc/nginx/nginx.conf"}, prefix = {len = 5,
            data = 0x5bcc94 "/usr/"}, lock_file = {len = 0, data = 0x0}, hostname = {len = 0, data = 0x0}}
        ccf = 0x890b70
(gdb)

The third time I ran it, I have seen no segfaults yet, I am running a vulnerability test against the test location and it has generated over 2000 log entries so far without any errors. Typically we would see a segfault almost every session.

DOSarrest · 2014-05-15T23:40:28Z

Over 15000 log entries with 0 segfaults so far, so it does appear to have temporarily resolved the problem.

zimmerle · 2014-05-15T23:47:14Z

Cool. Once nginx segfault it left a semaphore orphan. There is a limitation on the on the number of semaphores, when the limit is reached it is not able to create new ones leading to other problems as the original problem that you have reported. I will investigate this second segfault.

Still have to patch ModSecurity to warn the user, saying that the semaphore, or, mutex was not created. Thus, ModSecurity will not start. A similar problem was reported regarding to the SecGeoLookupDb utilization, I believe that the nature of the problem is the same, i will investigate to link those two issues if applicable.

DOSarrest · 2014-05-16T00:10:05Z

Before removing the semaphores there was roughly 130 entries for root. On all of our production servers running only nginx they all have about the same 130 entries. Our production servers have not run nginx with modsecurity yet, and they had any segfaults.

Our production servers handle a lot of traffic, could this be a issue of needing to increase the kernel.sem parameter to adequately handle our data levels when adding modsecurity into the equation?

We have the current settings in our test environment for this...

# ipcs -ls

------ Semaphore Limits --------
max number of arrays = 128
max semaphores per array = 250
max semaphores system wide = 32000
max ops per semop call = 32
semaphore max value = 32767

zimmerle · 2017-05-09T14:27:09Z

No longer a problem in libModSecurity. Marking this as won't fix in 2.x.

Please check further information on libModSecurity here:
https://github.com/SpiderLabs/ModSecurity/tree/v3/master

zimmerle added the Platform - Nginx label Feb 7, 2014

rcbarnett-zz added Type - Feature and removed Type - Feature labels Feb 12, 2014

ThraaxSession mentioned this issue Mar 31, 2014

Nginx 1.5.10 + ModSecurity 2.7.7 Process killed by segmentation fault at 10 #657

Closed

zimmerle mentioned this issue May 16, 2014

proxy abuse detection failing under 2.8.0 and nginx 1.4.x, 1.6.0 #710

Closed

zimmerle added the TBF by libmodsec label Oct 21, 2016

zimmerle self-assigned this Oct 21, 2016

zimmerle closed this as completed May 9, 2017

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Nginx 1.5.10 + ModSecurity 2.7.7: Process killed by segmentation fault at 10 #658

Nginx 1.5.10 + ModSecurity 2.7.7: Process killed by segmentation fault at 10 #658

ThraaxSession commented Feb 7, 2014

ThraaxSession commented Apr 28, 2014

zimmerle commented Apr 28, 2014

ThraaxSession commented Apr 28, 2014

DOSarrest commented May 15, 2014

zimmerle commented May 15, 2014

DOSarrest commented May 15, 2014

zimmerle commented May 15, 2014

DOSarrest commented May 15, 2014

DOSarrest commented May 15, 2014

DOSarrest commented May 15, 2014

zimmerle commented May 15, 2014

DOSarrest commented May 15, 2014

zimmerle commented May 15, 2014

DOSarrest commented May 15, 2014

DOSarrest commented May 15, 2014

zimmerle commented May 15, 2014

DOSarrest commented May 16, 2014

zimmerle commented May 9, 2017

Nginx 1.5.10 + ModSecurity 2.7.7: Process killed by segmentation fault at 10 #658

Nginx 1.5.10 + ModSecurity 2.7.7: Process killed by segmentation fault at 10 #658

Comments

ThraaxSession commented Feb 7, 2014

ThraaxSession commented Apr 28, 2014

zimmerle commented Apr 28, 2014

ThraaxSession commented Apr 28, 2014

DOSarrest commented May 15, 2014

zimmerle commented May 15, 2014

DOSarrest commented May 15, 2014

zimmerle commented May 15, 2014

DOSarrest commented May 15, 2014

DOSarrest commented May 15, 2014

DOSarrest commented May 15, 2014

zimmerle commented May 15, 2014

DOSarrest commented May 15, 2014

zimmerle commented May 15, 2014

DOSarrest commented May 15, 2014

DOSarrest commented May 15, 2014

zimmerle commented May 15, 2014

DOSarrest commented May 16, 2014

zimmerle commented May 9, 2017