New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Nginx workers segfault with nginx 1.9.5, naxsi 0.54 if http2 is enabled #227

Closed
selivan opened this Issue Sep 30, 2015 · 113 comments

Comments

Projects
None yet
@selivan
Contributor

selivan commented Sep 30, 2015

UPD: Problem occurs only if http2 is enabled in listen directive.

Here are core dump files, apport crash report and manulay built packages I used on Ubuntu 14.04: https://yadi.sk/d/6m32n8IFjRqrc

nasxi 0.54

nginx -V
nginx version: nginx/1.9.5
built with OpenSSL 1.0.1f 6 Jan 2014
TLS SNI support enabled
configure arguments: --add-module=/home/selivan/work/naxsi/nginx-1.9.5-custom/debian/modules/naxsi/naxsi_src/ --add-module=/home/selivan/work/naxsi/nginx-1.9.5-custom/debian/modules/nginx-upstream-fair/ --with-cc-opt='-g -O2 -fPIE -fstack-protector --param=ssp-buffer-size=4 -Wformat -Werror=format-security -D_FORTIFY_SOURCE=2' --with-ld-opt='-Wl,-Bsymbolic-functions -fPIE -pie -Wl,-z,relro -Wl,-z,now' --prefix=/usr/share/nginx --conf-path=/etc/nginx/nginx.conf --http-log-path=/var/log/nginx/access.log --error-log-path=/var/log/nginx/error.log --lock-path=/var/lock/nginx.lock --pid-path=/run/nginx.pid --http-client-body-temp-path=/var/lib/nginx/body --http-fastcgi-temp-path=/var/lib/nginx/fastcgi --http-proxy-temp-path=/var/lib/nginx/proxy --http-scgi-temp-path=/var/lib/nginx/scgi --http-uwsgi-temp-path=/var/lib/nginx/uwsgi --with-debug --with-pcre-jit --with-ipv6 --with-http_ssl_module --with-http_stub_status_module --with-http_realip_module --with-http_auth_request_module --with-http_addition_module --with-http_geoip_module --with-http_gunzip_module --with-http_gzip_static_module --with-http_sub_module --with-http_xslt_module --without-mail_pop3_module --without-mail_imap_module --without-mail_smtp_module --with-http_v2_module

Only module I used except naxsi is nginx-upstream-fair.

Problem appeared after couple of minutes of nginx working with production load, synthetic tests did not reproduce it.
nginx.conf:

http {
        # Naxsi
        include /etc/nginx/naxsi_core.rules;
...
server {
        server_name *********;
        include           /etc/nginx/naxsi.rules;
        set $naxsi_extensive_log 1;
        location /RequestDenied {
            return 418;
            access_log /var/log/nginx/denied.log;
        }
...

naxsi_core.rules taken unchanged from source.

naxsi.rules:

#Enables learning mode
LearningMode;
SecRulesEnabled;
#SecRulesDisabled;
DeniedUrl "/RequestDenied";
## check rules
CheckRule "$SQL >= 8" BLOCK;
CheckRule "$RFI >= 8" BLOCK;
CheckRule "$TRAVERSAL >= 4" BLOCK;
CheckRule "$EVADE >= 4" BLOCK;
CheckRule "$XSS >= 8" BLOCK;

Problem also appeared with naxsi 0.54rc3, which I accidentally built first.

@selivan

This comment has been minimized.

Contributor

selivan commented Sep 30, 2015

gdb /usr/sbin/nginx -c /var/lib/nginx/cores/core
...
Reading symbols from /usr/sbin/nginx...Reading symbols from /usr/lib/debug/.build-id/98/189920b31b8739442099c12774ed5f5a5e04f7.debug...done.
done.
[New LWP 20339]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `nginx: worker process                           '.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x00007feb1dabaf25 in nx_find_wl_in_hash (mstr=mstr@entry=0x7feb1f4449e8, cf=cf@entry=0x7feb1f049018, zone=zone@entry=HEADERS)
    at /home/selivan/work/naxsi/nginx-1.9.5-custom/debian/modules/naxsi/naxsi_src//naxsi_runtime.c:352
352         mstr->data[i] = tolower(mstr->data[i]);

Backtrace:

(gdb) bt
#0  0x00007feb1dabaf25 in nx_find_wl_in_hash (mstr=mstr@entry=0x7feb1f4449e8, cf=cf@entry=0x7feb1f049018, zone=zone@entry=HEADERS)
    at /home/selivan/work/naxsi/nginx-1.9.5-custom/debian/modules/naxsi/naxsi_src//naxsi_runtime.c:352
#1  0x00007feb1dabb4ca in ngx_http_dummy_is_rule_whitelisted_n (req=req@entry=0x7feb1f443c90, cf=cf@entry=0x7feb1f049018, 
    r=r@entry=0x7feb1f030178, name=name@entry=0x7feb1f4449e8, zone=zone@entry=HEADERS, target_name=target_name@entry=0)
    at /home/selivan/work/naxsi/nginx-1.9.5-custom/debian/modules/naxsi/naxsi_src//naxsi_runtime.c:641
#2  0x00007feb1dabcad5 in ngx_http_apply_rulematch_v_n (r=r@entry=0x7feb1f030178, ctx=ctx@entry=0x7feb1f45dff8, req=req@entry=0x7feb1f443c90, 
    name=name@entry=0x7feb1f4449e8, value=value@entry=0x7feb1f4449f8, zone=zone@entry=HEADERS, nb_match=1, target_name=target_name@entry=0)
    at /home/selivan/work/naxsi/nginx-1.9.5-custom/debian/modules/naxsi/naxsi_src//naxsi_runtime.c:1049
#3  0x00007feb1dabd40f in ngx_http_basestr_ruleset_n (pool=<optimized out>, name=name@entry=0x7feb1f4449e8, value=value@entry=0x7feb1f4449f8, 
    rules=0x7feb1f01ce20, req=req@entry=0x7feb1f443c90, ctx=ctx@entry=0x7feb1f45dff8, zone=zone@entry=HEADERS)
    at /home/selivan/work/naxsi/nginx-1.9.5-custom/debian/modules/naxsi/naxsi_src//naxsi_runtime.c:1429
#4  0x00007feb1dabd7f3 in ngx_http_dummy_headers_parse (main_cf=main_cf@entry=0x7feb1f008e08, cf=cf@entry=0x7feb1f049018, 
    ctx=ctx@entry=0x7feb1f45dff8, r=r@entry=0x7feb1f443c90)
    at /home/selivan/work/naxsi/nginx-1.9.5-custom/debian/modules/naxsi/naxsi_src//naxsi_runtime.c:2130
#5  0x00007feb1dabef8e in ngx_http_dummy_data_parse (ctx=ctx@entry=0x7feb1f45dff8, r=r@entry=0x7feb1f443c90)
    at /home/selivan/work/naxsi/nginx-1.9.5-custom/debian/modules/naxsi/naxsi_src//naxsi_runtime.c:2154
#6  0x00007feb1dac1020 in ngx_http_dummy_access_handler (r=0x7feb1f443c90)
    at /home/selivan/work/naxsi/nginx-1.9.5-custom/debian/modules/naxsi/naxsi_src//naxsi_skeleton.c:1247
#7  0x00007feb1da5c743 in ngx_http_core_rewrite_phase (r=0x7feb1f443c90, ph=0x7feb1f0ca1f8) at src/http/ngx_http_core_module.c:894
#8  0x00007feb1da57f05 in ngx_http_core_run_phases (r=r@entry=0x7feb1f443c90) at src/http/ngx_http_core_module.c:840
#9  0x00007feb1da57ff7 in ngx_http_handler (r=r@entry=0x7feb1f443c90) at src/http/ngx_http_core_module.c:823
#10 0x00007feb1da6401e in ngx_http_process_request (r=0x7feb1f443c90) at src/http/ngx_http_request.c:1901
#11 0x00007feb1da95958 in ngx_http_v2_run_request (r=0x7feb1f443c90) at src/http/v2/ngx_http_v2.c:3445
#12 ngx_http_v2_state_header_complete (h2c=0x7feb1f3de700, pos=0x7feb1f12ab33 "", end=0x7feb1f12ab33 "") at src/http/v2/ngx_http_v2.c:1704
#13 0x00007feb1da96ad1 in ngx_http_v2_state_header_block (h2c=0x7feb1f3de700, pos=0x7feb1f12ab33 "", end=0x7feb1f12ab33 "")
    at src/http/v2/ngx_http_v2.c:1266
#14 0x00007feb1da92e6d in ngx_http_v2_read_handler (rev=0x7feb1f0fe790) at src/http/v2/ngx_http_v2.c:357
#15 0x00007feb1da4d2d1 in ngx_epoll_process_events (cycle=0x7feb1f005020, timer=<optimized out>, flags=<optimized out>)
    at src/event/modules/ngx_epoll_module.c:822
#16 0x00007feb1da43a7a in ngx_process_events_and_timers (cycle=cycle@entry=0x7feb1f005020) at src/event/ngx_event.c:242
#17 0x00007feb1da4aa85 in ngx_worker_process_cycle (cycle=cycle@entry=0x7feb1f005020, data=data@entry=0x0) at src/os/unix/ngx_process_cycle.c:753
#18 0x00007feb1da4949a in ngx_spawn_process (cycle=cycle@entry=0x7feb1f005020, proc=proc@entry=0x7feb1da4aa30 <ngx_worker_process_cycle>, 
    data=data@entry=0x0, name=name@entry=0x7feb1dace476 "worker process", respawn=respawn@entry=-3) at src/os/unix/ngx_process.c:198
#19 0x00007feb1da4ad30 in ngx_start_worker_processes (cycle=cycle@entry=0x7feb1f005020, n=1, type=type@entry=-3)
    at src/os/unix/ngx_process_cycle.c:358
#20 0x00007feb1da4bb4f in ngx_master_process_cycle (cycle=0x7feb1f005020) at src/os/unix/ngx_process_cycle.c:130
#21 0x00007feb1da27ac4 in main (argc=<optimized out>, argv=<optimized out>) at src/core/nginx.c:415
@blotus

This comment has been minimized.

Contributor

blotus commented Sep 30, 2015

Hi,

From the full stacktrace, it seems that you are using HTTP/2, and naxsi has not been tested yet with it (we will start working on it soon).
Can you try to remove the http2 parameter from the listen directive in your configuration and see if the crash still occurs ?

@selivan

This comment has been minimized.

Contributor

selivan commented Sep 30, 2015

Seems that removing http2 fixed the problem, thank you :)

@selivan selivan changed the title from Nginx workers segfault with nginx 1.9.5, naxsi 0.54 to Nginx workers segfault with nginx 1.9.5, naxsi 0.54 if http2 is enabled Sep 30, 2015

@bbigras

This comment has been minimized.

bbigras commented Oct 28, 2015

Maybe related to the "Bugfix: a segmentation fault might occur in a worker process when using HTTP/2." in the latest nginx version.
http://nginx.org/en/CHANGES

I'll test tomorrow.

@rfnx

This comment has been minimized.

rfnx commented Oct 28, 2015

I just tested it and unfortunately it is still not working.

@rfnx

This comment has been minimized.

rfnx commented Nov 18, 2015

Same issue with nginx 1.9.7

EDIT : same with 1.9.8 & 1.9.9

@rfnx

This comment has been minimized.

rfnx commented Dec 20, 2015

@blotus : Hello, is there any news on this issue ? I really want to use naxsi with http2 :)

@buixor

This comment has been minimized.

Member

buixor commented Dec 21, 2015

Hi @rfnx :)

We didn't start working on the http2 compat.
I hope we will find some time in Jan to work on it, but it is not going to come that soon. http/2 brings a lot of new things that will definitely have serious impacts :)

@rfnx

This comment has been minimized.

rfnx commented Dec 21, 2015

@buixor Thanks for the answer !

@selivan

This comment has been minimized.

Contributor

selivan commented Feb 18, 2016

@buixor: may be you could insert some warning or even error if naxsi is used with HTTP/2? It would be a small and easy change. Now it may be a bit confusing for new users.

@123BLiN

This comment has been minimized.

123BLiN commented Feb 24, 2016

This is so sad :(

@buixor

This comment has been minimized.

Member

buixor commented Feb 24, 2016

Hi,

Sorry for delay :)
Yes, we are going to add a warning at least. Sorry, I didn't really have time to look at the subject in depth, I just already know it's going to be tricky as http/2 seems to have a lot of tricky things (from the RFC) that might have security implication.

I'll keep you posted ;)

@Promaethius

This comment has been minimized.

Promaethius commented Mar 20, 2016

Hate to make this thread sound like a broken record, but do you have any news concerning http2 compatibility?

@buixor

This comment has been minimized.

Member

buixor commented Mar 21, 2016

Hi,

Don't worry, you request is totally legit.
No news yet, except I started looking at it (no code written yet), I will try to squeeze some time to work on it, but it's definitely in my plans :)

@TheDeadCode

This comment has been minimized.

TheDeadCode commented Apr 25, 2016

+1

1 similar comment
@aledbf

This comment has been minimized.

aledbf commented Apr 25, 2016

+1

@AnoopAlias

This comment has been minimized.

AnoopAlias commented Apr 26, 2016

Can you let me the status of this now that nginx 1.10.0 stable is released . I believe the bug affects this version too?

@selivan

This comment has been minimized.

Contributor

selivan commented Apr 26, 2016

Yep, unlike 1.9.x, which was mainline version, 1.10 is stable version, like 1.8 was. Is there anything you guys are going to do with it? I would even pay money for it (if I could convince my bosses that it's good investment).

@buixor

This comment has been minimized.

Member

buixor commented Apr 27, 2016

Hello,

This is indeed something we need to work on, but as you might understand, we'd better not take the topic lightly. I plan to work on it as soon as we are done with release 0.55 (we have a few open bugs to close first), but tbh we didn't start working on http2 yet.

It will come, I know some of you would have hoped sooner, please a bit more patience.
I will come back to you as soon as we have something testable ;)

@buixor

This comment has been minimized.

Member

buixor commented Apr 28, 2016

Hi,

just started to do some basic tests with http2, and it seems to be working so far.
If any of you can possibly submit a test that triggers any bug, please do so !

@AnoopAlias

This comment has been minimized.

AnoopAlias commented Apr 29, 2016

I am using vers=0.55rc1 with nginx-1.10.0 and having http2 enabled in ssl vhost.

So far I have not seen any crashes and a test was blocked by naxsi just fine

2016/04/28 22:02:00 [error] 1463#1463: *891 NAXSI_FMT: ip=122.174.198.241&server=domain.net&uri=/index.php&learning=0&vers=0.55rc1&total_processed=1&total_blocked=1&block=1&cscore0=$XSS&score0=8&zone0=ARGS&id0=1302&var_name0=a, client: 122.174.198.241, server: domain.net, request: "GET /index.php?a=%3C%3E HTTP/2.0", host: "domain.net"

@Promaethius

This comment has been minimized.

Promaethius commented Apr 29, 2016

The initial comment clarified: "Problem appeared after couple of minutes of nginx working with production load, synthetic tests did not reproduce it." This would make debugging the issue difficult.

@buixor

This comment has been minimized.

Member

buixor commented Apr 29, 2016

@Promaethius : yes exactly, that's why I'm begging for some test cases.
@AnoopAlias : thanks, my initial tests seem to pass as well

@buixor

This comment has been minimized.

Member

buixor commented Apr 29, 2016

@selivan : any chance you have more indications on what caused the crash ?

@selivan

This comment has been minimized.

Contributor

selivan commented Apr 29, 2016

@buixor: was long time ago, all that I saved is in this ticket. And I can't test new build on production load now, but I'll see if I can use it for 5-10 minutes on one of backends some days later.

@rfnx

This comment has been minimized.

rfnx commented Apr 29, 2016

@buixor Hello, thanks again for your work. I tried and naxsi still crashed. This is very easy to reproduce : for me, it happens everytime I click the "connection" button on my wordpress site, to go to the admin connection page (default to /wp-admin).

Configuration :

  • nginx version : 1.10.0;
  • naxsi version : 0.55rc1;
  • http/2 enabled;
  • Linux kernel with grsecurity.

My system log after the crash :

kernel: grsec: From 127.0.0.6: Segmentation fault occurred at 0000000000ad37e4 in /usr/bin/nginx[nginx:11381] uid/euid:33/33 gid/egid:33/33, parent /usr/bin/nginx[nginx:11379] uid/euid:0/0 gid/egid:0/0
kernel: grsec: From 127.0.0.6: bruteforce prevention initiated for the next 30 minutes or until service restarted, stalling each fork 30 seconds.  Please investigate the crash report for /usr/bin/nginx[nginx:11381] uid/euid:33/33 gid/egid:33/33, parent /usr/bin/nginx[nginx:11379] uid/euid:0/0 gid/egid:0/0
systemd-coredump[11419]: Process 11381 (nginx) of user 33 dumped core.

                                                        Stack trace of thread 11381:
                                                        #0  0x00000000004c5964 nx_find_wl_in_hash (nginx)
                                                        #1  0x00000000004c5e85 ngx_http_dummy_is_rule_whitelisted_n (nginx)
                                                        #2  0x00000000004c7599 ngx_http_apply_rulematch_v_n (nginx)
                                                        #3  0x00000000004c7ee2 ngx_http_basestr_ruleset_n (nginx)
                                                        #4  0x00000000004c82c5 ngx_http_dummy_headers_parse (nginx)
                                                        #5  0x00000000004c9980 ngx_http_dummy_data_parse (nginx)
                                                        #6  0x00000000004cbc9e n/a (nginx)
                                                        #7  0x00000000004563fc ngx_http_core_rewrite_phase (nginx)
                                                        #8  0x0000000000451895 ngx_http_core_run_phases (nginx)
                                                        #9  0x000000000045c973 ngx_http_process_request (nginx)
                                                        #10 0x00000000004890a6 n/a (nginx)
                                                        #11 0x0000000000489d76 n/a (nginx)
                                                        #12 0x0000000000489fbe n/a (nginx)
                                                        #13 0x0000000000488855 n/a (nginx)
                                                        #14 0x000000000043e750 ngx_event_process_posted (nginx)
                                                        #15 0x0000000000444921 n/a (nginx)
                                                        #16 0x0000000000443350 ngx_spawn_process (nginx)
                                                        #17 0x0000000000444c84 n/a (nginx)
                                                        #18 0x00000000004455a4 ngx_master_process_cycle (nginx)
                                                        #19 0x0000000000421838 main (nginx)
                                                        #20 0x0000031d74329710 __libc_start_main (libc.so.6)
                                                        #21 0x0000000000421d09 _start (nginx)

                                                        Stack trace of thread 11392:
                                                        #0  0x0000031d761333e8 pthread_cond_timedwait@@GLIBC_2.3.2 (libpthread.so.0)
                                                        #1  0x00000000006f3095 n/a (nginx)
                                                        #2  0x00000000006efb3a n/a (nginx)
                                                        #3  0x000000000050b86e n/a (nginx)
                                                        #4  0x00000000006f4588 n/a (nginx)
                                                        #5  0x0000031d7612d424 start_thread (libpthread.so.0)
                                                        #6  0x0000031d743f0cbd __clone (libc.so.6)
@buixor

This comment has been minimized.

Member

buixor commented Apr 30, 2016

Thanks :)

Any chance you could provide a dump of the http request?
On 29 Apr 2016 22:53, "rfnx" notifications@github.com wrote:

@buixor https://github.com/buixor Hello, thanks again for your work. I
tried again and I still have a crash with naxsi. This is very easy to
reproduce : for me, it happens everytime I click the "connection" button on
my wordpress site, to go to the admin connection page (default to
/wp-admin).

Configuration :

  • nginx version : 1.10.0;
  • naxsi version : 0.55rc1;
  • http/2 enabled;
  • Linux kernel with grsecurity.

My system log after the crash :

kernel: grsec: From 127.0.0.6: Segmentation fault occurred at 0000000000ad37e4 in /usr/bin/nginx[nginx:11381] uid/euid:33/33 gid/egid:33/33, parent /usr/bin/nginx[nginx:11379] uid/euid:0/0 gid/egid:0/0
kernel: grsec: From 127.0.0.6: bruteforce prevention initiated for the next 30 minutes or until service restarted, stalling each fork 30 seconds. Please investigate the crash report for /usr/bin/nginx[nginx:11381] uid/euid:33/33 gid/egid:33/33, parent /usr/bin/nginx[nginx:11379] uid/euid:0/0 gid/egid:0/0
systemd-coredump[11419]: Process 11381 (nginx) of user 33 dumped core.

                                                    Stack trace of thread 11381:
                                                    #0  0x00000000004c5964 nx_find_wl_in_hash (nginx)
                                                    #1  0x00000000004c5e85 ngx_http_dummy_is_rule_whitelisted_n (nginx)
                                                    #2  0x00000000004c7599 ngx_http_apply_rulematch_v_n (nginx)
                                                    #3  0x00000000004c7ee2 ngx_http_basestr_ruleset_n (nginx)
                                                    #4  0x00000000004c82c5 ngx_http_dummy_headers_parse (nginx)
                                                    #5  0x00000000004c9980 ngx_http_dummy_data_parse (nginx)
                                                    #6  0x00000000004cbc9e n/a (nginx)
                                                    #7  0x00000000004563fc ngx_http_core_rewrite_phase (nginx)
                                                    #8  0x0000000000451895 ngx_http_core_run_phases (nginx)
                                                    #9  0x000000000045c973 ngx_http_process_request (nginx)
                                                    #10 0x00000000004890a6 n/a (nginx)
                                                    #11 0x0000000000489d76 n/a (nginx)
                                                    #12 0x0000000000489fbe n/a (nginx)
                                                    #13 0x0000000000488855 n/a (nginx)
                                                    #14 0x000000000043e750 ngx_event_process_posted (nginx)
                                                    #15 0x0000000000444921 n/a (nginx)
                                                    #16 0x0000000000443350 ngx_spawn_process (nginx)
                                                    #17 0x0000000000444c84 n/a (nginx)
                                                    #18 0x00000000004455a4 ngx_master_process_cycle (nginx)
                                                    #19 0x0000000000421838 main (nginx)
                                                    #20 0x0000031d74329710 __libc_start_main (libc.so.6)
                                                    #21 0x0000000000421d09 _start (nginx)

                                                    Stack trace of thread 11392:
                                                    #0  0x0000031d761333e8 pthread_cond_timedwait@@GLIBC_2.3.2 (libpthread.so.0)
                                                    #1  0x00000000006f3095 n/a (nginx)
                                                    #2  0x00000000006efb3a n/a (nginx)
                                                    #3  0x000000000050b86e n/a (nginx)
                                                    #4  0x00000000006f4588 n/a (nginx)
                                                    #5  0x0000031d7612d424 start_thread (libpthread.so.0)
                                                    #6  0x0000031d743f0cbd __clone (libc.so.6)


You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
#227 (comment)

@rfnx

This comment has been minimized.

rfnx commented Apr 30, 2016

@buixor Do you want the headers ?

@bbigras

This comment has been minimized.

bbigras commented May 2, 2016

I also have had a crash. I don't know how to reproduce it yet.

Reading symbols from /usr/sbin/nginx...Reading symbols from /usr/lib/debug/.build-id/7c/6f589235091c270b13731c005969fc22f4c29d.debug...done.
done.
[New LWP 30858]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/i386-linux-gnu/libthread_db.so.1".
Core was generated by `nginx: worker process                   '.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x080f880e in nx_find_wl_in_hash (mstr=mstr@entry=0x8823150, cf=cf@entry=0x86ea644, zone=zone@entry=HEADERS) at /home/bbigras/nginx/nginx-1.9.15/debian/modules/naxsi/naxsi_src/naxsi_runtime.c:342
342         mstr->data[i] = tolower(mstr->data[i]);
(gdb) bt
#0  0x080f880e in nx_find_wl_in_hash (mstr=mstr@entry=0x8823150, cf=cf@entry=0x86ea644, zone=zone@entry=HEADERS) at /home/bbigras/nginx/nginx-1.9.15/debian/modules/naxsi/naxsi_src/naxsi_runtime.c:342
#1  0x080f8d5c in ngx_http_dummy_is_rule_whitelisted_n (req=req@entry=0x88225c8, cf=cf@entry=0x86ea644, r=r@entry=0x86c32a4, name=name@entry=0x8823150, zone=zone@entry=HEADERS,
    target_name=target_name@entry=0) at /home/bbigras/nginx/nginx-1.9.15/debian/modules/naxsi/naxsi_src/naxsi_runtime.c:629
#2  0x080fa5b7 in ngx_http_apply_rulematch_v_n (r=r@entry=0x86c32a4, ctx=ctx@entry=0x882330c, req=req@entry=0x88225c8, name=name@entry=0x8823150, value=value@entry=0x8823158, zone=zone@entry=HEADERS,
    nb_match=0, target_name=target_name@entry=0) at /home/bbigras/nginx/nginx-1.9.15/debian/modules/naxsi/naxsi_src/naxsi_runtime.c:1030
#3  0x080faf0c in ngx_http_basestr_ruleset_n (pool=0x88225a0, name=name@entry=0x8823150, value=value@entry=0x8823158, rules=0x86bd8c0, req=req@entry=0x88225c8, ctx=ctx@entry=0x882330c,
    zone=zone@entry=HEADERS) at /home/bbigras/nginx/nginx-1.9.15/debian/modules/naxsi/naxsi_src/naxsi_runtime.c:1397
#4  0x080fb2e3 in ngx_http_dummy_headers_parse (main_cf=main_cf@entry=0x85b8598, cf=cf@entry=0x86ea644, ctx=ctx@entry=0x882330c, r=r@entry=0x88225c8)
    at /home/bbigras/nginx/nginx-1.9.15/debian/modules/naxsi/naxsi_src/naxsi_runtime.c:2078
#5  0x080fcd98 in ngx_http_dummy_data_parse (ctx=ctx@entry=0x882330c, r=r@entry=0x88225c8) at /home/bbigras/nginx/nginx-1.9.15/debian/modules/naxsi/naxsi_src/naxsi_runtime.c:2102
#6  0x080fed3f in ngx_http_dummy_access_handler (r=0x88225c8) at /home/bbigras/nginx/nginx-1.9.15/debian/modules/naxsi/naxsi_src/naxsi_skeleton.c:1228
#7  0x080973c3 in ngx_http_core_rewrite_phase (r=0x88225c8, ph=0x879df94) at src/http/ngx_http_core_module.c:901
#8  0x08092e29 in ngx_http_core_run_phases (r=0x88225c8) at src/http/ngx_http_core_module.c:847
#9  0x080c62f8 in ngx_http_v2_process_request_body (r=0x88225c8, pos=pos@entry=0x87bf711 "{\"Nom\":\"M05021\",\"Type\":\"NonPeinte\"}\320b\r&=LtA\352\373$\343\261\005L\034\067\341Y\357Ր؂ڇ",
    size=size@entry=35, last=1) at src/http/v2/ngx_http_v2.c:3587
#10 0x080c7513 in ngx_http_v2_state_read_data (h2c=0x86a5720, pos=0x87bf711 "{\"Nom\":\"M05021\",\"Type\":\"NonPeinte\"}\320b\r&=LtA\352\373$\343\261\005L\034\067\341Y\357Ր؂ڇ",
    end=0x87bf734 "\320b\r&=LtA\352\373$\343\261\005L\034\067\341Y\357Ր؂ڇ") at src/http/v2/ngx_http_v2.c:922
#11 0x080c8a0a in ngx_http_v2_read_handler (rev=<optimized out>) at src/http/v2/ngx_http_v2.c:362
#12 0x0807f78c in ngx_event_process_posted (cycle=<optimized out>, cycle@entry=0x85b64b8, posted=<optimized out>) at src/event/ngx_event_posted.c:33
#13 0x0807f35a in ngx_process_events_and_timers (cycle=cycle@entry=0x85b64b8) at src/event/ngx_event.c:259
#14 0x08085976 in ngx_worker_process_cycle (cycle=0x85b64b8, data=0x0) at src/os/unix/ngx_process_cycle.c:753
#15 0x080843a6 in ngx_spawn_process (cycle=cycle@entry=0x85b64b8, proc=0x80858f0 <ngx_worker_process_cycle>, data=0x0, name=0x8171fbb "worker process", respawn=respawn@entry=2)
    at src/os/unix/ngx_process.c:198
#16 0x080869f2 in ngx_reap_children (cycle=0x85b64b8) at src/os/unix/ngx_process_cycle.c:621
#17 ngx_master_process_cycle (cycle=<optimized out>, cycle@entry=0x85b2498) at src/os/unix/ngx_process_cycle.c:174
#18 0x080633de in main (argc=3, argv=0xbf8d5dd4) at src/core/nginx.c:367
@Louvremaster

This comment has been minimized.

Louvremaster commented Mar 31, 2017

I have the same problem with nginx 1.10.3 + naxsi 0.55.3 when http2 is enabled.
Will this be fixed?

@danlsgiga

This comment has been minimized.

danlsgiga commented Mar 31, 2017

I actually had to remove all the $HEADERS_VAR:Cookie from all MainRules in the core.rules file. For some reason, having it with HTTP2 causes the segfaults. Removed them and no issues for 2 days.

I know that removing it is not ideal but having HttpOnly; Secure in the headers kinda overcome most of the cookies attacks, so I'll leave it out for now.

Does anyone have more insights on the risks and implications of leaving that out from the core rules?

@buixor

This comment has been minimized.

Member

buixor commented Apr 6, 2017

Hello,

@danlsgiga / @Louvremaster : did this happen with the http2 branch ?

Thanks !

ps: sorry for delay, I was kinda busy, http2 branch is going for merge in next major

@danlsgiga

This comment has been minimized.

danlsgiga commented Apr 6, 2017

Hey @buixor, yeah, http2 branch and the latest nginx release.

@buixor

This comment has been minimized.

Member

buixor commented Apr 7, 2017

That is bad news :( I'm going to look at it when I come back from hollidays, end of April

@ManuelRighi

This comment has been minimized.

ManuelRighi commented Apr 7, 2017

Hello,
I have same problem, nginx 1.10.3, naxsi 0.55.2 and http2 enabled :|

@marcelomd

This comment has been minimized.

Contributor

marcelomd commented Apr 12, 2017

@danlsgiga @ManuelRighi Did you guys try PR #309 on top of http2 branch?

We've been running HTTP/2 + Naxsi for months with no errors.

The segfaults were caused by Naxsi trying to modify the header part of the request, which lives in read-only memory. See previous posts on this thread.

Unless these are new segfaults, #309 should solve them =)

@ManuelRighi

This comment has been minimized.

ManuelRighi commented Apr 12, 2017

Thanks @marcelomd ... I try http2 branch ;)

@buixor

This comment has been minimized.

Member

buixor commented Apr 12, 2017

Eh I guess I should really merge this one into http2 branch, @marcelomd : no update to do on this one ?

@marcelomd

This comment has been minimized.

Contributor

marcelomd commented Apr 12, 2017

@buixor Nothing new. Works beautifully for us as it is. Go ahead =)

@gutweiler

This comment has been minimized.

gutweiler commented Apr 12, 2017

@buixor No problem at all with http2 branch plus MR #309, would be nice to have it in master. 👍

@danlsgiga

This comment has been minimized.

danlsgiga commented Apr 12, 2017

Hi @marcelomd, yeah, I was just using the http2 branch without the suggested PR #309. @buixor It would be nice to have that merged in http2 since I have a job that builds nginx automatically from that branch.
Thanks for the patches!!

buixor added a commit that referenced this issue Apr 12, 2017

@buixor

This comment has been minimized.

Member

buixor commented Apr 12, 2017

merged into http2, which itself should make its way to master for 0.56 :)

@danlsgiga

This comment has been minimized.

danlsgiga commented Apr 12, 2017

Beautiful! Thanks @buixor

@danlsgiga

This comment has been minimized.

danlsgiga commented Apr 12, 2017

After recompiling nginx with the http2 branch with the merge of PR #309 I confirm that all issues and segfaults are gone. I've restored the original core.rules with all the $HEADERS_VAR:Cookie and everything is smooth again.
Thanks everyone!!!!

@ManuelRighi

This comment has been minimized.

ManuelRighi commented Apr 13, 2017

Hello, What exactly is the commands for download branch HTTP2 and merger of PR # 309 ?
Thanks

@gutweiler

This comment has been minimized.

gutweiler commented Apr 13, 2017

@ManuelRighi

It would be:
git clone https://github.com/nbs-system/naxsi.git && \ cd naxsi && \ git checkout http2 && \ git fetch origin pull/309/head:http2-309 && \ git checkout http2-309 && \

But meanwhile #309 is merged into http2 branch. -> you can just checkout the http2 branch.

@ManuelRighi

This comment has been minimized.

ManuelRighi commented Apr 13, 2017

@gutweiler thanks ;)

@ManuelRighi

This comment has been minimized.

ManuelRighi commented Aug 11, 2017

Hello,
with nginx 1.12.1, pagespeed 1.12.34.2-stable and naxsi with branch http2, I receive this error during compile:

objs/addon/src/ngx_url_async_fetcher.o:/opt/ngx_pagespeed-1.12.34.2-stable/src/ngx_url_async_fetcher.cc:89: first defined here
collect2: error: ld returned 1 exit status
objs/Makefile:382: recipe for target 'objs/nginx' failed
make[2]: *** [objs/nginx] Error 1
make[2]: Leaving directory '/opt/nginx_source/nginx-1.12.1/debian/build-nginx'
Makefile:8: recipe for target 'build' failed
make[1]: *** [build] Error 2
make[1]: Leaving directory '/opt/nginx_source/nginx-1.12.1/debian/build-nginx'
debian/rules:50: recipe for target 'build-arch.nginx' failed
make: *** [build-arch.nginx] Error 2
dpkg-buildpackage: error: debian/rules build gave error exit status 2

If I use branch master, no problem, but I have issue with http2 ..... I need to merge http2 on master ?

Thanks

@combro2k

This comment has been minimized.

combro2k commented Aug 11, 2017

@ManuelRighi
Looks more like an issue of pagespeed... did you try to compile without naxsi?
If the problem presists, you need to contact pagespeed's developers.
if the problem is gone, compile naxsi without pagespeed and see if that helps.

@ManuelRighi

This comment has been minimized.

ManuelRighi commented Aug 11, 2017

@combro2k nginx + pagespeed without naxsi compile is ok

@combro2k

This comment has been minimized.

combro2k commented Aug 11, 2017

@ManuelRighi Maybe this is related? apache/incubator-pagespeed-ngx#1194

Did you try to compile it against nginx mainline version?

@ManuelRighi

This comment has been minimized.

ManuelRighi commented Aug 11, 2017

Hello,
If I reverse the module order, compile works :)

@buixor

This comment has been minimized.

Member

buixor commented Aug 11, 2017

@ManuelRighi : can you provide extra info ? :D

@ManuelRighi

This comment has been minimized.

ManuelRighi commented Aug 11, 2017

If I compile with this ./configure order

--add-module=/opt/ngx_pagespeed-1.12.34.2-stable --add-module=/opt/naxsi-0.55.3/naxsi_src

compile fails. If I reverse order,

--add-module=/opt/naxsi-0.55.3/naxsi_src --add-module=/opt/ngx_pagespeed-1.12.34.2-stable

compile works

@ManuelRighi

This comment has been minimized.

ManuelRighi commented Sep 7, 2017

Hello,
nginx 1.10.3 with naxsi 0.55.3 is affected by this bug ?
If yes, If I checkout http2 branch for naxsi, the naxsi version is still the 0.55.3 ?

Thanks

@msrv

This comment has been minimized.

msrv commented Oct 11, 2017

EDIT: Solved it myself, error was caused by compiling as dynamic module. When using naxsi as static module, the http2 version works with the latest nginx version again.

The issue was fixed before by compiling nginx from the http2 branch. But with the nginx release
1.13.6 nginx will not start even if using the http2 branch of nginx.

Any ideas?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment