Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Already on GitHub? Sign in to your account

Conflict with other modules using libapr #202

Closed
jeffkaufman opened this Issue Apr 1, 2013 · 12 comments

Comments

Projects
None yet
3 participants
Contributor

jeffkaufman commented Apr 1, 2013

@chaizhenhua wrote:

hi, all, if other nginx module use libapr and call apr_terminate() in exit process before
pagespeed, nginx will crash. now i have no idea about how to fix it.

Owner

oschaaf commented Apr 2, 2013

@chaizhenhua
re: "if other nginx module use libapr and call apr_terminate() in exit process before pagespeed, nginx will crash"

According to the docs, apr_terminate() should be called as many times as apr_initialize() has been called.
Is there an easy way to trigger this crash? Can I reproduce this with another nginx module that uses libapr you know of?

Member

chaizhenhua commented Apr 3, 2013

Modsecurity(https://github.com/SpiderLabs/ModSecurity) nginx also use libapr. The crash can be reproduced in this way:
1.compile mod security and page speed together with nginx. Modsecurity should be configured before page speed
2.run some page speed tests so that driver factory creates apr_pool
3.nginx -s quit. Then nginx crash in driver factory dtor.

Can you reproduce it?

ÔÚ 2013-4-3£¬5:09£¬Otto van der Schaaf notifications@github.com дµÀ£º

@chaizhenhua
re: "if other nginx module use libapr and call apr_terminate() in exit process before pagespeed, nginx will crash"

According to the docs, apr_terminate() should be called as many times as apr_initialize() has been called.
Is there an easy way to trigger this crash? Can I reproduce this with another nginx module that uses libapr you know of?

¡ª
Reply to this email directly or view it on GitHub.

Owner

oschaaf commented Apr 3, 2013

@chaizhenhua
I can reproduce a crash using the current master of ModSecurity.

If I configure ngx_pagespeed like so:

./configure --with-debug --add-module=../ModSecurity/nginx/modsecurity/ --add-module=../ngx_pagespeed 

Run a request on it, and then signal it to exit, I get a SIGSEV, with this backtrace:

#0  0x00007ffff7769ddc in ?? () from /usr/lib/libapr-1.so.0
#1  0x00007ffff7768be2 in apr_pool_destroy () from /usr/lib/libapr-1.so.0
#2  0x000000000048bba0 in modsecTerminate () at api.c:264
#3  0x0000000000478934 in ngx_http_modsecurity_exit_process (cycle=<optimized out>) at ../ModSecurity/nginx/modsecurity//ngx_http_modsecurity.c:932
#4  0x000000000042d483 in ngx_master_process_exit (cycle=0xfabdc0) at src/os/unix/ngx_process_cycle.c:697
#5  0x000000000042f4ac in ngx_single_process_cycle (cycle=0xfabdc0) at src/os/unix/ngx_process_cycle.c:326
#6  0x0000000000410a89 in main (argc=<optimized out>, argv=<optimized out>) at src/core/nginx.c:407

So, here the backtrace does not point to RewriteDriverFactory.
As far as I can tell, psol and ngx_pagespeed are not calling apr_initialize / apr_terminate explicitly at all at this moment. But we do use libserf, which possibly does that?

Owner

oschaaf commented Apr 3, 2013

@chaizhenhua

I configured nginx with ModSecurity only, and that seems to give me the same backtrace.

Member

chaizhenhua commented Apr 3, 2013

@oschaaf i cant reproduce can you give me you nginx.conf file?
this is my backtrace after run some ps tests with ModSeurity configured:

#0  0x00007fd162dcd7fb in apr_pool_destroy () from /lib64/libapr-1.so.0
#1  0x0000000000486200 in net_instaweb::SerfUrlAsyncFetcher::~SerfUrlAsyncFetcher (this=0x2a55a60)
    at ../ngx_pagespeed/psol/include/net/instaweb/apache/serf_url_async_fetcher.cc:1075
#2  0x0000000000488d5a in net_instaweb::SerfThreadedFetcher::~SerfThreadedFetcher (this=0x2a55a60)
    at ../ngx_pagespeed/psol/include/net/instaweb/apache/serf_url_async_fetcher.cc:735
#3  0x00000000004889fc in net_instaweb::SerfThreadedFetcher::~SerfThreadedFetcher (this=0x2a55a60)
#4  0x00000000004861e5 in net_instaweb::SerfUrlAsyncFetcher::~SerfUrlAsyncFetcher (this=0x2a53890)
    at ../ngx_pagespeed/psol/include/net/instaweb/apache/serf_url_async_fetcher.cc:1072
#5  0x00000000004860ec in net_instaweb::SerfUrlAsyncFetcher::~SerfUrlAsyncFetcher (this=0x2a53890)
    at ../ngx_pagespeed/psol/include/net/instaweb/apache/serf_url_async_fetcher.cc:1054
#6  0x000000000052847b in net_instaweb::RewriteDriverFactory::~RewriteDriverFactory() ()
#7  0x000000000047e6d5 in net_instaweb::NgxRewriteDriverFactory::~NgxRewriteDriverFactory (this=0x2a5d8e0)
    at ../ngx_pagespeed/src/ngx_rewrite_driver_factory.cc:107
#8  0x000000000047e51c in net_instaweb::NgxRewriteDriverFactory::~NgxRewriteDriverFactory (this=0x2a5d8e0)
    at ../ngx_pagespeed/src/ngx_rewrite_driver_factory.cc:98
#9  0x000000000047944b in ngx_psol::(anonymous namespace)::ps_cleanup_srv_conf (data=0x2a7b970) at ../ngx_pagespeed/src/ngx_pagespeed.cc:526
#10 0x0000000000411659 in ngx_destroy_pool (pool=0x2a4de60) at src/core/ngx_palloc.c:55
#11 0x000000000042e73e in ngx_worker_process_exit (cycle=0x2a4deb0) at src/os/unix/ngx_process_cycle.c:1067
#12 0x000000000042e58a in ngx_worker_process_cycle (cycle=0x2a4deb0, data=<optimized out>) at src/os/unix/ngx_process_cycle.c:812
#13 0x000000000042be21 in ngx_spawn_process (cycle=0x2a4deb0, proc=0x42e3c0 <ngx_worker_process_cycle>, data=0x0, name=0xab1ad3 "worker process", 
    respawn=-3) at src/os/unix/ngx_process.c:198
#14 0x000000000042d378 in ngx_start_worker_processes (cycle=0x2a4deb0, n=1, type=-3) at src/os/unix/ngx_process_cycle.c:362
#15 0x000000000042caba in ngx_master_process_cycle (cycle=0x2a4deb0) at src/os/unix/ngx_process_cycle.c:136
#16 0x000000000040ffa3 in main (argc=<optimized out>, argv=<optimized out>) at src/core/nginx.c:412
Owner

oschaaf commented Apr 3, 2013

@chaizhenhua

I think get one part of this, the backtrace I posted is from a test that probably did not fire up the serf fetcher, and just rewrote html. So we seem to have two problems at hand.
I will send you my configuration later on, as I don't have it available right now.

Member

chaizhenhua commented Apr 4, 2013

@oschaaf i write a tiny module(https://github.com/chaizhenhua/ngx_apr.git) for libapr testing, compile with it modpagespeed also crash with same backtrace.

Owner

oschaaf commented Apr 4, 2013

@chaizhenhua thanks! I will look into it.

I also have a reproduction case for you for the other backtrace.
Configuring nginx:

./configure --with-debug  --add-module=../ModSecurity/nginx/modsecurity/ 

Test nginx configuration

worker_processes 1;
daemon off;
master_process off;
events {
    worker_connections  1024;
}
http {
    include       mime.types;
    default_type  application/octet-stream;
    sendfile        on;
    keepalive_timeout  65;
    server {
        listen       80;
        server_name  localhost;
        location / {
            root   html;
            index  index.html index.htm;
        }
        error_page   500 502 503 504  /50x.html;
        location = /50x.html {
            root   html;
        }
    }
}

I get the backtrace I mentioned using nginx 1.3.7, here is the valgrind trace as well:

==2779== Invalid read of size 8
==2779==    at 0x52A6DDC: ??? (in /usr/lib/libapr-1.so.0.4.6)
==2779==    by 0x52A5BE1: apr_pool_destroy (in /usr/lib/libapr-1.so.0.4.6)    
==2779==    by 0x473DDF: modsecTerminate (api.c:264)
==2779==    by 0x471073: ngx_http_modsecurity_exit_process (ngx_http_modsecurity.c:932)
==2779==    by 0x425BC2: ngx_master_process_exit (ngx_process_cycle.c:697)
==2779==    by 0x427BEB: ngx_single_process_cycle (ngx_process_cycle.c:326)
==2779==    by 0x4091C8: main (nginx.c:407)
==2779==  Address 0x70 is not stack'd, malloc'd or (recently) free'd

I'm curious, are you able to reproduce this one as well?

Owner

oschaaf commented Apr 4, 2013

@chaizhenhua
FYI, I succeeded in reproducing your backtrace using your reproduction module.

Owner

oschaaf commented Apr 4, 2013

@chaizhenhua

If I hook apr_terminate to atexit in your tiny module, instead of running it from nginx's exit_process/exit_master hooks, the crash is gone. Would that be a proper fix here?

chaizhenhua added a commit to chaizhenhua/ngx_pagespeed that referenced this issue Apr 4, 2013

Member

chaizhenhua commented Apr 4, 2013

@oschaaf i have reproduce the modsecurity crash and added patch to modsecurity.
I added explicit call to apr_initialize()/apr_terminate() in NgxRewriteDriverFactory::Initialize()/Terminate(), seems fixed without modify other module.

Contributor

jeffkaufman commented Jun 20, 2014

This was fixed with SystemRewriteDriverFactory::InitApr a while ago.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment