luajit segmentation fault #2

icodesource · 2010-08-24T18:48:44Z

I am using the lua nginx module successfully with the standard lua interpreter. I tried it with luajit version 2.0 beta 5, but nginx crashes with a segmentation fault. What information can I provide to help resolve this issue? I'm running it on Snow Leopard Server. I'll try it with other versions of luajit to see if I can narrow down the problem. What versions of luajit have you tested with? Which platforms?

chaoslawful · 2010-08-25T01:18:54Z

Can you provide the version of nginx, and the stack backtrace when nginx crashed?
Does nginx crash with simplest lua code (i.e. ngx.echo(1))?

chaoslawful · 2010-08-25T01:52:38Z

And did all lua-nginx-module tests passed on your build?

icodesource · 2010-08-25T02:50:09Z

I tried it with 0.8.49 and 0.7.67. The test code is from you example:

    location /inline_concat {

        default_type 'text/plain';

        set $a "hello";
        set $b "world";

        set_by_lua $res "return ngx.arg[1]..ngx.arg[2]" $a $b;
        echo $res;
    }

The stack trace only had two entries:

#0 0x0000000100085e48 in lj_alloc_malloc ()
#1 0x00000001000660c7 in lj_state_newstate ()

I have done some fairly complicated with the non-jit version, including loading modules, so the non-jit version seems fine.

When I start nginx, it fails immediately with the segment fault, before I issue any requests are handled.

agentzh · 2010-08-25T04:23:46Z

I've tried nginx 0.7.67 and 0.8.49 with luajit 2.0 git HEAD (almost identical to the beta5 release kicked out these days), the whole tests suite is passing successfully on my x86_64 slackware linux :)

It may be an issue for LuaJIT 2.0's immature x86_64 support on Mac? I'll try to find a macbook to verify this. Thanks for reporting.

Cheers,
-agentzh

icodesource · 2010-08-25T16:52:07Z

Could be a mac thing. My production machines are linux, but I do all my development on the mac. If you need more information, let me know.

The module looks fantastic so far.

chaoslawful · 2010-08-26T01:34:40Z

I have someone tried nginx-0.8.49 and LuaJIT-2.0beta5 on a 32-bit Mac, and all tests passed. Could you run some Lua codes with your LuaJIT build? Does it work well when running as a standalone interpreter?

icodesource · 2010-08-26T02:54:33Z

Yes, I can run code on the standalone interpreter:

/usr/local/bin/luajit-2.0.0-beta5

chaoslawful · 2010-08-26T16:40:00Z

Could you download this example code and try it out on your Mac?
http://gist.github.com/551728

Download both con_run1.cc and con_run1.lua.

Compile and link con_run1.cc by:
$ g++ -I/usr/local/include/luajit-2.0 con_run1.cc -o con_run1 -L/usr/local/lib -lluajit-5.1

And run:
$ ./con_run1 con_run1.lua

Could you tell me if it crashes?

icodesource · 2010-08-26T17:54:30Z

Yes, it crashes. Stack trace is:

#0 0x000000010002b048 in lj_alloc_malloc ()
#1 0x000000010000b2c7 in lj_state_newstate ()

chaoslawful · 2010-08-27T09:49:00Z

OK...I've created a more minified example. Would you please try this again?
http://gist.github.com/553111

Compile and link min_con.c by:
$ gcc -I/usr/local/include/luajit-2.0 min_con.c -o min_con -L/usr/local/lib -lluajit-5.1

Run:
$ ./min_con

If this also crashes, I will submit a bug to LuaJIT.

icodesource · 2010-08-29T07:26:49Z

Yes, it crashed also. Backtrace is:

#0 0x000000010002b048 in lj_alloc_malloc ()
#1 0x0070006f0043003e in ?? ()

chaoslawful · 2010-08-30T01:48:39Z

It seems that LuaJIT requires some special linking options on 64-bit MacOS...Please try this cmdline and try min_con.c example again:

$ gcc -I/usr/local/include/luajit-2.0 min_con.c -o min_con -Wl,-pagezero_size,10000 -Wl,-image_base,100000000 -L/usr/local/lib -lluajit-5.1

icodesource · 2010-08-30T02:42:37Z

This works. I'll add these link options and try on nginx.

chaoslawful · 2010-08-31T01:23:52Z

Is lua-nginx-module working on your server after added these linking options?

chaoslawful · 2010-09-06T13:40:01Z

OK, consider it solved.

nightsailer · 2010-12-12T15:44:18Z

I got same problem on my SnowLeopard. I think you should add this note to README. Thanks.

chaoslawful · 2010-12-13T02:53:37Z

Thanks for reminding me. The notes have been added to README, and I'll see if it can be auto done in config file later.

This issue appeared in our EC2 test cluster, which compiles Nginx with `-DNGX_LUA_USE_ASSERT` and `-DNGX_LUA_ABORT_AT_PANIC`. The lua-resty-redis test: === TEST 1: github issue openresty#108: ngx.locaiton.capture + redis.set_keepalive in t/bugs.t [1] would produce core dumps in the check leak testing mode. The backtrace for these core dumps was: #0 0x00007fd417bee277 in raise () from /lib64/libc.so.6 openresty#1 0x00007fd417bef968 in abort () from /lib64/libc.so.6 openresty#2 0x00007fd417be7096 in __assert_fail_base () from /lib64/libc.so.6 openresty#3 0x00007fd417be7142 in __assert_fail () from /lib64/libc.so.6 openresty#4 0x000000000050d227 in ngx_http_lua_socket_tcp_resume_conn_op (spool=c/ngx_http_lua_socket_tcp.c:3963 openresty#5 0x000000000050e51a in ngx_http_lua_socket_tcp_finalize (r=r@entry=0x5628) at ../../src/ngx_http_lua_socket_tcp.c:4195 openresty#6 0x000000000050e570 in ngx_http_lua_socket_tcp_cleanup (data=0x7fd419p_lua_socket_tcp.c:3755 openresty#7 0x0000000000463aa5 in ngx_http_free_request (r=r@entry=0xbfaec0, rc=http_request.c:3508 ... Which was caused by the following assertion in ngx_http_lua_socket_tcp.c with `NGX_DEBUG`: #if (NGX_DEBUG) ngx_http_lua_assert(spool->connections >= 0); #else Thanks to Mozilla's rr, a recorded session showed that `spool->connections` was `-1`. Unfortunately, reproducing this case does not seem possible, since the failure is due to the request cleanup (`ngx_http_free_request`). Here is an explanation: -- thread 1 local sock = ngx.socket.tcp() sock:connect() sock:setkeepalive() -- pool created, connections: 1 -- thread 2 local sock = ngx.socket.tcp() sock:connect() -- from pool, connections: 1 -- thread 1 -- sock from thread 1 idle timeout, closes, and calls -- ngx_http_lua_socket_tcp_finalize, connections: 0 -- thread 2 sock:setkeepalive() -- connections: -1 -- ngx_http_lua_socket_tcp_resume_conn_op gets called, assertion fails In order to avoid this race condition, we must determine whether the socket pool exists or not, not from the `ngx_http_lua_socket_tcp_upstream` struct, but from the Lua Registry. This way, when thread 2's socket enters the keepalive state, it will respect the previous call to `ngx_http_lua_socket_free_pool` (which unset the pool from the registry). [1]: https://github.com/openresty/lua-resty-redis/blob/master/t/bugs.t

This issue appeared in our EC2 test cluster, which compiles Nginx with `-DNGX_LUA_USE_ASSERT` and `-DNGX_LUA_ABORT_AT_PANIC`. The lua-resty-redis test: === TEST 1: github issue openresty#108: ngx.locaiton.capture + redis.set_keepalive in t/bugs.t [1] would produce core dumps in the check leak testing mode. The backtrace for these core dumps was: #0 0x00007fd417bee277 in raise () from /lib64/libc.so.6 openresty#1 0x00007fd417bef968 in abort () from /lib64/libc.so.6 openresty#2 0x00007fd417be7096 in __assert_fail_base () from /lib64/libc.so.6 openresty#3 0x00007fd417be7142 in __assert_fail () from /lib64/libc.so.6 openresty#4 0x000000000050d227 in ngx_http_lua_socket_tcp_resume_conn_op (spool=c/ngx_http_lua_socket_tcp.c:3963 openresty#5 0x000000000050e51a in ngx_http_lua_socket_tcp_finalize (r=r@entry=0x5628) at ../../src/ngx_http_lua_socket_tcp.c:4195 openresty#6 0x000000000050e570 in ngx_http_lua_socket_tcp_cleanup (data=0x7fd419p_lua_socket_tcp.c:3755 openresty#7 0x0000000000463aa5 in ngx_http_free_request (r=r@entry=0xbfaec0, rc=http_request.c:3508 ... Which was caused by the following assertion in ngx_http_lua_socket_tcp.c with `NGX_DEBUG`: #if (NGX_DEBUG) ngx_http_lua_assert(spool->connections >= 0); #else Thanks to Mozilla's rr, a recorded session showed that `spool->connections` was `-1`. Here is a reproducible test case: local sock1 = ngx.socket.tcp() local sock2 = ngx.socket.tcp() sock1:connect() sock2:connect() sock1:setkeepalive() -- pool created, connections: 1 sock2:setkeepalive() -- connections: 1 sock1:connect() -- connections: 1 sock2:connect() -- connections: 1 sock1:close() -- connections: 0 sock2:close() -- connections: -1 -- ngx_http_lua_socket_tcp_resume_conn_op gets called, assertion fails In order to avoid this race condition, we must determine whether the socket pool exists or not, not from the `ngx_http_lua_socket_tcp_upstream` struct, but from the Lua Registry. This way, when thread 2's socket enters the keepalive state, it will respect the previous call to `ngx_http_lua_socket_free_pool` (which unset the pool from the registry). [1]: https://github.com/openresty/lua-resty-redis/blob/master/t/bugs.t

==openresty==70603==ERROR: AddressSanitizer: memcpy-param-overlap: memory ranges [0x621000001500,0x621000002181) and [0x62100000187f, 0x621000002500) overlap #0 0x7f3db1899ffe (/lib64/libasan.so.5+0x99ffe) #1 0x9da926 (/usr/local/openresty-debug/nginx/sbin/nginx+0x9da926) #2 0x9dd1a1 (/usr/local/openresty-debug/nginx/sbin/nginx+0x9dd1a1) #3 0x4c89c6 (/usr/local/openresty-debug/nginx/sbin/nginx+0x4c89c6) #4 0x5d1e4e (/usr/local/openresty-debug/nginx/sbin/nginx+0x5d1e4e) #5 0x4c89c6 (/usr/local/openresty-debug/nginx/sbin/nginx+0x4c89c6) #6 0x5b8583 (/usr/local/openresty-debug/nginx/sbin/nginx+0x5b8583) #7 0x4c89c6 (/usr/local/openresty-debug/nginx/sbin/nginx+0x4c89c6) #8 0x4b4419 (/usr/local/openresty-debug/nginx/sbin/nginx+0x4b4419) #9 0x427f16 (/usr/local/openresty-debug/nginx/sbin/nginx+0x427f16) #10 0x7f3daff27554 (/lib64/libc.so.6+0x22554) #11 0x42d537 (/usr/local/openresty-debug/nginx/sbin/nginx+0x42d537)

…t_peer at function end. Avoid inserting new parameters in the middle of the function to prevent core dumps when using old lua-resty-core with new lua-nginx-module. Example stack trace: ``` Message: Process 1414245 (nginx) of user 1000 dumped core. Stack trace of thread 1414245: #0 0x00007ff596938285 __strlen_avx2 (libc.so.6 + 0x162285) openresty#1 0x00007ff596f623d2 lj_cf_ffi_string (libluajit-5.1.so.2 + 0x523d2) openresty#2 0x00007ff596f1bb4b lj_BC_FUNCC (libluajit-5.1.so.2 + 0xbb4b) openresty#3 0x00007ff596f74223 lua_pcall (libluajit-5.1.so.2 + 0x64223) openresty#4 0x00000000005044b7 n/a (/home/jiahao/work/org/lua-resty-core/work/nginx/sbin/nginx + 0x1044b7) ```

…t function end. Avoid inserting new parameters in the middle of the function to prevent core dumps when using old lua-resty-core with new lua-nginx-module. Example stack trace: ``` Message: Process 2199905 (nginx) of user 1000 dumped core. Stack trace of thread 2199905: #0 0x00007ffaf1e4b385 in OPENSSL_sk_num (st=st@entry=0xffffffff) at crypto/stack/stack.c:382 openresty#1 0x0000000000510aba in sk_X509_num (sk=0xffffffff) at /opt/ssl/include/openssl/x509.h:99 openresty#2 ngx_http_lua_ffi_ssl_verify_client (r=<optimized out>, client_certs=<optimized out>, trusted_certs=0xffffffff, depth=<optimized out>, err=0x0) at /home/jiahao/work/org/lua-nginx-module/src/ngx_http_lua_ssl_certby.c:1588 ```

…t_peer at function end. Avoid inserting new parameters in the middle of the function to prevent core dumps when using old lua-resty-core with new lua-nginx-module. Example stack trace: ``` Message: Process 1414245 (nginx) of user 1000 dumped core. Stack trace of thread 1414245: #0 0x00007ff596938285 __strlen_avx2 (libc.so.6 + 0x162285) openresty#1 0x00007ff596f623d2 lj_cf_ffi_string (libluajit-5.1.so.2 + 0x523d2) openresty#2 0x00007ff596f1bb4b lj_BC_FUNCC (libluajit-5.1.so.2 + 0xbb4b) openresty#3 0x00007ff596f74223 lua_pcall (libluajit-5.1.so.2 + 0x64223) openresty#4 0x00000000005044b7 n/a (/home/jiahao/work/org/lua-resty-core/work/nginx/sbin/nginx + 0x1044b7) ```

…t function end. Avoid inserting new parameters in the middle of the function to prevent core dumps when using old lua-resty-core with new lua-nginx-module. Example stack trace: ``` Message: Process 2199905 (nginx) of user 1000 dumped core. Stack trace of thread 2199905: #0 0x00007ffaf1e4b385 in OPENSSL_sk_num (st=st@entry=0xffffffff) at crypto/stack/stack.c:382 openresty#1 0x0000000000510aba in sk_X509_num (sk=0xffffffff) at /opt/ssl/include/openssl/x509.h:99 openresty#2 ngx_http_lua_ffi_ssl_verify_client (r=<optimized out>, client_certs=<optimized out>, trusted_certs=0xffffffff, depth=<optimized out>, err=0x0) at /home/jiahao/work/org/lua-nginx-module/src/ngx_http_lua_ssl_certby.c:1588 ```

…t_peer at function end. Avoid inserting new parameters in the middle of the function to prevent core dumps when using old lua-resty-core with new lua-nginx-module. Example stack trace: ``` Message: Process 1414245 (nginx) of user 1000 dumped core. Stack trace of thread 1414245: #0 0x00007ff596938285 __strlen_avx2 (libc.so.6 + 0x162285) openresty#1 0x00007ff596f623d2 lj_cf_ffi_string (libluajit-5.1.so.2 + 0x523d2) openresty#2 0x00007ff596f1bb4b lj_BC_FUNCC (libluajit-5.1.so.2 + 0xbb4b) openresty#3 0x00007ff596f74223 lua_pcall (libluajit-5.1.so.2 + 0x64223) openresty#4 0x00000000005044b7 n/a (/home/jiahao/work/org/lua-resty-core/work/nginx/sbin/nginx + 0x1044b7) ```

…t function end. Avoid inserting new parameters in the middle of the function to prevent core dumps when using old lua-resty-core with new lua-nginx-module. Example stack trace: ``` Message: Process 2199905 (nginx) of user 1000 dumped core. Stack trace of thread 2199905: #0 0x00007ffaf1e4b385 in OPENSSL_sk_num (st=st@entry=0xffffffff) at crypto/stack/stack.c:382 openresty#1 0x0000000000510aba in sk_X509_num (sk=0xffffffff) at /opt/ssl/include/openssl/x509.h:99 openresty#2 ngx_http_lua_ffi_ssl_verify_client (r=<optimized out>, client_certs=<optimized out>, trusted_certs=0xffffffff, depth=<optimized out>, err=0x0) at /home/jiahao/work/org/lua-nginx-module/src/ngx_http_lua_ssl_certby.c:1588 ```

dakanji mentioned this issue Jan 4, 2012

Argument Delimiter #77

Closed

0x5446 mentioned this issue Dec 17, 2012

Nginx worker process constantly crash #192

Closed

rfwatson mentioned this issue May 8, 2013

Segfault in ngx_http_set_cookie_header #237

Closed

cs0604 mentioned this issue Sep 6, 2013

100% cpu #278

Closed

ghost mentioned this issue Sep 14, 2011

Segfault caused by passing certain characters to ngx.re.match #59

Closed

DorianGray mentioned this issue Feb 1, 2014

Major cosockets patch #290

Closed

doujiang24 mentioned this issue Nov 17, 2015

ngx.location.capture and sendfile #603

Closed

tjliupeng mentioned this issue Aug 23, 2016

can't print the response header Set-Cookie in log #845

Closed

ghost mentioned this issue May 2, 2018

Random segfaults with req_raw_header #1313

Closed

sdcuike mentioned this issue Jun 8, 2018

nginx[27376]: segfault at 7f12e2a71ff3 ip 0000000000519331 sp 00007fffaf4b0070 error 7 in nginx[400000+396000] #1337

Closed

Wladimir-N mentioned this issue Jan 21, 2019

Заголовки не преобразуются в push #1458

Closed

fancy-rabbit mentioned this issue Jan 31, 2019

nginx -t segfaults after upgrading lua-nginx-module to v0.10.14rc4 #1462

Closed

ylfforme mentioned this issue Feb 1, 2019

not support Nginx 1.15.6? #1464

Closed

suyogdilipkale mentioned this issue Apr 25, 2020

srcache_store response hangs with request body when used with ngx.req.get_body_data() #1699

Open

holycd mentioned this issue Feb 24, 2023

Nginx - Worker Process Crash (Lua Parser) #2158

Open

This issue was closed.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

luajit segmentation fault #2

luajit segmentation fault #2

icodesource commented Aug 24, 2010

chaoslawful commented Aug 25, 2010

chaoslawful commented Aug 25, 2010

icodesource commented Aug 25, 2010

agentzh commented Aug 25, 2010

icodesource commented Aug 25, 2010

chaoslawful commented Aug 26, 2010

icodesource commented Aug 26, 2010

chaoslawful commented Aug 26, 2010

icodesource commented Aug 26, 2010

chaoslawful commented Aug 27, 2010

icodesource commented Aug 29, 2010

chaoslawful commented Aug 30, 2010

icodesource commented Aug 30, 2010

chaoslawful commented Aug 31, 2010

chaoslawful commented Sep 6, 2010

nightsailer commented Dec 12, 2010

chaoslawful commented Dec 13, 2010

luajit segmentation fault #2

luajit segmentation fault #2

Comments

icodesource commented Aug 24, 2010

chaoslawful commented Aug 25, 2010

chaoslawful commented Aug 25, 2010

icodesource commented Aug 25, 2010

agentzh commented Aug 25, 2010

icodesource commented Aug 25, 2010

chaoslawful commented Aug 26, 2010

icodesource commented Aug 26, 2010

chaoslawful commented Aug 26, 2010

icodesource commented Aug 26, 2010

chaoslawful commented Aug 27, 2010

icodesource commented Aug 29, 2010

chaoslawful commented Aug 30, 2010

icodesource commented Aug 30, 2010

chaoslawful commented Aug 31, 2010

chaoslawful commented Sep 6, 2010

nightsailer commented Dec 12, 2010

chaoslawful commented Dec 13, 2010