Segmentation fault when execution time of LUA script exceeds timeout #437

matej21 · 2014-02-20T23:05:53Z

Hi, I'm getting segmentation fault error when execution time of lua script exceeds timeout defined in Redis::connect(), here is example code:

<?php
$lua = '
for y = 1, 10, 1 do
       for i = 1, 100000, 1 do
               redis.call("sAdd", "foo", i)
       end
       for i = 1, 100000, 1 do
               redis.call("sRem", "foo", i)
       end
end
';
$redis = new Redis();
$redis->connect("localhost", NULL, 1);
$sha = $redis->script("load", $lua);
$redis->evalsha($sha);

and here is gdb output:

(gdb) run redis.php
Starting program: /usr/bin/php5 redis.php
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
[New Thread 0x7fffea0ee700 (LWP 7886)]
[Thread 0x7fffea0ee700 (LWP 7886) exited]

Program received signal SIGSEGV, Segmentation fault.
__memcpy_ssse3_back () at ../sysdeps/x86_64/multiarch/memcpy-ssse3-back.S:2611
2611    ../sysdeps/x86_64/multiarch/memcpy-ssse3-back.S: No such file or directory.
(gdb) bt
#0  __memcpy_ssse3_back () at ../sysdeps/x86_64/multiarch/memcpy-ssse3-back.S:2611
#1  0x000000000067b0c7 in _estrndup ()
#2  0x000000000069e2d4 in _zval_copy_ctor_func ()
#3  0x00007fffea76e4e8 in _zval_copy_ctor (zvalue=0x7ffff7fbb2f8) at /usr/include/php5/Zend/zend_variables.h:45
#4  redis_read_variant_reply (ht=ht@entry=1, return_value=return_value@entry=0x7ffff7fbb2f8, return_value_ptr=return_value_ptr@entry=0x0, 
    this_ptr=this_ptr@entry=0x7ffff7fb9680, return_value_used=return_value_used@entry=0, redis_sock=redis_sock@entry=0x7ffff7fbb568, z_tab=z_tab@entry=0x0)
    at /home/matej21/tmp/phpredis/library.c:1835
#5  0x00007fffea767e2f in zim_Redis_evalsha (ht=1, return_value=0x7ffff7fbb2f8, return_value_ptr=0x0, this_ptr=0x7ffff7fb9680, return_value_used=0)
    at /home/matej21/tmp/phpredis/redis.c:6112
#6  0x0000000000747d21 in ?? ()
#7  0x0000000000701737 in execute ()
#8  0x00000000006a0a9c in zend_execute_scripts ()
#9  0x0000000000640be3 in php_execute_script ()
#10 0x000000000074a6a3 in ?? ()
#11 0x000000000042d080 in ?? ()
#12 0x00007ffff58abea5 in __libc_start_main (main=0x42cbb0, argc=2, ubp_av=0x7fffffffdf98, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, 
    stack_end=0x7fffffffdf88) at libc-start.c:260
#13 0x000000000042d115 in _start ()

The text was updated successfully, but these errors were encountered:

michael-grunder · 2014-02-20T23:26:57Z

Hey,

Interesting. I will attempt to replicate this.

Cheers,
Mike

function if we get a protocol error, rather than just throwing and exception and breaking execution of the loop. Addresses #437

michael-grunder · 2014-02-20T23:50:16Z

Hey,

Please try this hotfix on the branch hotfix/eval_timeout_segfault and see if it works for you. It seems pretty obvious what's happening, and if it works on your end I'll merge it into master later today.

Cheers and thanks for the report!
Mike

matej21 · 2014-02-21T00:54:02Z

Thanks for quick response!

Segmantation fault error has disappeared :)

Now it throws this exception:

PHP Fatal error:  Uncaught exception 'RedisException' with message 'read error on connection' in /tmp/redis.php:16
Stack trace:
#0 /tmp/redis.php(16): Redis->select(0)
#1 {main}
  thrown in /tmp/redis.php on line 16
matej21@matej21-laptop:~/tmp/lua$ php redis.php 
PHP Fatal error:  Uncaught exception 'RedisException' with message 'socket error on read socket' in /tmp/redis.php:18
Stack trace:
#0 /tmp/redis.php(18): Redis->evalsha('09b96952cdf8673...')
#1 {main}

Next exception 'RedisException' with message 'protocol error, got '�' as reply-type byte
' in /tmp/redis.php:18
Stack trace:
#0 /tmp/redis.php(0): Redis->evalsha()
#1 {main}
  thrown in /tmp/redis.php on line 18

Which is quite ok, but more accurate exception would be great (saying that it is timeout related error)

michael-grunder · 2014-02-21T00:55:51Z

Yeah, that's just the error message that means, essentially, a timeout.
Not a bad idea to make a more specific error though.

I'll get this merged into master this evening.

Cheers,
Mike
On Feb 20, 2014 4:54 PM, "matej21" notifications@github.com wrote:

Thanks for quick response!

Segmantation fault error has disappeared :)

Now it throws this exception:

PHP Fatal error: Uncaught exception 'RedisException' with message 'read error on connection' in /tmp/redis.php:16
Stack trace:
#0 /tmp/redis.php(16): Redis->select(0)
#1 {main}
thrown in /tmp/redis.php on line 16
matej21@matej21-laptop:~/tmp/lua$ php redis.php
PHP Fatal error: Uncaught exception 'RedisException' with message 'socket error on read socket' in /tmp/redis.php:18
Stack trace:
#0 /tmp/redis.php(18): Redis->evalsha('09b96952cdf8673...')
#1 {main}

Next exception 'RedisException' with message 'protocol error, got '�' as reply-type byte
' in /tmp/redis.php:18
Stack trace:
#0 /tmp/redis.php(0): Redis->evalsha()
#1 {main}
thrown in /tmp/redis.php on line 18

Which is quite ok, but more accurate exception would be great (saying that
it is timeout related error)

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/437#issuecomment-35688062
.

dwguesswho · 2014-04-22T15:38:03Z

Hi, I am experiencing a similar situation, a LUA script that original gave a segmentation fault, we updated php redis, and now I get the same two exceptions as above, 'Socket error on read' followed by 'protocal error' with a non-standard character.

you said :

It seems pretty obvious what's happening

could you elaborate? My LUA was running fine a month ago, and appears to still run, though I am having an issue with the client returning these two exceptions

thanks

michael-grunder · 2014-04-22T16:23:42Z

Hey,

Sure. I meant that it was obvious what the bug was inside of the phpredis C code. Essentially, I was throwing an exception and not returning from the function, but rather just breaking out of a switch. This meant phpredis tried to access a NULL pointer which is a bad thing. :)

The first error seems good to me (it is in fact timing out), but the second one seems problematic. It shouldn't throw a protocol error exception after that.

Are you using connect or pconnect, and is your lua script by chance inside of a multi block?

Cheers,
Mike

dwguesswho · 2014-04-22T16:29:07Z

Hi, I am using connect, not pconnect.

I am not using a multi block.

My LUA script is very similar to the other person's above. I take the LUA and load it, then call it via the evalsha method.

michael-grunder · 2014-04-22T16:31:40Z

Cool. I will try to replicate the protocol error locally. Hopefully it's easy enough to track down. As a quick workaround you can try bumping your timeout so the script doesn't fail in that way.

dwguesswho · 2014-04-22T16:32:23Z

which timeout? do you mean when I connect? or do you mean the lua-time-limit?

michael-grunder · 2014-04-22T16:34:49Z

I'm speaking of your connect timeout, and also you should set this option in phpredis:

$obj_redis->setOption(Redis::OPT_READ_TIMEOUT, <some value>);

I'll also see what phpredis does if the script simply times out server side, but I believe that should work fine (would probably just return false).

dwguesswho · 2014-04-22T17:00:09Z

I wanted to send this privately, can add on github if you want.

I did that set option, and it appears to now throw 0 exceptions. [my connection time out, in method connect() was set to 0, which I thought meant unlimited ]

Do you think using set option is the Fix, or a temporary solution ?

Again, thanks for your help on this.

From: Michael Grunder [mailto:notifications@github.com]
Sent: Tuesday, April 22, 2014 12:35 PM
To: nicolasff/phpredis
Cc: Donald Welch
Subject: Re: [phpredis] Segmentation fault when execution time of LUA script exceeds timeout (#437)

I'm speaking of your connect timeout, and also you should set this option in phpredis:

$obj_redis->setOption(Redis::OPT_READ_TIMEOUT, );

I'll also see what phpredis does if the script simply times out server side, but I believe that should work fine (would probably just return false).

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/437#issuecomment-41062690.

michael-grunder · 2014-04-22T17:04:58Z

Hey,

I've never got a problem with communicating over GitHub as that makes it
available for other people to search for when they run into similar issues.
Email is good too though. :)

It can be a bit confusing, but phpredis uses two separate timeout values.
One is for connection/writing to the socket, and the other is for reading
from it. This level of granularity is useful because there are certain
commands which can take a bit of time (imagine a very large ZUNIONSTORE
operation), but you want to keep it low for simply connecting to the server.

I need to investigate what is causing the protocol error (as that part
shouldn't happen), but you probably want to continue using the read timeout
setting regardless.

Cheers!
Mike

On Tue, Apr 22, 2014 at 10:00 AM, dwguesswho notifications@github.comwrote:

I wanted to send this privately, can add on github if you want.

I did that set option, and it appears to now throw 0 exceptions. [my
connection time out, in method connect() was set to 0, which I thought
meant unlimited ]

Do you think using set option is the Fix, or a temporary solution ?

Again, thanks for your help on this.

From: Michael Grunder [mailto:notifications@github.com]
Sent: Tuesday, April 22, 2014 12:35 PM
To: nicolasff/phpredis
Cc: Donald Welch
Subject: Re: [phpredis] Segmentation fault when execution time of LUA
script exceeds timeout (#437)

I'm speaking of your connect timeout, and also you should set this option
in phpredis:

$obj_redis->setOption(Redis::OPT_READ_TIMEOUT, );

I'll also see what phpredis does if the script simply times out server
side, but I believe that should work fine (would probably just return
false).

—
Reply to this email directly or view it on GitHub<
https://github.com/nicolasff/phpredis/issues/437#issuecomment-41062690>.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/437#issuecomment-41065699
.

dwguesswho · 2014-04-22T17:47:08Z

I just tested 5x and everything went smoothly. I no longer get either exception.

again, thanks !

michael-grunder · 2014-04-22T19:12:33Z

Nice one. I'll leave this open while I track down the cause of that second exception you're seeing.

michael-grunder added a commit that referenced this issue Feb 20, 2014

We need to return immediately from our variant reply parsing

d2832bf

function if we get a protocol error, rather than just throwing and exception and breaking execution of the loop. Addresses #437

michael-grunder closed this as completed Feb 21, 2014

michael-grunder reopened this Apr 22, 2014

michael-grunder closed this as completed Jun 8, 2016

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Segmentation fault when execution time of LUA script exceeds timeout #437

Segmentation fault when execution time of LUA script exceeds timeout #437

matej21 commented Feb 20, 2014

michael-grunder commented Feb 20, 2014

michael-grunder commented Feb 20, 2014

matej21 commented Feb 21, 2014

michael-grunder commented Feb 21, 2014

dwguesswho commented Apr 22, 2014

michael-grunder commented Apr 22, 2014

dwguesswho commented Apr 22, 2014

michael-grunder commented Apr 22, 2014

dwguesswho commented Apr 22, 2014

michael-grunder commented Apr 22, 2014

dwguesswho commented Apr 22, 2014

michael-grunder commented Apr 22, 2014

dwguesswho commented Apr 22, 2014

michael-grunder commented Apr 22, 2014

Segmentation fault when execution time of LUA script exceeds timeout #437

Segmentation fault when execution time of LUA script exceeds timeout #437

Comments

matej21 commented Feb 20, 2014

michael-grunder commented Feb 20, 2014

michael-grunder commented Feb 20, 2014

matej21 commented Feb 21, 2014

michael-grunder commented Feb 21, 2014

dwguesswho commented Apr 22, 2014

michael-grunder commented Apr 22, 2014

dwguesswho commented Apr 22, 2014

michael-grunder commented Apr 22, 2014

dwguesswho commented Apr 22, 2014

michael-grunder commented Apr 22, 2014

dwguesswho commented Apr 22, 2014

michael-grunder commented Apr 22, 2014

dwguesswho commented Apr 22, 2014

michael-grunder commented Apr 22, 2014