New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

debug read error on connection #70

Closed
sorinv opened this Issue Oct 30, 2011 · 137 comments

Comments

Projects
None yet
@sorinv

sorinv commented Oct 30, 2011

Hi there,

I am getting a lot of "read error on connection" errors. Server is under load, but not a lot. I traced this to always happen here:

  • library.c
  • redis_sock_read
  • php_stream_gets -> this returns NULL, so it triggers the exception.

Any way I can further debug this to figure out why it happens?

Thanks,
Sorin

@nicolasff

This comment has been minimized.

Show comment
Hide comment
@nicolasff

nicolasff Oct 30, 2011

Contributor

Hello,

Could you check your timeout settings, and try changing them to higher value?
Try also looking at the number of clients that Redis allows, you might be hitting a limit.

Nicolas

On 30 oct. 2011, at 19:14, sorinvreply@reply.github.com wrote:

Hi there,

I am getting a lot of "read error on connection" errors. Server is under load, but not a lot. I traced this to always happen here:

  • library.c
  • redis_sock_read
  • php_stream_gets -> this returns NULL, so it triggers the exception.

Any way I can further debug this to figure out why it happens?

Thanks,
Sorin

Reply to this email directly or view it on GitHub:
#70

Contributor

nicolasff commented Oct 30, 2011

Hello,

Could you check your timeout settings, and try changing them to higher value?
Try also looking at the number of clients that Redis allows, you might be hitting a limit.

Nicolas

On 30 oct. 2011, at 19:14, sorinvreply@reply.github.com wrote:

Hi there,

I am getting a lot of "read error on connection" errors. Server is under load, but not a lot. I traced this to always happen here:

  • library.c
  • redis_sock_read
  • php_stream_gets -> this returns NULL, so it triggers the exception.

Any way I can further debug this to figure out why it happens?

Thanks,
Sorin

Reply to this email directly or view it on GitHub:
#70

@sorinv

This comment has been minimized.

Show comment
Hide comment
@sorinv

sorinv Oct 31, 2011

Hey Nicolas,

Thanks for the reply.

Timeout is set to 5 seconds (which is supposed to be like forever for redis). Number of clients is set to unlimited.

The server is very large and there are less than 200 clients connected at the same time. The redis process never maxes out a processor, so I assume it's not worked out too hard to issue timeouts.

I am using PHP-FPM and persistent connects (supposedly it'll save the connect time, so a bit faster). Can this be an issue?

Thanks,
Sorin

sorinv commented Oct 31, 2011

Hey Nicolas,

Thanks for the reply.

Timeout is set to 5 seconds (which is supposed to be like forever for redis). Number of clients is set to unlimited.

The server is very large and there are less than 200 clients connected at the same time. The redis process never maxes out a processor, so I assume it's not worked out too hard to issue timeouts.

I am using PHP-FPM and persistent connects (supposedly it'll save the connect time, so a bit faster). Can this be an issue?

Thanks,
Sorin

@rmarscher

This comment has been minimized.

Show comment
Hide comment
@rmarscher

rmarscher Oct 31, 2011

Related to this... I was confused about the timeout setting in connect/pconnect. I thought it pertained only to obtaining a connection to the server. But it turns out it sets the timeout for all requests. I set it to 2.5 seconds and was wondering why I kept getting the "read error on connection" exception thrown. The server also has it's own timeout which will cause the exception to be thrown when it is reached.

rmarscher commented Oct 31, 2011

Related to this... I was confused about the timeout setting in connect/pconnect. I thought it pertained only to obtaining a connection to the server. But it turns out it sets the timeout for all requests. I set it to 2.5 seconds and was wondering why I kept getting the "read error on connection" exception thrown. The server also has it's own timeout which will cause the exception to be thrown when it is reached.

@nicolasff

This comment has been minimized.

Show comment
Hide comment
@nicolasff

nicolasff Nov 1, 2011

Contributor

@sorinv, I'm not sure how PHP-FPM is working internally. If it doesn't reuse processes, there's no advantage to using pconnect instead of connect. pconnect is useful with Apache in prefork mode, where processes are reused a number of times.

php_stream_gets returning NULL means that we can't read from the server, either because there is a connection issue, a timeout, or that the server closed the connection.

Contributor

nicolasff commented Nov 1, 2011

@sorinv, I'm not sure how PHP-FPM is working internally. If it doesn't reuse processes, there's no advantage to using pconnect instead of connect. pconnect is useful with Apache in prefork mode, where processes are reused a number of times.

php_stream_gets returning NULL means that we can't read from the server, either because there is a connection issue, a timeout, or that the server closed the connection.

@sorinv

This comment has been minimized.

Show comment
Hide comment
@sorinv

sorinv Nov 1, 2011

The PHP-FPM does reuse processes, so pconnect should be a gain (albeit small).

On the php_stream_get returning null, I assume no way to know exactly what the cause is? If the server closed the connection, I would know to look at the server side, for example.

sorinv commented Nov 1, 2011

The PHP-FPM does reuse processes, so pconnect should be a gain (albeit small).

On the php_stream_get returning null, I assume no way to know exactly what the cause is? If the server closed the connection, I would know to look at the server side, for example.

@bobrik

This comment has been minimized.

Show comment
Hide comment
@bobrik

bobrik Nov 1, 2011

@sorinv, what ulimit -n says from redis user?

Btw, providing more meaningful exceptions if a big plus for debugging.

bobrik commented Nov 1, 2011

@sorinv, what ulimit -n says from redis user?

Btw, providing more meaningful exceptions if a big plus for debugging.

@sorinv

This comment has been minimized.

Show comment
Hide comment
@sorinv

sorinv Nov 1, 2011

@bobrik, that command shows 100000, so open files should not be an issue.

sorinv commented Nov 1, 2011

@bobrik, that command shows 100000, so open files should not be an issue.

@bobrik

This comment has been minimized.

Show comment
Hide comment
@bobrik

bobrik Nov 1, 2011

+1 for this issue, we have some errors like this and have no idea what's the reason.

bobrik commented Nov 1, 2011

+1 for this issue, we have some errors like this and have no idea what's the reason.

@michael-grunder

This comment has been minimized.

Show comment
Hide comment
@michael-grunder

michael-grunder Nov 10, 2011

Member

This specific error has plagued us as well, and it is incredibly hard to track down. I decided to hack at it a bit tonight and may have found something.

You can cause this error to happen by issuing a $connection->subscribe() command on a channel where you don't publish anything. After a certain amount of time it'll timeout (it's idle) and you'll get this error. This very simple program will do this for you:

$start = time();

$r = new Redis();
$r->connect('localhost',6379, 0);

try {
    $r->subscribe(array('test-channel'),'SubCallback');
} catch(Exception $ex) {
    $elapsed = time() - $start;

    echo "Exception: " . $ex->getMessage() . "\n";
    echo "Elapsed: " . $elapsed . "\n";
}

function SubCallback($objConnection, $strChannel, $message) {
    echo "Channel: $strChannel, message: $message\n";   
}

When I run this, I get the "read error on connection" message at 60 seconds, every time. My timeout in redis.conf is zero.
If I set the number to something huge (like 300 seconds), then it'll timeout at 300 seconds. Perhaps somehow passing zero is overridden as 60 seconds somewhere?

@nicolasff: I just wanted to mention how awesome phpredis is. Truly fantastic! :)

Member

michael-grunder commented Nov 10, 2011

This specific error has plagued us as well, and it is incredibly hard to track down. I decided to hack at it a bit tonight and may have found something.

You can cause this error to happen by issuing a $connection->subscribe() command on a channel where you don't publish anything. After a certain amount of time it'll timeout (it's idle) and you'll get this error. This very simple program will do this for you:

$start = time();

$r = new Redis();
$r->connect('localhost',6379, 0);

try {
    $r->subscribe(array('test-channel'),'SubCallback');
} catch(Exception $ex) {
    $elapsed = time() - $start;

    echo "Exception: " . $ex->getMessage() . "\n";
    echo "Elapsed: " . $elapsed . "\n";
}

function SubCallback($objConnection, $strChannel, $message) {
    echo "Channel: $strChannel, message: $message\n";   
}

When I run this, I get the "read error on connection" message at 60 seconds, every time. My timeout in redis.conf is zero.
If I set the number to something huge (like 300 seconds), then it'll timeout at 300 seconds. Perhaps somehow passing zero is overridden as 60 seconds somewhere?

@nicolasff: I just wanted to mention how awesome phpredis is. Truly fantastic! :)

@nicolasff

This comment has been minimized.

Show comment
Hide comment
@nicolasff

nicolasff Nov 30, 2011

Contributor

@michael-grunder, could you please try setting default_socket_timeout to 0 in php.ini? phpredis uses the php sockets, which will timeout after a set amount of time.

Contributor

nicolasff commented Nov 30, 2011

@michael-grunder, could you please try setting default_socket_timeout to 0 in php.ini? phpredis uses the php sockets, which will timeout after a set amount of time.

@daviddoran

This comment has been minimized.

Show comment
Hide comment
@daviddoran

daviddoran Dec 14, 2011

For me, setting default_socket_timeout = 0 made the connection time out immediately.

I've been using default_socket_timeout = -1 and that seems to be working as "no timeout".

daviddoran commented Dec 14, 2011

For me, setting default_socket_timeout = 0 made the connection time out immediately.

I've been using default_socket_timeout = -1 and that seems to be working as "no timeout".

@RVN-BR

This comment has been minimized.

Show comment
Hide comment
@RVN-BR

RVN-BR Feb 7, 2012

+1 on this issue...

When using a long timeout, occasionally the redis server is screwed up and the php lags trying to connect (when it should error out and go to the next server).... a short timeout starts showing up read error.... catch 22?

RVN-BR commented Feb 7, 2012

+1 on this issue...

When using a long timeout, occasionally the redis server is screwed up and the php lags trying to connect (when it should error out and go to the next server).... a short timeout starts showing up read error.... catch 22?

@wa0x6e

This comment has been minimized.

Show comment
Hide comment
@wa0x6e

wa0x6e Feb 20, 2012

+1

There's 2 pattern in OS X crash report :

Exception Type: EXC_BAD_ACCESS (SIGSEGV)
Exception Codes: 0x000000000000000d, 0x0000000000000000

VM Regions Near 0:
-->
__TEXT 0000000109a21000-0000000109a5a000 [ 228K] r-x/rwx SM=COW /opt/local/apache2/*

Application Specific Information:
objc[381]: garbage collection is OFF

Thread 0 Crashed:: Dispatch queue: com.apple.main-thread
0 libphp5.so 0x000000010a297df0 _php_stream_set_option + 16
1 libphp5.so 0x000000010a2a32e2 _php_stream_xport_create + 194
2 redis.so 0x000000010b6e9b41 redis_sock_connect + 321
3 redis.so 0x000000010b6e99c2 redis_check_eof + 162
4 redis.so 0x000000010b6e9e5c redis_sock_read + 44
5 redis.so 0x000000010b6eaaa4 redis_send_discard + 180
6 redis.so 0x000000010b6e7e74 zim_Redis_pconnect + 132

And

Exception Type: EXC_BAD_ACCESS (SIGSEGV)
Exception Codes: KERN_INVALID_ADDRESS at 0x00000000000000c0

VM Regions Near 0xc0:
-->
__TEXT 0000000109a21000-0000000109a5a000 [ 228K] r-x/rwx SM=COW /opt/local/apache2/*

Application Specific Information:
objc[381]: garbage collection is OFF

Thread 0 Crashed:: Dispatch queue: com.apple.main-thread
0 libphp5.so 0x000000010a297e79 _php_stream_eof + 9
1 redis.so 0x000000010b6e9932 redis_check_eof + 18
2 redis.so 0x000000010b6e9c24 redis_sock_write + 68
3 redis.so 0x000000010b6d7377 zim_Redis_hGet + 247

Rest of the log doesn't concern redis

  • Bug appears in 2.1.1 and 2.1.3 version
  • Redis version 2.4.7
  • Appears on both 32bits and 64bits

Using connect() instead of pconnect() seems to be a temporary fix

wa0x6e commented Feb 20, 2012

+1

There's 2 pattern in OS X crash report :

Exception Type: EXC_BAD_ACCESS (SIGSEGV)
Exception Codes: 0x000000000000000d, 0x0000000000000000

VM Regions Near 0:
-->
__TEXT 0000000109a21000-0000000109a5a000 [ 228K] r-x/rwx SM=COW /opt/local/apache2/*

Application Specific Information:
objc[381]: garbage collection is OFF

Thread 0 Crashed:: Dispatch queue: com.apple.main-thread
0 libphp5.so 0x000000010a297df0 _php_stream_set_option + 16
1 libphp5.so 0x000000010a2a32e2 _php_stream_xport_create + 194
2 redis.so 0x000000010b6e9b41 redis_sock_connect + 321
3 redis.so 0x000000010b6e99c2 redis_check_eof + 162
4 redis.so 0x000000010b6e9e5c redis_sock_read + 44
5 redis.so 0x000000010b6eaaa4 redis_send_discard + 180
6 redis.so 0x000000010b6e7e74 zim_Redis_pconnect + 132

And

Exception Type: EXC_BAD_ACCESS (SIGSEGV)
Exception Codes: KERN_INVALID_ADDRESS at 0x00000000000000c0

VM Regions Near 0xc0:
-->
__TEXT 0000000109a21000-0000000109a5a000 [ 228K] r-x/rwx SM=COW /opt/local/apache2/*

Application Specific Information:
objc[381]: garbage collection is OFF

Thread 0 Crashed:: Dispatch queue: com.apple.main-thread
0 libphp5.so 0x000000010a297e79 _php_stream_eof + 9
1 redis.so 0x000000010b6e9932 redis_check_eof + 18
2 redis.so 0x000000010b6e9c24 redis_sock_write + 68
3 redis.so 0x000000010b6d7377 zim_Redis_hGet + 247

Rest of the log doesn't concern redis

  • Bug appears in 2.1.1 and 2.1.3 version
  • Redis version 2.4.7
  • Appears on both 32bits and 64bits

Using connect() instead of pconnect() seems to be a temporary fix

@timsweb

This comment has been minimized.

Show comment
Hide comment
@timsweb

timsweb Feb 20, 2012

Another +1 on this when attempting to subscribe to a channel.

Setting the default_socket_timeout = 0 throws the exception immediately, -1 and the app hangs forever. All running on php 5.3 on Centos.

timsweb commented Feb 20, 2012

Another +1 on this when attempting to subscribe to a channel.

Setting the default_socket_timeout = 0 throws the exception immediately, -1 and the app hangs forever. All running on php 5.3 on Centos.

@rookie7799

This comment has been minimized.

Show comment
Hide comment
@rookie7799

rookie7799 Feb 22, 2012

+1 to the issue

rookie7799 commented Feb 22, 2012

+1 to the issue

@akitsukada

This comment has been minimized.

Show comment
Hide comment
@akitsukada

akitsukada commented Feb 29, 2012

+1

1 similar comment
@wayne530

This comment has been minimized.

Show comment
Hide comment
@wayne530

wayne530 commented Mar 19, 2012

+1

@colinmollenhour

This comment has been minimized.

Show comment
Hide comment
@colinmollenhour

colinmollenhour Mar 27, 2012

Just adding another data point.. I ran into this when benchmarking Redis as a cache backend. It does not occur for me if connecting over TCP, only if using a unix socket. It occurs more frequently with more concurrent processes. For example, when testing with 64 concurrent clients, all but 17 had the error. When using a standalone PHP driver with all else being equal, there were no such errors.

Debian 64bit with Redis 2.4.9 (dotdeb), phpredis 2.1.3 (master), PHP 5.3.10 (dotdeb) from CLI.

Standalone PHP driver used with no errors is Credis_Client which connects with stream_socket_client and reads with fgets and stream_get_contents.

colinmollenhour commented Mar 27, 2012

Just adding another data point.. I ran into this when benchmarking Redis as a cache backend. It does not occur for me if connecting over TCP, only if using a unix socket. It occurs more frequently with more concurrent processes. For example, when testing with 64 concurrent clients, all but 17 had the error. When using a standalone PHP driver with all else being equal, there were no such errors.

Debian 64bit with Redis 2.4.9 (dotdeb), phpredis 2.1.3 (master), PHP 5.3.10 (dotdeb) from CLI.

Standalone PHP driver used with no errors is Credis_Client which connects with stream_socket_client and reads with fgets and stream_get_contents.

@colinmollenhour

This comment has been minimized.

Show comment
Hide comment
@colinmollenhour

colinmollenhour Mar 27, 2012

Forgot to mention, I am not using persistent connections with either phpredis or standalone mode,

colinmollenhour commented Mar 27, 2012

Forgot to mention, I am not using persistent connections with either phpredis or standalone mode,

@bobrik

This comment has been minimized.

Show comment
Hide comment
@bobrik

bobrik Mar 27, 2012

If you have a BIG bunch of servers (over 100, is really enough) then this error may occur because of syncookie enabled in the kernel (freebsd and linux). Disabling syncookie fixed issue for us.

bobrik commented Mar 27, 2012

If you have a BIG bunch of servers (over 100, is really enough) then this error may occur because of syncookie enabled in the kernel (freebsd and linux). Disabling syncookie fixed issue for us.

@rookie7799

This comment has been minimized.

Show comment
Hide comment
@rookie7799

rookie7799 Mar 27, 2012

setting syncookies to off should be done on redis servers or on clients' servers or both ?
thanks!

rookie7799 commented Mar 27, 2012

setting syncookies to off should be done on redis servers or on clients' servers or both ?
thanks!

@bobrik

This comment has been minimized.

Show comment
Hide comment
@bobrik

bobrik Mar 27, 2012

We disabled it everywhere, but I think server side is more important.

bobrik commented Mar 27, 2012

We disabled it everywhere, but I think server side is more important.

@colinmollenhour

This comment has been minimized.

Show comment
Hide comment
@colinmollenhour

colinmollenhour Mar 27, 2012

I am only running one instance of Redis. What baffles me is that the standalone PHP driver has no read errors on the socket for the exact same benchmark tests, but phpredis does.. So, there is something different with phpredis' connection handling that makes it less stable.

colinmollenhour commented Mar 27, 2012

I am only running one instance of Redis. What baffles me is that the standalone PHP driver has no read errors on the socket for the exact same benchmark tests, but phpredis does.. So, there is something different with phpredis' connection handling that makes it less stable.

@asuth

This comment has been minimized.

Show comment
Hide comment
@asuth

asuth Apr 11, 2012

Same issue on Redis 2.4.10 and php-redis 2.1.3 on OS X 10.7.3 / macports. $redis->blpop('mykey', 0); and getting "read error on connection" after 5.0 seconds every time.

asuth commented Apr 11, 2012

Same issue on Redis 2.4.10 and php-redis 2.1.3 on OS X 10.7.3 / macports. $redis->blpop('mykey', 0); and getting "read error on connection" after 5.0 seconds every time.

@michael-grunder

This comment has been minimized.

Show comment
Hide comment
@michael-grunder

michael-grunder Apr 12, 2012

Member

@asuth How are you connecting to Redis. Are you setting a 5 second timeout? Also, have you checked your redis.conf timeout setting and the default_socket_timeout php.ini setting?

I don't get this problem with the following script:

$r = new Redis();
$r->connect('localhost',6379);

$st = microtime(true);
$r->blpop('mykey',0);
$et = microtime(true);
echo "took: " . ($et-$st) . " to die\n";

I let it run for like ten minutes, and it blocked until I pushed something onto 'mykey'

Member

michael-grunder commented Apr 12, 2012

@asuth How are you connecting to Redis. Are you setting a 5 second timeout? Also, have you checked your redis.conf timeout setting and the default_socket_timeout php.ini setting?

I don't get this problem with the following script:

$r = new Redis();
$r->connect('localhost',6379);

$st = microtime(true);
$r->blpop('mykey',0);
$et = microtime(true);
echo "took: " . ($et-$st) . " to die\n";

I let it run for like ten minutes, and it blocked until I pushed something onto 'mykey'

@rookie7799

This comment has been minimized.

Show comment
Hide comment
@rookie7799

rookie7799 Apr 12, 2012

@michael-grunder not everyone connects to localhost ...

rookie7799 commented Apr 12, 2012

@michael-grunder not everyone connects to localhost ...

@michael-grunder

This comment has been minimized.

Show comment
Hide comment
@michael-grunder

michael-grunder Apr 12, 2012

Member

@rookie7799 They don't? I thought the whole internet was just 'localhost'! Seriously though, I just used localhost as an example rather than put the IP address for one of our production instances. Seemed prudent.

Member

michael-grunder commented Apr 12, 2012

@rookie7799 They don't? I thought the whole internet was just 'localhost'! Seriously though, I just used localhost as an example rather than put the IP address for one of our production instances. Seemed prudent.

@rookie7799

This comment has been minimized.

Show comment
Hide comment
@rookie7799

rookie7799 Apr 12, 2012

I'm just saying that the problem lies somewhere in the network layer and since you're testing on localhost you can't really use that as an argument - no?

rookie7799 commented Apr 12, 2012

I'm just saying that the problem lies somewhere in the network layer and since you're testing on localhost you can't really use that as an argument - no?

@michael-grunder

This comment has been minimized.

Show comment
Hide comment
@michael-grunder

michael-grunder Apr 12, 2012

Member

@rookie7799 For sure, there are just other variables. For instance if you connect with $r->connect($svr,$port,5), you'll get a 5 second timeout. I believe you'll also get that if your default_socket_timeout is 5 in php.ini

The following script will timeout @5 seconds with a "Read error on connection" message, for example

try {
        $r = new Redis();
        if(!$r->connect('localhost',6379,5)) {
                echo "Couldn't connect\n";
                die();
        }
        $st = microtime(true);
        $val = $r->blpop('mykey',0);
        $et = microtime(true);
} catch(Exception $ex) {
        echo "ex: " . $ex->getMessage() . "\n";
        $et = microtime(true);
}

echo "Took: " . ($et-$st) . " to get the key\n";

It could totally be a bug, just good to track those bits down first is all.

Member

michael-grunder commented Apr 12, 2012

@rookie7799 For sure, there are just other variables. For instance if you connect with $r->connect($svr,$port,5), you'll get a 5 second timeout. I believe you'll also get that if your default_socket_timeout is 5 in php.ini

The following script will timeout @5 seconds with a "Read error on connection" message, for example

try {
        $r = new Redis();
        if(!$r->connect('localhost',6379,5)) {
                echo "Couldn't connect\n";
                die();
        }
        $st = microtime(true);
        $val = $r->blpop('mykey',0);
        $et = microtime(true);
} catch(Exception $ex) {
        echo "ex: " . $ex->getMessage() . "\n";
        $et = microtime(true);
}

echo "Took: " . ($et-$st) . " to get the key\n";

It could totally be a bug, just good to track those bits down first is all.

@yatsukhnenko

This comment has been minimized.

Show comment
Hide comment
@yatsukhnenko

yatsukhnenko Mar 29, 2017

Member

Unfortunately TravisCI is not available and I can't test my changes, so I'll continue tomorrow

Member

yatsukhnenko commented Mar 29, 2017

Unfortunately TravisCI is not available and I can't test my changes, so I'll continue tomorrow

@yatsukhnenko

This comment has been minimized.

Show comment
Hide comment
@yatsukhnenko

yatsukhnenko Mar 30, 2017

Member

Who can test a patch?

diff --git a/library.c b/library.c
index 9256005..f5463d2 100644
--- a/library.c
+++ b/library.c
@@ -2102,9 +2102,12 @@ redis_sock_gets(RedisSock *redis_sock, char *buf, int buf_size,
         return -1;
     }
 
+    again: *line_size = 0;
     if(php_stream_get_line(redis_sock->stream, buf, buf_size, line_size) 
                            == NULL) 
     {
+        int err = php_socket_errno();
+        if (*line_size == 0 && (err == EAGAIN || err == EWOULDBLOCK)) goto again;
         // Close, put our socket state into error
         REDIS_STREAM_CLOSE_MARK_FAILED(redis_sock);
Member

yatsukhnenko commented Mar 30, 2017

Who can test a patch?

diff --git a/library.c b/library.c
index 9256005..f5463d2 100644
--- a/library.c
+++ b/library.c
@@ -2102,9 +2102,12 @@ redis_sock_gets(RedisSock *redis_sock, char *buf, int buf_size,
         return -1;
     }
 
+    again: *line_size = 0;
     if(php_stream_get_line(redis_sock->stream, buf, buf_size, line_size) 
                            == NULL) 
     {
+        int err = php_socket_errno();
+        if (*line_size == 0 && (err == EAGAIN || err == EWOULDBLOCK)) goto again;
         // Close, put our socket state into error
         REDIS_STREAM_CLOSE_MARK_FAILED(redis_sock);
@yatsukhnenko

This comment has been minimized.

Show comment
Hide comment
@yatsukhnenko

yatsukhnenko Apr 11, 2017

Member

Hey there, I need a feedback :)

Member

yatsukhnenko commented Apr 11, 2017

Hey there, I need a feedback :)

@patademahesh

This comment has been minimized.

Show comment
Hide comment
@patademahesh

patademahesh Apr 27, 2017

We too facing the same issue. @yatsukhnenko does this patch fixes this issue?

patademahesh commented Apr 27, 2017

We too facing the same issue. @yatsukhnenko does this patch fixes this issue?

@yatsukhnenko

This comment has been minimized.

Show comment
Hide comment
@yatsukhnenko

yatsukhnenko Apr 27, 2017

Member

@patademahesh, It should :)

Member

yatsukhnenko commented Apr 27, 2017

@patademahesh, It should :)

@yatsukhnenko

This comment has been minimized.

Show comment
Hide comment
@yatsukhnenko

yatsukhnenko Apr 27, 2017

Member

I'm going to rewrite this patch without goto and merge if it fixes issue for everyone but I'm still waiting for feedback.

Member

yatsukhnenko commented Apr 27, 2017

I'm going to rewrite this patch without goto and merge if it fixes issue for everyone but I'm still waiting for feedback.

@viion

This comment has been minimized.

Show comment
Hide comment
@viion

viion May 13, 2017

@yatsukhnenko how would I be able to test it?

I get this error every day for the past year, driving me insane!

viion commented May 13, 2017

@yatsukhnenko how would I be able to test it?

I get this error every day for the past year, driving me insane!

@yatsukhnenko

This comment has been minimized.

Show comment
Hide comment
@yatsukhnenko

yatsukhnenko May 13, 2017

Member

@viion you should apply the patch I've listed to the source code and compile extension

Member

yatsukhnenko commented May 13, 2017

@viion you should apply the patch I've listed to the source code and compile extension

@matteomartinelli

This comment has been minimized.

Show comment
Hide comment
@matteomartinelli

matteomartinelli May 13, 2017

Hey guys,
I read most of the issue but I didnt find any solution: I'm trying to subscribe to a redis channel but after a fixed amount of time it crashes.

suggestion?

matteomartinelli commented May 13, 2017

Hey guys,
I read most of the issue but I didnt find any solution: I'm trying to subscribe to a redis channel but after a fixed amount of time it crashes.

suggestion?

@yatsukhnenko

This comment has been minimized.

Show comment
Hide comment
@yatsukhnenko

yatsukhnenko May 13, 2017

Member

@matteomartinelli, definitely you are doing something wrong 😂

Member

yatsukhnenko commented May 13, 2017

@matteomartinelli, definitely you are doing something wrong 😂

@matteomartinelli

This comment has been minimized.

Show comment
Hide comment
@matteomartinelli

matteomartinelli May 13, 2017

@yatsukhnenko There's no logic inside my callback, it just raises a PHP Fatal error: Uncaught RedisException: read error on connection independently I receive an event or not

matteomartinelli commented May 13, 2017

@yatsukhnenko There's no logic inside my callback, it just raises a PHP Fatal error: Uncaught RedisException: read error on connection independently I receive an event or not

@matteomartinelli

This comment has been minimized.

Show comment
Hide comment
@matteomartinelli

matteomartinelli May 13, 2017

I misunderstood the $timeout parameter in pconnect method. I thought it would raise an exception after timeout seconds.

BTW, Solved.

matteomartinelli commented May 13, 2017

I misunderstood the $timeout parameter in pconnect method. I thought it would raise an exception after timeout seconds.

BTW, Solved.

@dermanov-ru

This comment has been minimized.

Show comment
Hide comment
@dermanov-ru

dermanov-ru Jun 13, 2017

in my CLI php script was this line
// ini_set('default_socket_timeout', 0); // DOESNT WORK WITH REDIS!!!
just remove this line - solve problem.

dermanov-ru commented Jun 13, 2017

in my CLI php script was this line
// ini_set('default_socket_timeout', 0); // DOESNT WORK WITH REDIS!!!
just remove this line - solve problem.

@virgofx

This comment has been minimized.

Show comment
Hide comment
@virgofx

virgofx Jun 13, 2017

@yatsukhnenko Is there anyway you can test with the code in issue: comment: #70 (comment).... and confirm failure as outlined in the issue and then after, apply your your fix, and confirm that the failure goes away

virgofx commented Jun 13, 2017

@yatsukhnenko Is there anyway you can test with the code in issue: comment: #70 (comment).... and confirm failure as outlined in the issue and then after, apply your your fix, and confirm that the failure goes away

@yatsukhnenko

This comment has been minimized.

Show comment
Hide comment
@yatsukhnenko

yatsukhnenko Jun 14, 2017

Member

@virgofx, are you asking me to test my own patch? 😂

Member

yatsukhnenko commented Jun 14, 2017

@virgofx, are you asking me to test my own patch? 😂

@viion

This comment has been minimized.

Show comment
Hide comment
@viion

viion Jun 14, 2017

@yatsukhnenko i couldn't get your patch working, or it didn't fix my issue. Not sure how to confirm. My issue was happening every day for about 15 minutes constant I'd get the error.

I modified my pconnect timeout to 5 seconds, and now I get this error a couple times a week, which is an improvement. I also have try/catch connect.

I can deal with it a few times a week since its self-recovery. This is on a site that does about 4000-5000 interactions with Redis every minute (about 80% get GETs). I do not know if that has any effect.

viion commented Jun 14, 2017

@yatsukhnenko i couldn't get your patch working, or it didn't fix my issue. Not sure how to confirm. My issue was happening every day for about 15 minutes constant I'd get the error.

I modified my pconnect timeout to 5 seconds, and now I get this error a couple times a week, which is an improvement. I also have try/catch connect.

I can deal with it a few times a week since its self-recovery. This is on a site that does about 4000-5000 interactions with Redis every minute (about 80% get GETs). I do not know if that has any effect.

@virgofx

This comment has been minimized.

Show comment
Hide comment
@virgofx

virgofx Jun 14, 2017

@yatsukhnenko Yes :) Simply because it's probably easier for you once patch built to php redis-test.php than for those who don't work in redis land to try and pull, build, install, copy patch changes, rebuild, etc.

The script is literally self-contained in that comment.

virgofx commented Jun 14, 2017

@yatsukhnenko Yes :) Simply because it's probably easier for you once patch built to php redis-test.php than for those who don't work in redis land to try and pull, build, install, copy patch changes, rebuild, etc.

The script is literally self-contained in that comment.

@yatsukhnenko

This comment has been minimized.

Show comment
Hide comment
@yatsukhnenko

yatsukhnenko Jun 14, 2017

Member

@virgofx, I've used your test script while created patch

Member

yatsukhnenko commented Jun 14, 2017

@virgofx, I've used your test script while created patch

@yatsukhnenko

This comment has been minimized.

Show comment
Hide comment
@yatsukhnenko

yatsukhnenko Jun 14, 2017

Member

@viion ini_set('default_socket_timeout', -1); also doesn't help?

Member

yatsukhnenko commented Jun 14, 2017

@viion ini_set('default_socket_timeout', -1); also doesn't help?

@finelog

This comment has been minimized.

Show comment
Hide comment
@finelog

finelog Jul 12, 2017

hey, guys

I am using 3.1.2 version and encountered this issue too, using ini_set('default_socket_timeout', -1); works.

but it's affecting the whole php process life cycle, so I did some digging, and found this
$redis->setOption(Redis::OPT_READ_TIMEOUT, -1);. just hope it would help.

and from git blame, setOption can be used since 2.2.8

update:
I think I've misunderstood the issue before, there are two situations which triggers the read error:

  1. php.ini default_socket_timeout and Redis::OPT_READ_TIMEOUT, when one of these two values set to a not -1 value(which means no limit), and the phpredis reading a socket timeout exceeds the limit, the RedisException 'read error' will be throwed

  2. the test script provided by eeroniemi will cause the error no matter what timeout value is set,
    although I am not able to solve this issue, I think I find the cause: when you establish a connection between redis-server and the phpredis as $redis and fork, the connection will be used by all children processes, and process 1 could read the response of process 2, or when process 2 trying to read from the socket while process 1 is writing to it (well, I'm not quite sure here...after all sockets are full-duplex), then process 2 will read empty response leads to a RedisException 'read error'.

so, if I make some changes to the test script

@@ -58,9 +58,6 @@
 $iterationsPerFork = 10;
 $timeout = 2;
 
-$redis = new \Redis();
-$redis->connect($host, $port, $timeout);
-
 $forker = new \Forker();
 
 function setRedisData($redis, $key, $data, $i, $idx) {
@@ -72,7 +69,9 @@
 }
 
 for ($i = 0; $i < $forks; $i++) {
-      $forker->fork(function() use ($iterationsPerFork, $forks, $i, $redis, $timeout, $host, $port) {
+      $forker->fork(function() use ($iterationsPerFork, $forks, $i, $timeout, $host, $port) {
+      $redis = new \Redis();
+      $redis->connect($host, $port, $timeout);
         for ($idx = 0; $idx < $iterationsPerFork; $idx++) {
             $key = 'S:' . md5(uniqid($i . $idx, true));
             $data = 'small-data';

then everything goes perfectly

finelog commented Jul 12, 2017

hey, guys

I am using 3.1.2 version and encountered this issue too, using ini_set('default_socket_timeout', -1); works.

but it's affecting the whole php process life cycle, so I did some digging, and found this
$redis->setOption(Redis::OPT_READ_TIMEOUT, -1);. just hope it would help.

and from git blame, setOption can be used since 2.2.8

update:
I think I've misunderstood the issue before, there are two situations which triggers the read error:

  1. php.ini default_socket_timeout and Redis::OPT_READ_TIMEOUT, when one of these two values set to a not -1 value(which means no limit), and the phpredis reading a socket timeout exceeds the limit, the RedisException 'read error' will be throwed

  2. the test script provided by eeroniemi will cause the error no matter what timeout value is set,
    although I am not able to solve this issue, I think I find the cause: when you establish a connection between redis-server and the phpredis as $redis and fork, the connection will be used by all children processes, and process 1 could read the response of process 2, or when process 2 trying to read from the socket while process 1 is writing to it (well, I'm not quite sure here...after all sockets are full-duplex), then process 2 will read empty response leads to a RedisException 'read error'.

so, if I make some changes to the test script

@@ -58,9 +58,6 @@
 $iterationsPerFork = 10;
 $timeout = 2;
 
-$redis = new \Redis();
-$redis->connect($host, $port, $timeout);
-
 $forker = new \Forker();
 
 function setRedisData($redis, $key, $data, $i, $idx) {
@@ -72,7 +69,9 @@
 }
 
 for ($i = 0; $i < $forks; $i++) {
-      $forker->fork(function() use ($iterationsPerFork, $forks, $i, $redis, $timeout, $host, $port) {
+      $forker->fork(function() use ($iterationsPerFork, $forks, $i, $timeout, $host, $port) {
+      $redis = new \Redis();
+      $redis->connect($host, $port, $timeout);
         for ($idx = 0; $idx < $iterationsPerFork; $idx++) {
             $key = 'S:' . md5(uniqid($i . $idx, true));
             $data = 'small-data';

then everything goes perfectly

@itbdw

This comment has been minimized.

Show comment
Hide comment
@itbdw

itbdw Jul 14, 2017

I'm using predis, I solved the problem several days ago.

I set the redis server timeout to a none zero value.

What I found is my redis server hold some long time idle connections, when the client try to new connection, error comes out.

So after I set the timeout, redis serve kill the outdated connection, and here comes peace. Hope helps :)

itbdw commented Jul 14, 2017

I'm using predis, I solved the problem several days ago.

I set the redis server timeout to a none zero value.

What I found is my redis server hold some long time idle connections, when the client try to new connection, error comes out.

So after I set the timeout, redis serve kill the outdated connection, and here comes peace. Hope helps :)

@PuShaoWei

This comment has been minimized.

Show comment
Hide comment
@PuShaoWei

PuShaoWei Oct 10, 2017

@ itbdw you said predis is phpredis extensions or other anything new? You are set up ini_set (' default_socket_timeout ', 1);? It's not very basic

PuShaoWei commented Oct 10, 2017

@ itbdw you said predis is phpredis extensions or other anything new? You are set up ini_set (' default_socket_timeout ', 1);? It's not very basic

@yatsukhnenko

This comment has been minimized.

Show comment
Hide comment
@yatsukhnenko

yatsukhnenko Dec 4, 2017

Member

Closing this issue as it is too old and contains a lot of unnecessary information...
Lets say that ini_set('default_socket_timeout', -1); and $redis->setOption(Redis::OPT_READ_TIMEOUT, -1); fixes it 😄
Please open new issue If somebody can reproduce read error on connection with the latest stable or develop version of phpredis.

Member

yatsukhnenko commented Dec 4, 2017

Closing this issue as it is too old and contains a lot of unnecessary information...
Lets say that ini_set('default_socket_timeout', -1); and $redis->setOption(Redis::OPT_READ_TIMEOUT, -1); fixes it 😄
Please open new issue If somebody can reproduce read error on connection with the latest stable or develop version of phpredis.

@PuShaoWei

This comment has been minimized.

Show comment
Hide comment
@PuShaoWei

PuShaoWei Dec 5, 2017

ini_set('default_socket_timeout', -1)

Ashamed, however, do not solve the problem 😲

php-version: PHP 5.5.25 
phpredis-version: 2.2.8.1-2.el6
redis> info
# Server
redis_version:2.8.20

We use the pconnect, because limit production environment, the number of connections,Is it a problem?😂

PuShaoWei commented Dec 5, 2017

ini_set('default_socket_timeout', -1)

Ashamed, however, do not solve the problem 😲

php-version: PHP 5.5.25 
phpredis-version: 2.2.8.1-2.el6
redis> info
# Server
redis_version:2.8.20

We use the pconnect, because limit production environment, the number of connections,Is it a problem?😂

@hubiter

This comment has been minimized.

Show comment
Hide comment
@hubiter

hubiter Apr 10, 2018

very good~

hubiter commented Apr 10, 2018

very good~

@pavarnos

This comment has been minimized.

Show comment
Hide comment
@pavarnos

pavarnos Jul 10, 2018

for those who don't want to ini_set and change the socket timeout for all sockets on the server, i worked around the issue with the following code
(i needed to pass a timeout in from the command line, so had to parameterise it)

class MyClass 
{
    /** @var \Redis */
    private $redis;

    /** @var int php default socket timeout is 60 seconds for ubuntu */
    private $maxTimeout = 0;

    public function __construct(\Redis $redis)
    {
        $this->redis = $redis;
        // the value may be -1 for some configurations
        $this->maxTimeout = ini_get('default_socket_timeout');
    }
  
    private function getTimeout(int $timeout): int
    {
        // see https://github.com/phpredis/phpredis/issues/492
        if ($timeout <= 0) {
            // want to run forever, but may not be allowed to
            return max(0, $this->maxTimeout - 1);
        }
        if ($this->maxTimeout <= 0) {
            // there is no php maximum
            return $timeout;
        }
        // run for as long as you can
        return min($this->maxTimeout - 1, $timeout);
    }

then call eg $this->redis->blpop('myid',$this->getTimeout($timeout)); in a loop with sleep(1) or usleep() etc

pavarnos commented Jul 10, 2018

for those who don't want to ini_set and change the socket timeout for all sockets on the server, i worked around the issue with the following code
(i needed to pass a timeout in from the command line, so had to parameterise it)

class MyClass 
{
    /** @var \Redis */
    private $redis;

    /** @var int php default socket timeout is 60 seconds for ubuntu */
    private $maxTimeout = 0;

    public function __construct(\Redis $redis)
    {
        $this->redis = $redis;
        // the value may be -1 for some configurations
        $this->maxTimeout = ini_get('default_socket_timeout');
    }
  
    private function getTimeout(int $timeout): int
    {
        // see https://github.com/phpredis/phpredis/issues/492
        if ($timeout <= 0) {
            // want to run forever, but may not be allowed to
            return max(0, $this->maxTimeout - 1);
        }
        if ($this->maxTimeout <= 0) {
            // there is no php maximum
            return $timeout;
        }
        // run for as long as you can
        return min($this->maxTimeout - 1, $timeout);
    }

then call eg $this->redis->blpop('myid',$this->getTimeout($timeout)); in a loop with sleep(1) or usleep() etc

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment