Error while reading line from the server

[googleguy]

Using redis for php 5.2 branch. I'm getting random "Error while reading line from the server" error. The command i used is lpush a serialized string.

I'm using predis on a php daemon script, meaning it's up all the time. I've already set the timeout for redis.conf to 0.

[nrk]

Have you already followed the steps described here to debug your issue before opening a new issue with the very same title? See also this thread on the Redis list, it might also be something unrelated to client-side issues.

[googleguy]

Thank you nrk. I will try setting the socket_timeout and read_write_timeout and report my findings here later.

[nrk]

If you are using Predis in a daemon-like script you should set read_write_timeout to -1 if you want to completely disable the timeout (this value works with older and newer versions of Predis). Also, remember that you must disable the default timeout of Redis by setting timeout = 0 in redis.conf or Redis will drop the connection of idle clients after 300 seconds of inactivity.

[googleguy]

nrk, tried as mentioned and it's now working perfectly. Thank you!

However, there's still a lot of connection timeouts now and then. I've already set timeout = 0 and read_write_timeout to -1. Is there anything that we can do to debug this when it occurs?

[nrk]

timeout is not a connection parameter recognized by Predis, you should use connection_timeout instead. The default value for connection_timeout is 5 seconds, you could try to raise its value but I don't think it's a solution to your problem. You should try and see what's going on on your server when those timeouts occurs, it really depends on your application so I can't be of any help. The possible reasons for those timeouts could be:

1. network connectivity issues if your Redis server is not on the localhost
2. are you sure you are not using expensive calls to the KEYS command somewhere in you scripts? Depending on the number of keys store on Redis, KEYS can end up blocking the server which, in the meanwhile, it will not be able to process other requests or incoming connections.

I'll leave this issue open for now, but I'm quite sure it's definitely not something related to Predis.

[googleguy]

nrk. As advised, I've raised connection_timeout to 30. Will be monitoring this.

1. Yes, we are using multiple redis nodes that are not localhost but they are on private network (so at least should be more stable?)
2. Not using KEYS. However, we are snapshotting every 900s. From the log it appears that the system is taking 8s to save to HD. Shouldn't be the cause of the connection timeout right?
3. Any advise as to how to monitor the server when the timeout occurs? The timeout's pretty random. I checked redis log, and mostly it's just snapshotting messages.

[googleguy]

Another update, I measured the connection, and it appears that indeed the exception occured after 5s. I tried modifying the connection_timeout via:

new Predis_Client($param, array('read_write_timeout' => -1, 'connection_timeout' => 30));

Is that correct? It doesn't seems to take effect the 30s connection_timeout and still throw exception.

[nrk]

They are connection parameters and not client options, so read_write_timeout and connection_timeout must be specified as parameters in $param.

1. Unless it's not a private network on Amazon EC2 (high latency and various issues from time to time), then it shouldn't.
2. Snapshotting shouldn't block the server as far as I know
3. You can check the timings of the timeouts against the snapshotting logs. Other than that, it really depends on the architecture of your application and your infrastructure, sorry but I can't give you support on that.

Honestly, Predis doesn't do anything fancy with the socket resources when connecting to a host since it's something delegated almost entirely to PHP's internals, so it can be either a bug in PHP (unlikely, but still a possibility) or some configuration/runtime issue on your servers, or Redis doing some heavy operations that end up blocking the server for a while.

[googleguy]

nrk, thank you again for your reply. I found the problem, and it's just the server running out of ip_conntrack issue. Once the conntrack is fixed, the connection timeout goes away as well.

[nrk]

Nice to know that you have finally found the actual problem behind those timeouts. I will probably add this somewhere in the FAQ to give users an initial list of checks for troubleshooting timeout issues.

[amitchhajer]

was getting the same error, solved using setting read_write_timeout as 0.
I did that in predis, can we do same using predis client ?
as shown here http://code.google.com/p/dires/source/browse/trunk/predis/examples/PubSubContext.php?r=4
I am making predis client using : $redis = new Predis\Client("tcp://".$turboConfig->getActivitiesRedisHost()); how should i pass read_write_timeout in my implementation.

[nrk]

@amitchhajer the client accepts the same connection parameters with both named arrays or URI strings, so if you are using an URI string you can just add it as if it were a query string: tcp://127.0.0.1:6379?read_write_timeout=0.

[amitchhajer]

@nrk thanks for the info, works pretty well now.

[bondeg]

What if we need to change read_write_timeout at runtime? Is some setter for timeout params (except constructor) exists?

[nrk]

@bondeg connection parameters are intentionally immutable which means you can't change them once the connection has been initialized, but you can work around this limitation by fetching the underlying stream resource from the connection object used by the client and change the stream options accordingly. Here is a small snippet, assuming you are not using the client in cluster or replication mode:

$connection = $client->getConnection();
$stream = $connection->getResource();
stream_set_timeout($stream, 2);

Please note that doing $connection->getResource() effectively triggers the connect() operation to Redis so you might end up losing the benefits of lazy connections depending on how you are going to use this feature.

[szegad]

Please notice that the parameter connection_timeout has been renamed to timeout in the new version of Predis.

[mityukov]

I'm using 'hkeys' pretty extensively. Can this lead to locks?

[aditya-rewari-cb]

If you are using Predis in a daemon-like script you should set read_write_timeout to -1 if you want to completely disable the timeout (this value works with older and newer versions of Predis). Also, remember that you must disable the default timeout of Redis by setting timeout = 0 in redis.conf or Redis will drop the connection of idle clients after 300 seconds of inactivity.

Any drawback or caution associated with setting timeout=0 at redis.conf ?

What I suspect is ..as connections will never drop
Redis utilisation will stay high

[aditya-rewari-cb]

nrk, thank you again for your reply. I found the problem, and it's just the server running out of ip_conntrack issue. Once the conntrack is fixed, the connection timeout goes away as well.

How to fix this contract.. where to look for ?

[nrk]

Any drawback or caution associated with setting timeout=0 at redis.conf ?

@aditya-rewari-cb actually timeout = 0 in redis.conf is the default since Redis 2.4 (released a few years ago) so I'd say there are no drawbacks.

[aditya-rewari-cb]

If you are using Predis in a daemon-like script you should set read_write_timeout to -1 if you want to completely disable the timeout (this value works with older and newer versions of Predis). Also, remember that you must disable the default timeout of Redis by setting timeout = 0 in redis.conf or Redis will drop the connection of idle clients after 300 seconds of inactivity.

@nrk I am using Redis with daemon proceses (laravel supervisor workers), as well as , with regular caching
Any caution for using 'read_write_timeout' => -1 ?
I wanted to confirm for any possibility of breakage or bug.. in my regular caching usage of Redis, owing to this change !

Thanks !

[jackbaron]

@aditya-rewari-cb I have a same problem. You found a solution for it?

[aditya-rewari-cb]

@aditya-rewari-cb I have a same problem. You found a solution for it?

I have vague memory of this now!
But, we went ahead with read_write_timeout=0