Is it possible high availability with Redis Cluster when using it as session handler?

[Ciges]

Hi,

We are trying to use Redis Cluster for storing session data using last version of phpredis, 2.2.5, downloaded from github repository the 19 Juin of 2015, configuring redis as sessions handler in php.ini configuraiton file.

We have 6 instances, three as masters and three as slaves and we have tested that storing and reading session data works.

However if we shutdown any master the following error is shown by our application:

Fatal error: Uncaught exception 'RedisException' with message 'Connection closed' in /users/ets01/lib/php/DEIN/class.PSASession.php:64 Stack trace: #0 /users/ets01/lib/php/DEIN/class.PSASession.php(64): session_start() #1 /users/ets01/lib/php/DEIN/class.PSAPage.php(102): PSASession->__construct() #2 /users/ets01/web/inc/class.APP2Page.php(11): PSAPage->__construct('b', 11) #3 /users/ets01/web/html/index.php(11): APP2Page->__construct() #4 {main} thrown in /users/ets01/lib/php/DEIN/class.PSASession.php on line 64

Fatal error: Uncaught exception 'RedisException' with message 'Connection closed' in [no active file]:0 Stack trace: #0 {main} thrown in [no active file] on line 0 

Connection is closed, and cluster does not works as we would like to, we would like that connection is redone to another instance.

Does phpredis driver manage connections to the cluster? Does it provides automatically the chance of getting the value from another cluster instance?

[michael-grunder]

Hey,

First of all, I need to update the version in the develop branch to something like 2.2.9 (as it's a pre-release for 3.0.0) but I missed it!

In terms of failover, you have two options. You can set failover=error and failover=distribute (in your session.save_path) for such a solution. When you set failover to error it should attempt to read session data from a slave in the event that a given master fails, and if you set it to distribute it should randomly pick from the master (or one of the slaves) for every request.

That being said, you may have identified a bug. Even if a master goes down you shouldn't get that exception at all. To use a cluster as your session storage you need to set session.save_handler to rediscluster instead of redis.

Also, the session.save_path for redis cluster based session storage has a different format than the non cluster handler. Here are some example scripts which might be helpful:

ses-std.php

<?php
ini_set('session.save_handler', 'rediscluster');
ini_set('session.save_path', 'seed[]=127.0.0.1:7000&seed[]=127.0.0.1:7001&seed[]=127.0.0.1:7002&timeout=3&read_timeout=3&failover=error');
session_id('my_session_id');
?>

ses-in.php

<?php
require_once('ses-std.php');

session_start();

for ($i = 0; $i < 100; $i++) {
    $_SESSION['var-'.$i] = 'my_special_value:' . $i . (isset($argv[1]) ? '-c' : '');
}
?>

ses-out.php

<?php
require_once('ses-std.php');

session_start();

for ($i = 0; $i < 100; $i++) {
    var_dump($_SESSION['var-'.$i]);
}
?>

Thanks very much for the report!

Cheers,
Mike

[CarlitoxxPro]

Hello michael
I describe the procedure we followed to configure and manage the php sessions with the redis cluster:

we have in the same host, 3 master instances of redis and 1 slave for each one.

runing the instances:

./bin/redis-server /redis-cluster/red01/conf/redis.conf
./bin/redis-server /redis-cluster/red02/conf/redis.conf
./bin/redis-server /redis-cluster/red03/conf/redis.conf
./bin/redis-server /redis-cluster/red04/conf/redis.conf
./bin/redis-server /redis-cluster/red05/conf/redis.conf
./bin/redis-server /redis-cluster/red06/conf/redis.conf

create a cluster:
./redis-trib.rb create --replicas 1 127.0.0.1:7001 127.0.0.1:7002 127.0.0.1:7003 127.0.0.1:7004 127.0.0.1:7005 127.0.0.1:7006

in other host configure the php.ini:

session.save_handler = rediscluster
session.save_path = "seed[]=HOSTNAME:7001&seed[]=HOSTNAME:7002&seed[]=HOSTNAME:7003&seed[]=HOSTNAME:7004&seed[]=HOSTNAME:7005&seed[]=HOSTNAME:7006&timeout=3&read_timeout=3&failover=distribute"

and we receive this error when try to connect:
Fatal error: Uncaught exception 'RedisClusterException' with message 'Can't communicate with any node in the cluster'

we use apache 2.2, php 5.4.36 and the last development version of phpredis.

any help is apreciated.

Cordial regards.

[michael-grunder]

Hi,

The error you're getting means exactly what it says. It's thrown when phpredis is unable to connect/talk to any of the nodes (and therefore can not map the key space).

Can you connect to the cluster with redis-cli or by using the non-session functionality of phpredis? The error you're getting strikes me as an issue with putting the wrong host:port combination for the seed(s), or some kind of firewall/permissions issue preventing the client machine from connecting to the server.

Cheers,
Mike

[Ciges]

Hello again,

We have been doing some tests (with last code version) with the following results:

• With redis-cli console client connexion is done with success, so network or firewall problem is discarded
• With session configuration the 'RedisClusterException' is thrown
• With a little php script using RedisCluster class to send a GET command also the same 'RedisClusterException' is thrown
• If instead of connecting to remote machine we connect to localhost (using exactly the same code and the RedisCluster class as before) connexion and key recover is done
• With the same php script but using Redis class to connect to one instance the GET command is succesfull

So, in conclusion, using RedisCluster class to connect to a remote machine does not work, so I think we have discovered a bug.

Does this data helps you? If you need more help or collaboration from us to make tests don't hesitate to ask for it.

It could be interesting maybe to add log commands through the code to save in a text file what is happening.

Thanks you very much for your time and reactivity!

[michael-grunder]

Hey,

Thanks for the update. The information certainly helps narrow it down although I will admit it's odd. Connecting to localhost or a remote machine is transparent to me (I'm just using PHP streams), so in theory it shouldn't matter at all.

Something is obviously happening though, so the trick is to figure it out 😃

Have you played around with your timeout and read_timeout options? Perhaps the remote connection is triggering a timeout? Also, have you tried running your test script via the command line (vs. inside the web server)? That's another variable that we can remove.

I'll try and replicate the issue by connecting to a remote server with cluster myself and maybe I'll get lucky and trigger the error.

[michael-grunder]

Hey there,

Another quick update. I re-checked the cluster logic and this error occurs after I'm able to map the keyspace. What that means is that phpredis cluster is able to at least execute a CLUSTER SLOTS command on one of the seed nodes, such that it can determine which server to send your get command.

Other than timeouts, I'm still not totally sure what's happening here, but that at least provides more information. If increasing your timeout values don't work, I might have to instrument the code for you (in a different branch) to print some debug information which might help us.

Cheers
Mike

[Ciges]

Hello,

I have been making some new tests and there is one thing that has called my attention.

The code I have tested, which runs in one server and connects to a remote serve with IP 10.68.10.201, is the following

$obj_cluster = new RedisCluster(NULL, Array('10.68.10.201:7001', '10.68.10.201:7002', '10.68.10.201:7003', '10.68.10.201:7004', '10.68.10.201:7005', '10.68.10.201:7006'));
var_dump($obj_cluster->_masters());
$obj_cluster->set("prueba", "mec");

If there is a Redis cluster in the same machine that runs the script, locally, then the script waits for ever, in the set command. If I kill the local redis cluster then the 'RedisClusterException' with message 'Can't communicate with any node in the cluster'is thrown.

I have a console with tcpdump open to verify network connections and I see that there is communication between both machines.

So I think there is a bug in the code of set and get commands which causes that requests are beeing sent to localhost instead of the IP's signaled when RedisCluster instance is created.

Does this helps you?

[michael-grunder]

Aha,

I may have an idea. Can you please run CLUSTER SLOTS on the remote instance and paste the output here?

I bet it's exporting '127.0.0.1' for all of the nodes, and this is a problem with the cluster configuration. Just a guess though. 😃

Thanks!
Mike

[Ciges]

127.0.0.1:7002> cluster slots
1) 1) (integer) 10923
   2) (integer) 16383
   3) 1) "127.0.0.1"
      2) (integer) 7006
   4) 1) "127.0.0.1"
      2) (integer) 7003
2) 1) (integer) 5461
   2) (integer) 10922
   3) 1) "127.0.0.1"
      2) (integer) 7002
   4) 1) "127.0.0.1"
      2) (integer) 7005
3) 1) (integer) 0
   2) (integer) 5460
   3) 1) "127.0.0.1"
      2) (integer) 7004
   4) 1) "127.0.0.1"
      2) (integer) 7001

[michael-grunder]

There is your answer. It's showing 127.0.0.1 as the host for the cluster even on the remote machine. This means that phpredis is trying to connect to 127.0.0.1 even though you're trying to connect to a different machine.

You'll need to use the machine's ip address instead of 127.0.0.1 in your redis-trib command.

Cheers
Mike

[michael-grunder]

Were you able to get the issue fixed?

[Ciges]

Hello,

Thank you very much for your help. Yes, we resolved the problems.

At the moment of creating the cluster we have to use the server IP instead of 127.0.0.1

redis-trib.rb create --replicas 1 10.68.10.201:7001 10.68.10.201:7002 10.68.10.201:7003 10.68.10.201:7004 10.68.10.201:7005 10.68.10.201:7006

Now setting and getting values in the cluster from a PHP script works. We have problems now with using it as session handler, depending of the PHP version we use.

We have tested versions 5.3.10, 5.4.36 and 5.6.1 and works only for the last one, for previous versions we have a "segmentation fault" error.

Anyway we are still making tests, we need to package php 5.6.80 for our linux servers (very, very customized).

Again, thanks you very much for your support.

[michael-grunder]

Interesting. OK looks like I need to download older versions of php and see what the issue is. There are other users reporting similar issues so I think the segfault situation is an actual bug.

Cheers!
Mike

[mdular]

Hey, thanks for the info in this thread. Hit the same roadblock though - is there any update on this?
We are also experiencing the SIGSEGV
PHP 5.4.41

[Rastusik]

This wasn't merged into master, was it?

[leafpeak]

@michael-grunder is a new release going to be cut with cluster support? It seems 2.2.9 release was never cut.

[yatsukhnenko]

Old issue without any activity for a long time. Feel free to open new issue if you see this issue with the latest stable version of phpredis.