Add support for Sentinel and RE Discovery Service

[tanguylebarzic]

Hi,

Are there plans to make node_redis sentinel aware? I'm thinking about being able to use a list of sentinels at startup, then discover and connect to the masters, and subscribing to sentinels message to take into account masters changes. If not, would be happy to start working on it!

Tanguy

[joaojeronimo]

+1 , why not make a wrapper on top of this node_redis client ? If you start working on it please let me know because I'd like to star that repository.

[tanguylebarzic]

Hi,

I've started working on this, the result can be checked at https://github.com/tanguylebarzic/node_redis
An example of use:

var RedisMetaClient = require("./../node_redis/sentinel").RedisMetaClient;
var sentinels = [
    {host: "127.0.0.1", port: 26382},
    {host: "127.0.0.1", port: 26383},
    {host: "127.0.0.1", port: 26384}
];
var redisMetaClient = new RedisMetaClient("mymaster", sentinels);
var i = 0;

var client = redisMetaClient.createMasterClient();
client.on('error', function(error){
    console.log(error);
});

setInterval(function(){
    client.set('test:' + i, i, function(error, result){
        if(!error){
            console.log('success, ' + i);
            ++i;
        }
        else {
            console.log('error, ' + i);
        }
   });
}, 500);

Basically, you first have to create a 'RedisMetaClient', configured with the name of the master and a list of sentinels. Then, calling redisMetaClient.createMasterClient(); will give you a client similar to the usual redis.createClient();

What works:

• Basic functionnality, ie. setup of the correct master, failover... If the master change, all the current masterClients (created by redisMetaClient.createMasterClient(); will be updated to point to the new master, so that one doesn't need to look for client.on('error') to update the connection.
  I have try to preserve the idea of an offline queue, ie. being able to send commands before the master is found and configured.
• I have tested with simple requests, pub/sub... but not with multi (may work, just not sure).

How was is it done:
I've tried to modify as less as possible the code and behaviour of the single client (what's in 'client.js'). sentinel.js contains kind of a wrapper around client.js, to deal with sentinels.

What remains to be done:

• Dealing with slaves (for now, nothing is done around them).
• Testing (both by hand and automatic) of multiple scenarios
• Documentation

Known issues:

• the event ready is triggered multiple times for a masterClient if a failover happened, which can result in unexpected behaviours based on the way it's used
• many others that I don't know about

[benbuckman]

tanguylebarzic, thanks for starting to tackle sentinel support. I'm confused by your approach, however: You seem to be implementing the sentinel functionality (e.g. identifying the master and slaves, handling a quorum) in your node.js code. But in the sentinel docs that is all set up in the redis-sentinel daemon's config. Node shouldn't care about who the master or slaves are for a given sentinel, it should simply connect to the sentinel, which is handling all that already. (And that SentinelClient should behave like a regular RedisClient, so it's backwards-compatible with connect-redis session store, etc.)

Can you clarify your approach? Thanks!

[benbuckman]

I've started implementing an alternate approach with a RedisSentinelClient that behaves transparently like a RedisClient: https://github.com/DocuSignDev/node_redis

Always interested in feedback, suggestions, and/or collaborators.

[tanguylebarzic]

newleafdigital: sorry for the late reply. indeed, I noticed it was not clean to implement this the way I did (too much logic on the client). I've changed it to better match antirez' guidelines, although there are still some questions remaining IMO. Your approach looks interesting (and add cluster awareness!), I'm going to look at it!

[jochenonline]

@newleafdigital: I have tried your fork together with socket.io. When I setup the redisstore:

io.set('store', new socketio.RedisStore({
    redis: redis,
    redisPub:   redis.createClient( 26379, "127.0.0.1", {sentinel: true} ),
    redisSub:   redis.createClient( 26379, "127.0.0.1", {sentinel: true} ),
    redisClient:redis.createClient( 26379, "127.0.0.1", {sentinel: true} )
 }));

My app crashes on startup:

 Error: Redis connection to 127.0.0.1:6379 failed - connect ECONNREFUSED
     at RedisClient.on_error (/home/green/debug/node_modules/redis/index.js:168:24)
     at Socket.RedisClient.initialize_stream_listeners.stream.on.self.should_buffer (/home/green/debug/node_modules/redis/index.js:93:14)
     at Socket.EventEmitter.emit (events.js:96:17)
     at Socket._destroy.self.errorEmitted (net.js:328:14)
     at process.startup.processNextTick.process._tickCallback (node.js:244:9)
 [Wed, 13 Feb 2013 16:46:56 GMT] INFO worker 6622 died

Please be aware that the error message sais that the connection failed to port 6379, instead of 26379!!!

Any idea what could be the problem? When I connect directly to the redis-server (master) everything is fine.

[benbuckman]

Try using a single hash for the parameters:

redis.createClient({ port: 26379, host: "127.0.0.1", sentinel: true } );

When you connect to the sentinel via redis-cli -p 23679 (not to the master - sentinel != master), does it connect?

[jochenonline]

1. Yes, connecting to the sentinel via redis-cli -p 26379 connects successfully.
2. Using the alternate syntax doesn't crash on connect but returns an error on the first get of the redisSentinelClient:

[Error: ERR unknown command 'get']

In that case I used the normal redis functionality (not socket.io).

[jochenonline]

Hhhhhm. Maybe no problem of your module....When I connect to the sentinel it connects successfully, but it does not know any redis commands:

green@mycomp1:~/redis/redis$ src/redis-cli -p 26379
redis 127.0.0.1:26379> set test 1
(error) ERR unknown command 'set'
redis 127.0.0.1:26379>

What could be the reason for this? Just to be clear: I am connecting to the sentinel, neither to the master nor to one of the slaves. Correct?

ADDENDUM: Reading the docs more in detail I saw that this behavior is normal. But why does the your client do the same? Shouldn't it behave as a "normal" redisClient?

[benbuckman]

The sentinel itself shouldn't handle get/set/etc, but the sentinel client should handle it, by delegating it to the master.
Could you put some debugging code in the SentinelClient constructor, to make sure it's instantiating a SentinelClient and not a regular client?
There's also a sentinel test suite that we added, run with Mocha -- maybe run that (changing the port) and see if those pass on your system?

Thanks

[jochenonline]

@newleafdigital: Is it possible that you expect the sentinel running in the same machine as redis? I have my sentinel running on a different machine (the one on which node runs) and I doubt that this scenario works for the current implementation. I still get the error

 Error: Redis connection to 127.0.0.1:6379 failed - connect ECONNREFUSED

and that leads me to the conclusion that it tries to connect to a redis instance on 127.0.0,1 === this machine.

[jochenonline]

This seems to be a specific problem of using your client with socket.io. Did you ever test the two together?

[jochenonline]

@newleafdigital: Now I seem to have found out what is going on. I did not deeply dig into your code but I can quite exactly describe the behaviour:

In my current configuration (node-app on machine1, redis_master on machine2, redis_slaves on machines3+4, redis_sentinels on machines2+3+4) your code only works if I connect to one of the slave_sentinels on machine3+4. It does not work if I connect to the sentinel on the master (machine2).

When I connect to one of the two slave_sentinels the systems logs correctly:

debug connected to sentinel listener
debug connected to sentinel talker
debug new master info { host: 'the.ip.of.master', port: '6379' }
debug Changing master from 127.0.0.1:9999 to the.ip.of.master:6379
debug New master is ready [Yippieee!]

When I connect to the sentinel on the master it sais (and does not work afterwards):

debug connected to sentinel listener
debug connected to sentinel talker
debug new master info { host: '127.0.0.1', port: '6379' } 
debug Changing master from 127.0.0.1:9999 to 127.0.0.1:6379

and no debug New master is ready. As you can see, when I connect to the master's sentinel, the master is not addressed correctly (127.0.0.1 instead of the correct ip).

If the sentinels run on other machines but the redis' machines (i.e. on the same machine like the node-app - as in my very first configuration) it only works too, if the sentinel is connected to one of the slaves (and not the master).

And...together wird socket.io/RedisStore it does not work either (concerning my configuration) in any way, even if I connect to the sentinel on one of the slaves.

[DTrejo]

+1 this is great as a separate module as it is much more likely to change.

If you'd like to have your module mentioned in the wiki, come up with a tag for it e.g. #redis-sentinel and then
I'll add a link in the README to all packages with that tag e.g. https://npmjs.org/browse/keyword/redis-sentinel

[benbuckman]

Thanks for all the feedback here, and sorry I haven't had a chance to reply in depth. I've allocated myself time for this next week (March 4).

@DTrejo, you're right about making this its own module, I think I'll do that (next week as well).