[Question]how to set redis get command timeout

[cekimy]

I've meet a problem several times. redis.get cost much time, more than 1s (even more than 1 minute)
This is a terrible online disaster, so i want to know how to set timeout of get command.
Or what could cause redis cost so much time?

[stockholmux]

Could you tell me more about your setup and the code that is causing this delay? Is the server remote or local? Is this your first call to Redis in your script after a restart?

This is very unusual - Redis usually takes ~1ms to do a simple operation (like get). I would be surprised if this is actually a node_redis problem and more likely something to do with your environment and how Node is communicating (or not) with your Redis server.

[cekimy]

server is remote, but in the same IDC. Most of the time, it only takes ~1ms, but this >1s situation came about 1time/4-5months, and continue several seconds.

i have 4 redis servers, each key will hash to one of them, and then redis.get(This caused delay).
Our service has about 500,000,000+ requests one day, and each redis server has 10000+ operation per second on the rush hour.
The server is usually stable running, this problem appears suddenly, and disappears a few seconds later.

[stockholmux]

It's hard to say, but I would venture it is actually not a node_redis issue from what you're telling me - I can't think of a thing in the project code that would cause a stable service to take that long. I'm guessing that it has to do with something busy on the server (redis or node).

As far as a timeout, I don't think there is anything in the project to set a specific timeout for a single command. It could be done in your app - perhaps with a closure and a setTimeout - run the setTimeout and the get at the same time, then use a variable to see which one has completed first. It's definitely a hack, but it could work.

[stockholmux]

I got around to writing a gist on how to do a timeout for a redis command:

https://gist.github.com/stockholmux/3a4b2d1480f27df8be67#file-timelimitedredis-js

@cekimy Can we close this issue since it's seems more like a redis server stall issue and there is a non-library way of addressing the issue?

[cekimy]

@stockholmux ok, thanks very much, I'll try this gist.

[cekimy]

i still have this problem, when the network between node server and redis server has problem, redis.get will cost hundreds seconds.

see our monitor: mostly it takes avg/1ms

this is my simple code:

redis.js

var redis = require("redis"), h = null;
var client = redis.createClient(6379, '10.173.2.22', {enable_offline_queue: false, retry_max_delay: 100});

addListener(client, 'error', null);
addListener(client, 'connect', client);

function addListener(handler, evt, value) {
    handler.on(evt, function() {
        h = value;
    });
}

var puppet = {
    'get': function(key, callback) {
        if (h === null) {
            //reporting error
            callback("redis error", null);
        } else {
            h.get(key, callback);
        }
    },
    //other commands
};

module.exports = puppet;

app.js

var http = require('http');
var redis = require("redis.js");

http.createServer(function (req, res) {
    //...
    var t1 = Date.now();
    redis.get('key', function(err, reply) {
        var t2 = Date.now();
        monitor("time_redis", t2 - t1);     //this is reporting to above pic's monitor,calculate the average value
        //...
    });
    //...
}).listen(1337, "10.23.5.42");

[BridgeAR]

@cekimy what node_redis version and what parser do you use?

[cekimy]

redis@0.12.1
hiredis@0.4.0
and use default parser:hiredis

[BridgeAR]

Hm, this is weird. Since you do not use the offline queue. I can't think of a reason why this could happen. Are you certain that you only use .get? Or what commands do you use?

[cekimy]

i use .get, .set, .incr, .hgetall...and so on, but i only monitor the cost of every .get command.

[BridgeAR]

Hm, you said there were connection issues while the huge delay occured? I can't explain such a delay right now and wonder what happend on your side in this case.

And did you try the code from @stockholmux? By having a glimpse at it, it looks good to me for solveing your issue.

[cekimy]

ok，thanks