StackExchange.Redis not able to detect automatic failover

[hrishi18pathak]

Hello
We have 2 node redis server setup where 1 node is master and the other is a slave. We have observed that StackExchange.Redis can take anything from few seconds to several minutes to reconnect to the server if master/slave failover happens in the background. i.e. if we keep on using the same ConnectionMultiplexer object as the one created before the failover was triggered we get keep on getting a connection exception. The connectionMultiplexer object takes atleast 1 minute(sometimes more) to recover from this condition. Is this expected? If we create a new ConnectionMultiplexer object during this time its able to connect to the redis server successfully
Thanks

[mgravell]

Can you be specific about what happens in this failover scenario? Socket
death can be quite... particular. Most socket terminations should be
immediately detected, but there is a heartbeat to catch others. Also: what
exact error is it reporting?
On 1 Jul 2014 22:53, "Hrishikesh Pathak" notifications@github.com wrote:

Hello
We have 2 node redis server setup where 1 node is master and the other is
a slave. We have observed that StackExchange.Redis can take anything from
few seconds to several minutes to reconnect to the server if master/slave
failover happens in the background. i.e. if we keep on using the same
ConnectionMultiplexer object as the one created before the failover was
triggered we get keep on getting a connection exception. The
connectionMultiplexer object takes atleast 1 minute(sometimes more) to
recover from this condition. Is this expected? If we create a new
ConnectionMultiplexer object during this time its able to connect to the
redis server successfully
Thanks

—
Reply to this email directly or view it on GitHub
#65.

[hrishi18pathak]

So for a duration of approximately 1 minute i see the following error in logs:
System.TimeoutException: Timeout performing GET key1, inst: 0, queue: 0, qu=0, qs=0, qc=0, wr=
at StackExchange.Redis.ConnectionMultiplexer.ExecuteSyncImpl[T](Message message, ResultProc
nt server) in c:\Users\hrishp\Documents\GitHub\StackExchange.Redis\StackExchange.Redis\StackEx
xer.cs:line 1773
at StackExchange.Redis.RedisBase.ExecuteSync[T](Message message, ResultProcessor`1 processo
\Users\hrishp\Documents\GitHub\StackExchange.Redis\StackExchange.Redis\StackExchange\Redis\Red
at StackExchange.Redis.RedisDatabase.StringGet(RedisKey key, CommandFlags flags) in c:\User
Exchange.Redis\StackExchange.Redis\StackExchange\Redis\RedisDatabase.cs:line 1346
at SampleConsoleApp.Program.Main(String[] args) in c:\Users\hrishp\Documents\Visual Studio
SampleConsoleApp\Program.cs:line 64
StackExchange.Redis.RedisConnectionException: No connection is available to service this opera
at StackExchange.Redis.ConnectionMultiplexer.ExecuteSyncImpl[T](Message message, ResultProc
nt server) in c:\Users\hrishp\Documents\GitHub\StackExchange.Redis\StackExchange.Redis\StackEx
xer.cs:line 1738

If i create a new connection in this interval, everything works as expected. If it helps here is my sample console app code:

static ConnectionMultiplexer Connection
{
get
{
if ((_connection == null))
{
Console.WriteLine("Connection to the server is broken. Trying to connect...");
try
{
string configurationString = "<conn_str with hostname and password>";
_connection = ConnectionMultiplexer.Connect(configurationString);
Console.WriteLine("Connected successfully to the server : {0}", configurationString);
}
catch (Exception e)
{
Console.WriteLine("Exception in connecting to server : {0}", e);
_connection = null;
}
}
return _connection;
}
}

    static void Main(string[] args)
    {

        Connection.GetDatabase().StringSet("key1","value1");
        while (true)
        {
            try
            {

                {
                    Console.WriteLine(Connection.GetDatabase().StringGet("key1"));
                }
            }
            catch (Exception ex)
            {
                Console.WriteLine(ex);
            }

        }

        Console.ReadKey();

    }

[hrishi18pathak]

Any luck reproing this?

[deepakverma]

What I understood is that when Redis Cache is under load and with the failover happening, Stackexchange.redis would try only once to be able to re-establish the connection. I have a pull request that would help to make it honor configuration.retry (which is 3 be default). Please review.

[NickCraver]

There have been many, many improvements on connections and retry since this was logged - I'm going to close this one out. If someone can repro on a current version we'll open it right back up.

[NickMRamirez]

Using StackExchange.Redis 1.2.1

I have three redis nodes, with Sentinel promoting another to master if one fails. I log into the Redis master and cause it to fail vis debug sleep 30. The following client code will receive a System.TimeoutException when trying to write to the master that has failed:

public static void Main(string[] args)
{
        ConnectionMultiplexer redis = ConnectionMultiplexer.Connect("redis1-exp.aclens.local:6300,redis2-exp.aclens.local:6300,redis3-exp.aclens.local:6300");

        while (true)
        {
                var db = redis.GetDatabase();
                db.StringSet("mytime", DateTime.Now.ToLongTimeString()); // throws exception
                Console.WriteLine(db.StringGet("mytime"));
                Thread.Sleep(5000);
        }
}

The following code fixed this for me:

public static void Main(string[] args)
{
            ConnectionMultiplexer redis = ConnectionMultiplexer.Connect("redis1-exp.aclens.local:6300,redis2-exp.aclens.local:6300,redis3-exp.aclens.local:6300");

            while (true)
            {
                try
                {
                    var db = redis.GetDatabase();
                    db.StringSet("mytime", DateTime.Now.ToLongTimeString());
                    Console.WriteLine(db.StringGet("mytime"));
                    Thread.Sleep(5000);
                }
                catch (StackExchange.Redis.RedisConnectionException)
                {
                    // reconnect
                    redis = ConnectionMultiplexer.Connect("redis1-exp.aclens.local:6300,redis2-exp.aclens.local:6300,redis3-exp.aclens.local:6300");
                }
                catch (System.TimeoutException)
                {
                    // reconnect
                    redis = ConnectionMultiplexer.Connect("redis1-exp.aclens.local:6300,redis2-exp.aclens.local:6300,redis3-exp.aclens.local:6300");
                }
            }
}