ServiceStack.Redis: "Unable to Connect: sPort: 0" and Unable to cast object of type 'System.Byte[]' to type 'System.Object[]'

[joe-udwin-lrn]

I have a test harness that repeatedly gets and sets DateTime objects so we can test fail-over of our new Redis Sentinel deployment.

I have added my license key to the static constructor of the calling class.

I have not made any client configuration so it should be using the defaults.

Gets and Sets succeed 99% of the time but I am getting intermittent failures like the ones below.

1. Unable to Connect: sPort: 0

   ServiceStack.Redis.RedisException: Unable to Connect: sPort: 0
   at ServiceStack.Redis.RedisNativeClient.CreateConnectionError()
   at ServiceStack.Redis.RedisNativeClient.SendReceive[T](Byte[][] cmdWithBinaryArgs, Func1 fn, Action1 completePipelineFn, Boolean sendWithoutRead)
   at ServiceStack.Redis.RedisNativeClient.SendExpectMultiData(Byte[][] cmdWithBinaryArgs)
   at ServiceStack.Redis.RedisNativeClient.SentinelGetMasterAddrByName(String masterName)
   at ServiceStack.Redis.RedisSentinelWorker.GetMasterHost(String masterName)

   Other posts claim this could be related to the ConnectTimeout, but we are not setting it specifically and therefore the default (-1 / not set) should be being used.
   What is the issue?

2 . Unable to cast object of type 'System.Byte[]' to type 'System.Object[]'

2016-07-15 15:00:04,825 ERROR Caching.CacheCore.RedisCacheCore - Get<T> failed with key @#qm:qmth:testhardness-3be0f6bb-5df5-4205-afe4-45ee56b59a71
2016-07-15 15:00:04,825 ERROR Tools.RedisTestHarness.Program - READ - EXCEPTION [Duration: 00:00:00.4976532]: Unable to cast object of type 'System.Byte[]' to type 'System.Object[]'.
2016-07-15 15:00:04,825 DEBUG Tools.RedisTestHarness.Program -    at System.Linq.Enumerable.<CastIterator>d__1`1.MoveNext()
   at System.Linq.Enumerable.WhereSelectEnumerableIterator`2.MoveNext()
   at System.Collections.Generic.List`1..ctor(IEnumerable`1 collection)
   at System.Linq.Enumerable.ToList[TSource](IEnumerable`1 source)
   at ServiceStack.Redis.RedisNativeClient.SentinelSlaves(String masterName)
   at ServiceStack.Redis.RedisSentinelWorker.GetSlaveHosts(String masterName)
   at ServiceStack.Redis.RedisSentinelWorker.GetSentinelInfo()
   at ServiceStack.Redis.RedisSentinel.GetSentinelInfo()
   at ServiceStack.Redis.RedisSentinel.GetRedisManager()
   at ServiceStack.Redis.RedisSentinel.GetValidSentinelWorker()
   at ServiceStack.Redis.RedisSentinel.Start()
   at Caching.CacheCore.RedisCacheCore.CreateManagerForSentinel(String connectionString, String sentinelHostFilter)
   at Caching.CacheCore.RedisCacheCore.CreateClientManager(String connectionString, Boolean useSentinel, String sentinelHostFilter)
   at Caching.CacheCore.RedisCacheCore.GetClient()
   at Caching.CacheCore.RedisCacheCore.Get[T](String key)
   at Caching.CacheCore.CacheCoreWrapper.<>c__DisplayClass1`1.<Get>b__0()
   at Caching.FunctionHandlers.ExecuteFunctionHandler.HandleFunction[T](Func`1 functionToHandle)
   at Caching.CacheCore.CacheCoreWrapper.Get[T](String key)
   at Caching.Cache.Get[T](String key)
   at Tools.RedisTestHarness.Program.Main(String[] args) in ...\Tools.RedisTestHarness\Program.cs:line 123

This makes no sense to me. Any ideas?

Both of these issues are preventing us from deploying to production.

Thanks for your help in advance.

[mythz]

The first error suggests the client is unable to connect to the server, and the second error suggests the server returned an unexpected response which can happen if you're using the same client instance across multiple threads (i.e. instead of retrieving a new client from the pool in each thread).

But we'd need a small, stand-alone repro to identify the issue. Can you provide a small, stand-alone repro of the issue, if it's in one file you can post it here, otherwise can you put it in a new Github repository?

[joe-udwin-lrn]

I have significantly reduced the frequency of the errors occurrence.
The code to make the IRedisClientsManager a singleton was failing so the code was effectively like the following:

 static void Main(string[] args)
        {
            for (int i = 0; i < 10000000; i++)
            {
                using (var cache = CreateManager().GetClient())
                {
                    cache.Set<string>("key", "value");
                    cache.Get<string>("key");
                }
                Thread.Sleep(10);
            }
        }

        static IRedisClientsManager CreateManager()
        {
            return new RedisSentinel(sentinelList, MASTER_NAME).Start();
        }

I am now storing the IRedisClientsManager in a ConcurrentDictionary keyed by the connection string details but I find if I start my test harness 1 time out of 20 I still get the error on first launch. Any ideas? The code is currently in a single threaded console application.

[xplicit]

That's looks strange that you create manager every time in a loop. It should be created once and then get the client from it on every operation. Is the following code not working for you?

    static void Main(string[] args)
    {
       var manager =  new RedisSentinel(sentinelList, MASTER_NAME).Start();

        for (int i = 0; i < 10000000; i++)
        {
            using (var cache = manager.GetClient())
            {
                cache.Set<string>("key", "value");
                cache.Get<string>("key");
            }
            Thread.Sleep(10);
        }
    }

[joe-udwin-lrn]

The code sample that I gave was what was effectively happening before I made the fix to my ConcurrentDictionary code. Now my code is effectively the same as your code sample but I am still experiencing the issue on the first set 1 in 20 times when I start the console app.

In either setup should I really ever experience these errors as the code is single threaded?

[xplicit]

Do I correctly understand that you run exactly the same code sample as provided above without any additional code and sometimes you've got an error on start?

[mythz]

@Joe-U-Questionmark Can you please provide us with a small, stand-alone example of the code you're running that causes the errors you're seeing, creating a new sentinel in a loop as done in your first example is invalid, there should only ever be 1 instance of RedisSentinel started as seen in @xplicit's example.

[mythz]

If there's still an issue with valid usage of the RedisSentinel client please reply with the code we can run to repro the issue.

[joe-udwin-lrn]

I have one. Do you have an email address I can send it to?

[mythz]

If the code isn't small enough to fit here, can you publish it in a new Github repo?

[mythz]

I've received the code and configured it to run locally, but I can't reproduce the error with this code after baby sitting several times with multiple instances running. I eventually see an error because you're setting a expiry with your cache entry, i.e:

var success = cache.Set(key, valueToSet ,TimeSpan.FromMinutes(5));

So eventually after 5 mins each instance will eventually no longer be able to retrieve the value that's automatically expired. So your App starts to log errors even though it's working as intended.

Note this code hammers the Redis instance, more so when multiple apps are running so even though it's rare, the redis server is going to eventually deny client TCP connections at which point the client wont be able to connect and end up throwing the Unable to Connect error you've seen, but what happens after that point? i.e. are you able to re-connect to the redis server in subsequent calls to redis?

[joe-udwin-lrn]

So eventually after 5 mins each instance will eventually no longer be able to retrieve the value

You can change that behaviour by setting <add key="SetUpdatedValueEachLoop" value="true" />.

The redis server is going to eventually deny client TCP connections at which point the client wont be able to connect and end up throwing the Unable to Connect error you've seen

This sometimes happens as soon as we start the test harness so I don't think it can be attributed to TCP connections being exhausted.

I cannot reproduce the error on a 1 server setup either only when connecting to a 3 sentinels with 1 master and 2 slaves setup.

The first error suggests the client is unable to connect to the server, and the second error suggests the server returned an unexpected response which can happen if you're using the same client instance across multiple threads

So you can see that I am not using the same instance across multiple threads as this is a single threaded application. If issue 1 means "Connection error" can you explain issue 2 in my context?

Thanks again for your help.

[mythz]

I've added some additional logging with timestamps when the client is unable to connect in this commit and for when the Sentinel Worker receives an invalid response in this commit which will hopefully tell us what the invalid response it receives is instead.

This change is available from v4.0.61 that's now available on MyGet.

If you can post the updated error logs it will hopefully provide a better idea on what went wrong.

[joe-udwin-lrn]

Thanks for that. QA reproduced the issue 3 times:

2016-08-01 12:13:43,478 DEBUG QM.Caching.CacheCore.RedisCacheCore - Calling Set with key @#qm:qmth:testhardness-a434861f-ce0e-49e7-b8b2-10d9bf22f8f0 value 8/1/2016 12:13:43 PM and ttl
2016-08-01 12:13:43,603 ERROR QM.Caching.CacheCore.RedisCacheCore - Redis Worker error
ServiceStack.Redis.RedisException: [12:13:43.603] Unable to Connect: sPort: 0, Error: Object reference not set to an instance of an object.
   at ServiceStack.Redis.RedisNativeClient.CreateConnectionError(Exception originalEx)
   at ServiceStack.Redis.RedisNativeClient.SendReceive[T](Byte[][] cmdWithBinaryArgs, Func`1 fn, Action`1 completePipelineFn, Boolean sendWithoutRead)
   at ServiceStack.Redis.RedisNativeClient.SendExpectMultiData(Byte[][] cmdWithBinaryArgs)
   at ServiceStack.Redis.RedisNativeClient.SentinelGetMasterAddrByName(String masterName)
   at ServiceStack.Redis.RedisSentinelWorker.GetMasterHost(String masterName)

---
  2016-08-01 12:08:02,703 DEBUG QM.Caching.CacheCore.RedisCacheCore - Calling Get<T> with key @#qm:qmth:testhardness-9206a681-424b-41e3-bd52-5b2f85602c6d
2016-08-01 12:08:02,703 ERROR QM.Caching.CacheCore.RedisCacheCore - Redis Worker error
ServiceStack.Redis.RedisException: [12:08:02.703] Unable to Connect: sPort: 0, Error: Object reference not set to an instance of an object.
   at ServiceStack.Redis.RedisNativeClient.CreateConnectionError(Exception originalEx)
   at ServiceStack.Redis.RedisNativeClient.SendReceive[T](Byte[][] cmdWithBinaryArgs, Func`1 fn, Action`1 completePipelineFn, Boolean sendWithoutRead)
   at ServiceStack.Redis.RedisNativeClient.SendExpectMultiData(Byte[][] cmdWithBinaryArgs)
   at ServiceStack.Redis.RedisNativeClient.SentinelGetMasterAddrByName(String masterName)
   at ServiceStack.Redis.RedisSentinelWorker.GetMasterHost(String masterName)

 ---

   2016-08-01 12:25:50,017 DEBUG QM.Caching.CacheCore.RedisCacheCore - Calling Set with key @#qm:qmth:testhardness-e2b82d9f-8130-4eec-868c-910083c6287b value 8/1/2016 12:25:50 PM and ttl
2016-08-01 12:25:50,017 ERROR QM.Caching.CacheCore.RedisCacheCore - Redis Worker error
ServiceStack.Redis.RedisException: [12:25:50.017] Unable to Connect: sPort: 0, Error: Object reference not set to an instance of an object.
   at ServiceStack.Redis.RedisNativeClient.CreateConnectionError(Exception originalEx)
   at ServiceStack.Redis.RedisNativeClient.SendReceive[T](Byte[][] cmdWithBinaryArgs, Func`1 fn, Action`1 completePipelineFn, Boolean sendWithoutRead)
   at ServiceStack.Redis.RedisNativeClient.SendExpectDeeplyNestedMultiData(Byte[][] cmdWithBinaryArgs)
   at ServiceStack.Redis.RedisNativeClient.SentinelSlaves(String masterName)
   at ServiceStack.Redis.RedisSentinelWorker.GetSlaveHosts(String masterName)
   at ServiceStack.Redis.RedisSentinelWorker.GetSentinelInfo()
   at ServiceStack.Redis.RedisSentinel.GetSentinelInfo()
   at ServiceStack.Redis.RedisSentinel.GetRedisManager()
   at ServiceStack.Redis.RedisSentinel.GetValidSentinelWorker()