SocketFailure and Timeout on EVAL under dotnet core

[rangp]

I'm sporadically seeing one of the following two exceptions when using redis under dotnet core (netstandard1.6)

Timeout performing EVAL, inst: 5, queue: 3, qu: 0, qs: 1, qc: 2, wr: 0, wq: 0, in: 144, ar: 0, clientName: 
RD0003FF768B5D 

SocketFailure on EVAL. Unable to read data from the transport connection: A connection attempt 
failed because the connected party did not properly respond after a period of time, or established c 
onnection failed because connected host has failed to respond

These exceptions seem to be correlated. I'm using the Redis Caching Implementation by Microsoft to access Redis, but I think the root cause is somewhere in StackExchange.Redis. Unfortunately I wasn't able to figure out any pattern regarding those exceptions. It doesn't look like my client is under high pressure and the redis server isn't either.

First I thought of it as a server problem but I'm connection against redis on azure and the classic .net clients that are running against the same server (also using stackexchange.redis) experience no connectivity issues. Only dotnet core applications seem to be affected, so I'm assuming the server isn't at fault here.

Full exception messages:

StackExchange.Redis.RedisConnectionException: SocketFailure on EVAL ---> System.AggregateException: One or more errors occurred. (Unable to read data from the transport connection: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.) ---> System.IO.IOException: Unable to read data from the transport connection: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond. ---> System.Net.Sockets.SocketException: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond
   at System.Net.Sockets.Socket.EndReceive(IAsyncResult asyncResult)
   at System.Net.Sockets.NetworkStream.EndRead(IAsyncResult asyncResult)
   --- End of inner exception stack trace ---
   at System.Net.Security.SslStreamInternal.EndRead(IAsyncResult asyncResult)
   at System.Net.Security.SslStream.<>c.<ReadAsync>b__85_1(IAsyncResult iar)
   at System.Threading.Tasks.TaskFactory`1.FromAsyncCoreLogic(IAsyncResult iar, Func`2 endFunction, Action`1 endAction, Task`1 promise, Boolean requiresSynchronization)
   --- End of inner exception stack trace ---
   at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean includeTaskCanceledExceptions)
   at System.Threading.Tasks.Task`1.GetResultCore(Boolean waitCompletionNotification)
   at System.Threading.Tasks.Task`1.get_Result()
   at StackExchange.Redis.PhysicalConnection.EndReading(Task`1 result)
   --- End of inner exception stack trace ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Extensions.Caching.Redis.RedisExtensions.<HashMemberGetAsync>d__2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Extensions.Caching.Redis.RedisCache.<GetAndRefreshAsync>d__20.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Extensions.Caching.Redis.RedisCache.<GetAsync>d__12.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

... Only External Code from here on

System.AggregateException: One or more errors occurred. (Timeout performing EVAL, inst: 6, queue: 2, qu: 0, qs: 1, qc: 1, wr: 0, wq: 0, in: 0, ar: 0, clientName: RD0003FF768B5D (Please take a look at this article for some common client-side issues that can cause timeouts: https://github.com/StackExchange/StackExchange.Redis/tree/master/Docs/Timeouts.md)) ---> System.TimeoutException: Timeout performing EVAL, inst: 6, queue: 2, qu: 0, qs: 1, qc: 1, wr: 0, wq: 0, in: 0, ar: 0, clientName: RD0003FF768B5D (Please take a look at this article for some common client-side issues that can cause timeouts: https://github.com/StackExchange/StackExchange.Redis/tree/master/Docs/Timeouts.md)
   at StackExchange.Redis.ConnectionMultiplexer.ExecuteSyncImpl[T](Message message, ResultProcessor`1 processor, ServerEndPoint server)
   at StackExchange.Redis.RedisBase.ExecuteSync[T](Message message, ResultProcessor`1 processor, ServerEndPoint server)
   at StackExchange.Redis.RedisDatabase.ScriptEvaluate(String script, RedisKey[] keys, RedisValue[] values, CommandFlags flags)
   at Microsoft.Extensions.Caching.Redis.RedisExtensions.HashMemberGet(IDatabase cache, String key, String[] members)
   at Microsoft.Extensions.Caching.Redis.RedisCache.GetAndRefresh(String key, Boolean getData)
   at Microsoft.Extensions.Caching.Redis.RedisCache.Get(String key)

..... Only External Code from here on

[BrianVallelunga]

I've seen similar issues as well, but don't have a sense of what causes them. I'm also using ASP.NET Core 2.0 with Azure-hosted Redis. I can't guarantee that this isn't just an Azure hiccup, but only one of my two sites that accesses the Redis instance had issues.

My specific error message is:

SocketFailure on HGETALLOne or more errors occurred. (Unable to transfer data on the transport connection: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.)Unable to transfer data on the transport connection: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond

In my case, restarting the application fixed the issue.

[niemyjski]

I've been unable to get a timeout like this but It sure feels like this issue as well as others (search the issues for core). Once we migrated the same code to asp.net core and deployed to azure, it took a dump real fast

[mholec]

I encountered the same issues. I have installed Redis 4.0.9 on macOS and ASP.NET Core App 2.0.0 running up .NET Core 2.0. I have tried connect to Azure Redis with the same result. I have tried to change StackExchange.Redis to various version 1.4 - 1.6 as well as changing to StackExchange.Redis.StrongName.

[NickCraver]

What timeout is everyone using here? The numbers would really help. For example: while <= 1 seconds is usually fine inside our data center, in Azure that's not really true. I'm curious what timeout levels people have set.

[mgravell]

Closing: please see #871