Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Timeout in Redis #1150

Closed
swapnashinde opened this issue May 21, 2019 · 47 comments
Closed

Timeout in Redis #1150

swapnashinde opened this issue May 21, 2019 · 47 comments
Labels
⏱️ timeout

Comments

@swapnashinde
Copy link

We are getting below errors after updating the StackExchange version to 2.0.601.
Could you please check.

Exception=StackExchange.Redis.RedisTimeoutException: Timeout performing GET (10000ms), next: PING, inst: 0, qu: 0, qs: 3, aw: False, rs: ReadAsync, ws: Idle, in: 0, in-pipe: 0, out-pipe: 0, serverEndpoint: Unspecified/:6379, mgr: 10 of 10 available, clientName: , IOCP: (Busy=0,Free=1000,Min=2,Max=1000), WORKER: (Busy=1,Free=2046,Min=2,Max=2047), v: 2.0.601.3402 (Please take a look at this article for some common client-side issues that can cause timeouts: https://stackexchange.github.io/StackExchange.Redis/Timeouts)
at StackExchange.Redis.ConnectionMultiplexer.ExecuteSyncImpl[T](Message message, ResultProcessor1 processor, ServerEndPoint server) at StackExchange.Redis.RedisBase.ExecuteSync[T](Message message, ResultProcessor1 processor, ServerEndPoint server)
at StackExchange.Redis.RedisDatabase.StringGet(RedisKey key, CommandFlags flags)
@swapnashinde
Copy link
Author

Could you please help as we are getting this error in production.

@swapnashinde
Copy link
Author

We are also getting below error
StackExchange.Redis.RedisConnectionException: No connection is available to service this operation: SETEX DBStreamingStatus; IOCP: (Busy=0,Free=1000,Min=4,Max=1000), WORKER: (Busy=0,Free=2047,Min=4,Max=2047), Local-CPU: n/a
at StackExchange.Redis.ConnectionMultiplexer.ExecuteSyncImpl[T](Message message, ResultProcessor1 processor, ServerEndPoint server) in C:\projects\stackexchange-redis\src\StackExchange.Redis\ConnectionMultiplexer.cs:line 2237 at StackExchange.Redis.RedisBase.ExecuteSync[T](Message message, ResultProcessor1 processor, ServerEndPoint server) in C:\projects\stackexchange-redis\src\StackExchange.Redis\RedisBase.cs:line 54
at StackExchange.Redis.RedisDatabase.StringSet(RedisKey key, RedisValue value, Nullable`1 expiry, When when, CommandFlags flags) in C:\projects\stackexchange-redis\src\StackExchange.Redis\RedisDatabase.cs:line 2407

@drakefang
Copy link

same problem.
but If we deploy server on windows , it's ok;
if we deploy on docker, there're lots of timeout exception.

Timeout performing EXISTS (5000ms), next: EXISTS test:insert:herokillrecord, inst: 5, qu: 0, qs: 23, aw: False, rs: ReadAsync, ws: Idle, in: 3885, in-pipe: 0, out-pipe: 45, serverEndpoint: 192.168.1.161:6380, mgr: 9 of 10 available, clientName: 5f42773d6dc1, IOCP: (Busy=0,Free=1000,Min=4,Max=1000), WORKER: (Busy=20,Free=32747,Min=4,Max=32767), v: 2.0.601.3402 (Please take a look at this article for some common client-side issues that can cause timeouts: https://stackexchange.github.io/StackExchange.Redis/Timeouts)

@sampsonye
Copy link

we get the same issue,wait for help！

@oshvartz
Copy link

Hi
we have the same problem - on high rate of Reads .net 4.6.2 with one master and 2 read replicas
StackExchange.Redis.RedisTimeoutException: Timeout performing GET (5000ms), next: GET 4CE01B05F4970DAC31C6DC06CA4D75, inst: 4, qu: 0, qs: 161, aw: False, rs: ReadAsync, ws: Idle, in: 61120, in-pipe: 168, out-pipe: 934, serverEndpoint: XXXXX, mgr: 10 of 10 available, clientName: XXXXX, IOCP: (Busy=85,Free=915,Min=8,Max=1000), WORKER: (Busy=100,Free=32667,Min=8,Max=32767), v: 2.0.601.3402 (Please take a look at this article for some common client-side issues that can cause timeouts: https://stackexchange.github.io/StackExchange.Redis/Timeouts)

@opendomain
Copy link

We are also having similar issues.
We have tried to change the size of the cache, moving to smaller object stored in the cache, asnyc/await, thread blocks, and serializing to JSON - nothing works.

@pengweiqhca
Copy link
Contributor

Hi
we have the same problem - on high rate of Reads .net 4.6.2 with one master and 2 read replicas
StackExchange.Redis.RedisTimeoutException: Timeout performing GET (5000ms), next: GET 4CE01B05F4970DAC31C6DC06CA4D75, inst: 4, qu: 0, qs: 161, aw: False, rs: ReadAsync, ws: Idle, in: 61120, in-pipe: 168, out-pipe: 934, serverEndpoint: XXXXX, mgr: 10 of 10 available, clientName: XXXXX, IOCP: (Busy=85,Free=915,Min=8,Max=1000), WORKER: (Busy=100,Free=32667,Min=8,Max=32767), v: 2.0.601.3402 (Please take a look at this article for some common client-side issues that can cause timeouts: https://stackexchange.github.io/StackExchange.Redis/Timeouts)

85 > 8 and 100 > 8

@pengweiqhca
Copy link
Contributor

same problem.
but If we deploy server on windows , it's ok;
if we deploy on docker, there're lots of timeout exception.

Timeout performing EXISTS (5000ms), next: EXISTS test:insert:herokillrecord, inst: 5, qu: 0, qs: 23, aw: False, rs: ReadAsync, ws: Idle, in: 3885, in-pipe: 0, out-pipe: 45, serverEndpoint: 192.168.1.161:6380, mgr: 9 of 10 available, clientName: 5f42773d6dc1, IOCP: (Busy=0,Free=1000,Min=4,Max=1000), WORKER: (Busy=20,Free=32747,Min=4,Max=32767), v: 2.0.601.3402 (Please take a look at this article for some common client-side issues that can cause timeouts: https://stackexchange.github.io/StackExchange.Redis/Timeouts)

20 > 4

@philiahe
Copy link

philiahe commented Jun 1, 2019

same problem

@MatanShabtay
Copy link

MatanShabtay commented Jun 4, 2019 •
edited

Has anyone try downgrading back to 1.2.6 ?
In all this time I only heard that version 2.x.xis far more stable than 1.x.x but it seems to be the opposite.

@Syncworx
Copy link

Syncworx commented Jun 8, 2019

I was planning to migrate to the latest version. Should I stop it? I was getting CPU Spikes using version 1.2.6 when we have high traffic and the reason was related to Socket Manger.WriteToAllQueue. We are dead in the water now?

@liu-yfeng
Copy link

same problem.
but If we deploy server on windows , it's ok;
if we deploy on docker, there're lots of timeout exception.

Timeout performing EXISTS (5000ms), next: EXISTS test:insert:herokillrecord, inst: 5, qu: 0, qs: 23, aw: False, rs: ReadAsync, ws: Idle, in: 3885, in-pipe: 0, out-pipe: 45, serverEndpoint: 192.168.1.161:6380, mgr: 9 of 10 available, clientName: 5f42773d6dc1, IOCP: (Busy=0,Free=1000,Min=4,Max=1000), WORKER: (Busy=20,Free=32747,Min=4,Max=32767), v: 2.0.601.3402 (Please take a look at this article for some common client-side issues that can cause timeouts: https://stackexchange.github.io/StackExchange.Redis/Timeouts)

in 'xxx.runtimeconfig.json' file add
"System.Threading.ThreadPool.MinThreads": 100,
"System.Threading.ThreadPool.MaxThreads": 1000
like this:
{
"runtimeOptions": {
"tfm": "netcoreapp2.2",
"framework": {
"name": "Microsoft.AspNetCore.App",
"version": "2.2.0"
},
"configProperties": {
"System.GC.Server": true,
"System.Threading.ThreadPool.MinThreads": 100,
"System.Threading.ThreadPool.MaxThreads": 1000
}
}
}

@witskeeper
Copy link

witskeeper commented Jun 15, 2019 •
edited

i think StackExchange.Redis need warmup before handle high rate request.

i tested 2.x(2.0.601 & 2.0.513) in docker using jmeter

FROM microsoft/dotnet:2.2-aspnetcore-runtime AS base
WORKDIR /app
EXPOSE 80

FROM microsoft/dotnet:2.2-sdk AS build
WORKDIR /src
COPY ["RedisTest/RedisTest.csproj", "RedisTest/"]
RUN dotnet restore "RedisTest/RedisTest.csproj"
COPY . .
WORKDIR "/src/RedisTest"
RUN dotnet build "RedisTest.csproj" -c Release -o /app

FROM build AS publish
RUN dotnet publish "RedisTest.csproj" -c Release -o /app

FROM base AS final
WORKDIR /app
COPY --from=publish /app .
ENTRYPOINT ["dotnet", "RedisTest.dll"]

Test Code

[HttpPost]
public ActionResult<string> Post([FromServices]IDatabase db)
{
      var key = Guid.NewGuid().ToString();
      db.StringSet(key, key);
      return db.StringGet(key).ToString();
}

Setting of Test

number of thread : 20
Ramp-up Period : 1

if i start test just after i start docker container
i will get timeout error.

if i stop test and restart test.
timeout error never show up.

if i restart docker container and set Ramp-up Period to 100
timeout never show up.

if i stop test , set Ramp-up Period to 1 and wait for 20 min, then restart test.
i got timeout error.

so ,i think StackExchange.Redis need warmup before handle high rate request.
and it will cooling down if no request came in.

@Vamp282
Copy link

Vamp282 commented Jun 28, 2019

Same problem.
version 2.0.601

@marafiq
Copy link
Contributor

marafiq commented Jul 21, 2019

What is the size of data, you are all trying to store? I have reduced my time outs by compressing the data, i store on Redis because your network pipe speed also matters.
Always set minIoThreads to at least 50 per processor.

@Vamp282
Copy link

Vamp282 commented Jul 22, 2019

Fixed. Increased for the min size of minWorker and minIOC.
https://gist.github.com/JonCole/e65411214030f0d823cb

@milovidov983
Copy link

milovidov983 commented Aug 26, 2019 •
edited

Hi
we have the same problem - on high rate of Reads .net 4.6.2 with one master and 2 read replicas
StackExchange.Redis.RedisTimeoutException: Timeout performing GET (5000ms), next: GET 4CE01B05F4970DAC31C6DC06CA4D75, inst: 4, qu: 0, qs: 161, aw: False, rs: ReadAsync, ws: Idle, in: 61120, in-pipe: 168, out-pipe: 934, serverEndpoint: XXXXX, mgr: 10 of 10 available, clientName: XXXXX, IOCP: (Busy=85,Free=915,Min=8,Max=1000), WORKER: (Busy=100,Free=32667,Min=8,Max=32767), v: 2.0.601.3402 (Please take a look at this article for some common client-side issues that can cause timeouts: https://stackexchange.github.io/StackExchange.Redis/Timeouts)

85 > 8 and 100 > 8

Sorry, I'm have same problem. My case:
IOCP: (Busy=0,Free=1000,Min=2,Max=1000), WORKER: (Busy=7,Free=32760,Min=2,Max=32767)
As you can see, the number of the busy workers is greater than the minimum value 7 > 2. Could this be the cause of the error? How does this affect the library? And what should I do to fix it?

Full exception:

! StackExchange.Redis.RedisTimeoutException: Timeout awaiting response (outbound=0KiB, inbound=0KiB, 5000ms elapsed, timeout is 5000ms), command=GET, next: GET CRM.UsersMainService.GetUserInfoAsync.ais1@google.com, inst: 0, qu: 0, qs: 4, aw: False, rs: ReadAsync, ws: Idle, in: 1168, in-pipe: 0, out-pipe: 0, serverEndpoint: 5.178.85.30:7001, mgr: 10 of 10 available, clientName: f0b7b81f5ce5, PerfCounterHelperkeyHashSlot: 16089, IOCP: (Busy=0,Free=1000,Min=2,Max=1000), WORKER: (Busy=7,Free=32760,Min=2,Max=32767), v: 2.0.601.3402 (Please take a look at this article for some common client-side issues that can cause timeouts: https://stackexchange.github.io/StackExchange.Redis/Timeouts) (Most recent call last)

@pengweiqhca
Copy link
Contributor

As you can see, the number of the busy workers is greater than the minimum value 7 > 2. Could this be the cause of the error? How does this affect the library? And what should I do to fix it?

ThreadPool.SetMinThreads(int workerThreads, int completionPortThreads); //100 or more

@milovidov983
Copy link

As you can see, the number of the busy workers is greater than the minimum value 7 > 2. Could this be the cause of the error? How does this affect the library? And what should I do to fix it?

ThreadPool.SetMinThreads(int workerThreads, int completionPortThreads); //100 or more

I know about this, but I worry that Microsoft itself does not recommend setting arbitrary values....

@mgravell
Copy link
Collaborator

mgravell commented Aug 28, 2019 via email

@milovidov983
Copy link

Synopsis: "it is chewing through an inbound stream with 161 replies needed and 61k of data to look at; the next reply expected is the reply to the GET shown" Now; this doesn't tell us everything, unfortunately; it could be that this "GET" is a huge oversized blob that is causing the stall, or it could be that something earlier caused a stall, and it is just trying to catch up when it ran out of time. But: take a look at what that key holds to see whether there's anything alarming. You should also look at the server-side logs to see if the server is taking a long time to process anything; see "SLOWLOG GET". Ignore the "busy workers" - that shouldn't matter here.
…
On Wed, 28 Aug 2019 at 19:18, Bullock @.***> wrote: As you can see, the number of the busy workers is greater than the minimum value 7 > 2. Could this be the cause of the error? How does this affect the library? And what should I do to fix it? ThreadPool.SetMinThreads(int workerThreads, int completionPortThreads); //100 or more I know about this, but I worry that Microsoft itself does not recommend setting arbitrary values.... — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#1150?email_source=notifications&email_token=AAAEHME2BKSHPS5I4DOOS33QG26OLA5CNFSM4HOLXHB2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD5MAQ2A#issuecomment-525863016>, or mute the thread https://github.com/notifications/unsubscribe-auth/AAAEHMHLX6XS2VGZJ56EKTDQG26OLANCNFSM4HOLXHBQ .
-- Regards, Marc

Thanks for your reply. If I understand you correctly, the problem may be the size of the saved object? But all the objects that we save have a maximum size of 100 kilobytes

@mgravell
Copy link
Collaborator

mgravell commented Aug 28, 2019 via email

@placidseven
Copy link

should I do to f

Did you ever get a response? I'm having timeout issues and seeing similar stats:

"WORKER: (Busy=5,Free=8186,Min=2,Max=8191"

I have a feeling this is being caused by queries the developers wrote are returning too much data, but I don't know how to prove it yet.

@timavaza
Copy link

timavaza commented Oct 22, 2019 via email

@xiaoMaPrincess
Copy link

It is said that using all asynchronous methods can solve this problem.
We are ready to change csredis.
https://github.com/2881099/csredis

@NickCraver
Copy link
Collaborator

We're exploring a theory on a now-known cause and could use your help. For anyone experiencing issues on a .NET Framework application - are you in either of the following cases?

.NET Framework Web Projects

In web.config, either explicitly setting

<add key="aspnet:UseTaskFriendlySynchronizationContext" value="false" />

or, something less than 4.5 in <httpRuntime> (also web.config):

<httpRuntime targetFramework="4.x" />

.NET Framework Non-web Projects

In app.config, something less than 4.5 in <supportedRuntime>:

<configuration>
    <startup> 
        <supportedRuntime version="v4.x" sku=".NETFramework,Version=v4.x"/>
    </startup>
</configuration>

To clarify: it doesn't matter what your .csproj (or other) target framework is for this case. These .config flags define runtime quirk behavior and alter how threads are handled - resulting in some of our should-be-dedicated bits being stolen and our queue indefinitely hung in a bad state.

If this matches anyone here, it'd be hugely helpful to know. @mgravell is working on a workaround for this scenario.

@chrisdunne
Copy link

@NickCraver

Having the same intermittent timeout issues, using .NET Framework web project, however UseTaskFriendlySynchronizationContext is not explicitly set and the httpruntime isn't 4.x

@chrisdunne
Copy link

As mentioned by @drakefang I'm using Docker, that's the only commonality I can see in this thread.

@longjxchina
Copy link

asp.net core 3.1 on ubuntu 16.0.4:
if SetMinThreas(1000,1000), is ok. but if more than 1000, timeout happed again.

test script:
ab -n 10000 -c 1000 http://192.168.0.22:88/

@pengweiqhca
Copy link
Contributor

asp.net core 3.1 on ubuntu 16.0.4:
if SetMinThreas(1000,1000), is ok. but if more than 1000, timeout happed again.

test script:
ab -n 10000 -c 1000 http://192.168.0.22:88/

https://stackexchange.github.io/StackExchange.Redis/ThreadTheft

@longjxchina
Copy link

longjxchina commented Apr 22, 2020 •
edited

asp.net core 3.1 on ubuntu 16.0.4:
if SetMinThreas(1000,1000), is ok. but if more than 1000, timeout happed again.
test script:
ab -n 10000 -c 1000 http://192.168.0.22:88/

https://stackexchange.github.io/StackExchange.Redis/ThreadTheft

i think isn’t the reason.
that's my code:

env value:1100

private static void SetThreads()
{
    int miniThreads = Convert.ToInt32(Environment.GetEnvironmentVariable("MIN_THREADS") ?? "1000");
    ThreadPool.SetMinThreads(miniThreads, miniThreads);
    ThreadPool.GetMinThreads(out int workerThreads, out int completionPortThreads);
    
    Console.WriteLine($"env set threads:{miniThreads}");
    Console.WriteLine($"worker threads:{workerThreads}");
    Console.WriteLine($"completionPort threads:{completionPortThreads}");
}

result: (only 1 cpu)

env set threads:1100
worker threads:1
completionPort threads:1

@longjxchina
Copy link

asp.net core 3.1 on ubuntu 16.0.4:
if SetMinThreas(1000,1000), is ok. but if more than 1000, timeout happed again.
test script:
ab -n 10000 -c 1000 http://192.168.0.22:88/

https://stackexchange.github.io/StackExchange.Redis/ThreadTheft

i think isn’t the reason.
that's my code:

env value:1100

private static void SetThreads()
{
    int miniThreads = Convert.ToInt32(Environment.GetEnvironmentVariable("MIN_THREADS") ?? "1000");
    ThreadPool.SetMinThreads(miniThreads, miniThreads);
    ThreadPool.GetMinThreads(out int workerThreads, out int completionPortThreads);
    
    Console.WriteLine($"env set threads:{miniThreads}");
    Console.WriteLine($"worker threads:{workerThreads}");
    Console.WriteLine($"completionPort threads:{completionPortThreads}");
}

result: (only 1 cpu)

env set threads:1100
worker threads:1
completionPort threads:1

i had found the answer: dotnet/aspnetcore#17090 (comment)

@Greatsamps
Copy link

Just to chime in here.

Same issue as above, also running in docker. I this instance the client and redis server are on same physical machine! physical machine cpu circa 4%

StackExchange.Redis.RedisTimeoutException: Timeout performing RPOP (5000ms), next: RPOP TradeReplicatorMessage, inst: 0, qu: 0, qs: 1, aw: False, rs: ReadAsync, ws: Idle, in: 0, in-pipe: 0, out-pipe: 0, serverEndpoint: 172.16.6.20:6379, mc: 1/1/0, mgr: 10 of 10 available, clientName: ln-tcmp2, IOCP: (Busy=0,Free=1000,Min=24,Max=1000), WORKER: (Busy=6,Free=32761,Min=24,Max=32767), v: 2.1.30.38891 (Please take a look at this article for some common client-side issues that can cause timeouts: https://stackexchange.github.io/StackExchange.Redis/Timeouts)
at StackExchange.Redis.ConnectionMultiplexer.ExecuteSyncImpl[T](Message message, ResultProcessor1 processor, ServerEndPoint server) in /_/src/StackExchange.Redis/ConnectionMultiplexer.cs:line 2624 at StackExchange.Redis.RedisBase.ExecuteSync[T](Message message, ResultProcessor1 processor, ServerEndPoint server) in //src/StackExchange.Redis/RedisBase.cs:line 54
at StackExchange.Redis.RedisDatabase.ListRightPop(RedisKey key, CommandFlags flags) in //src/StackExchange.Redis/RedisDatabase.cs:line 998
at Inovine.RedisMessagingBus.Queuer.<>c__DisplayClass13_0.b__1(ChannelMessage subMessage)

@EminemJK EminemJK mentioned this issue Jan 20, 2021
Closed
@Xiaobaicoder-pixel
Copy link

One or more errors occurred. (Timeout performing HGETALL (5000ms), inst: 0, qu: 0, qs: 4, aw: False, rs: ComputeResult, ws: Idle, in: 0, in-pipe: 127930169, out-pipe: 0, serverEndpoint: 192.168.1.61:6379, mgr: 8 of 10 available, clientName: WIN-2NDF9JN0529, IOCP: (Busy=1,Free=999,Min=4,Max=1000), WORKER: (Busy=1,Free=32766,Min=4,Max=32767), v: 2.0.600.65315 (Please take a look at this article for some common client-side issues that can cause timeouts: https://stackexchange.github.io/StackExchange.Redis/Timeouts))
I Use .Net Core 3.1 ,also meet this problem!!!

@RanderGabriel
Copy link

Any updates on this? We are experiencing the same error in production, using version 2.0.600.65315

@NickCraver
Copy link
Collaborator

For anyone hitting this, please try the latest release as we added more debugging information to the exceptions in 2.2.x - we're working more on stability with the cloud folks now (for scenarios we don't generally see, but do want to solve).

@Xiaobaicoder-pixel For your case, it looks like you're pulling back a few keys totalling 128MB, that's quite a bit of data and depending on your bandwidth, a higher timeout is likely needed.

@RanderGabriel please try the latest 2.2.x release!

@NickCraver
Copy link
Collaborator

Update on the above: the stability improvements (especially on connecting and re-connecting) that we've been working on are now on NuGet, in version 2.2.50.

@mataness
Copy link

Update on the above: the stability improvements (especially on connecting and re-connecting) that we've been working on are now on NuGet, in version 2.2.50.

Thanks for the update Nick.
For learning purposes, can you please share what was done to improve the stability?
And do you guys need help? I will be glad to contribute

@NickCraver
Copy link
Collaborator

@mataness You can see the PR numbers in the release notes. The basics are we found some races in connections and reconnections under certain circumstances and worked to generally fix the ordering problem. I'm not trying to be vague about circumstances, they're just really numerous...it's the speed and race by latency, thread pool, etc. that could cause it - there was no specific setup to use as an example. It was (by nature) more likely in higher latency environments in general, though.

@jineshpatel99
Copy link

Anyone tested with the latest version if this still issue or not we are having the same issue.

Exception : StackExchange.Redis.RedisTimeoutException: Timeout performing EXISTS (5000ms), next: EXISTS e85ef8ed-f047-48dd-a7dc-9003f9be07farelease-l-beta.ancileuperform.comAccountCreateden-US, inst: 1, qu: 0, qs: 7, aw: False, rs: ReadAsync, ws: Idle, in: 231, serverEndpoint: master.elr1ig6kxl5p1gvi.lm2pze.use1.cache.amazonaws.com:6379, mc: 1/1/0, mgr: 10 of 10 available, clientName: 169, IOCP: (Busy=0,Free=1000,Min=2,Max=1000), WORKER: (Busy=7,Free=32760,Min=2,Max=32767), v: 2.1.30.38891 (Please take a look at this article for some common client-side issues that can cause timeouts: https://stackexchange.github.io/StackExchange.Redis/Timeouts)

@mgravell
Copy link
Collaborator

mgravell commented Jul 1, 2021 via email

@jineshpatel99
Copy link

Okay, We updated our readis size to t2.Medium so we have more resources to respond to the requested this happens when it's running code in the Parallels and trying to fetch data from the cache.

list.Users.ForEach(u => sendTasks.Add(Task.Run(() => processUser(not, u))));

Waiting for tasks after that.
So when it goes to the processUser it tries to get some values from the cache and at that time I saw this error.

Not sure if there is anything wrong I am doing here. I will try to update and see if that fixes it or not

@mgravell
Copy link
Collaborator

mgravell commented Jul 2, 2021

That code doesn't show a single library usage, so: I can only speculate, but: it looks like you're spawning a bunch of tasks (presumably quite a big number), each of which is appears to us the synchronous API surface. That would be a good way to block your own thread-pool and hammer a server. Since redis server is a beast, I'd expect the thread-pool to lock up first.

Suggestions:

  • if you're going to use tasks, use tasks; there is a fully async API surface available for you usage that won't tie up threads
  • consider Parallel.ForEach, which may help restrict parallelism to reasonable rather than unbounded levels
  • there's also the batch API which may have utility here, hard to say without more context

But: we're not a psychic debugging service; we can't comment much on code we can't see.

@RyanLiu99
Copy link

Curious what does "mc: 1/1/0" mean? It is not included in https://github.com/StackExchange/StackExchange.Redis/blob/main/docs/Timeouts.md.

@Donnotron666
Copy link

Any update on this issue? I'm seeing this on my aspnetcore service running on azure. It seems to correspond to load, but the actual worker/thread counts on our redis and web instances show them as having a ton of unused capacity. The error we're getting is the same as the ones mentioned, and we're on StackExchange 2.2.50
StackExchange.Redis.RedisTimeoutException: at StackExchange.Redis.ConnectionMultiplexer.ExecuteSyncImpl (StackExchange.Redis, Version=2.0.0.0, Culture=neutral, PublicKeyToken=c219ff1ca8c2ce46: /_/src/StackExchange.Redis/ConnectionMultiplexer.cs:2856) at StackExchange.Redis.RedisBase.ExecuteSync (StackExchange.Redis, Version=2.0.0.0, Culture=neutral, PublicKeyToken=c219ff1ca8c2ce46: /_/src/StackExchange.Redis/RedisBase.cs:54)

@mataness
Copy link

mataness commented Nov 3, 2021

2 years ago I wrote a NuGet that adds connection pool abstraction on top of the Redis connection multiplexer.
Using this business logic together with some few tweaks such as reducing the cached item size reduced our timeouts to the minimum.
I hope it will help you guys too.
https://www.nuget.org/packages/StackExchange.Redis.MultiplexerPool/1.0.2

@NickCraver
Copy link
Collaborator

Since a lot of these timeouts are environment specific and we do our best to help, issues that are a conglomerate of many and a myriad of different causes unfortunately turn into a bit of a support mess over time. I'm going to close this one out as I think the people having issues here are resolved, but people experiencing timeouts today are much better off opening a new issue (if the timeouts documentation doesn't help already) with their specific error message (which we've improved many times over the releases) allowing us to help with advice much better.

If anyone here is still having an issue on latest, please respond and let me know with a current error message!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
⏱️ timeout
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests