Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Blocking ExtendLockLifetime might cause ThreadPool starvation #110

Open
foriequal0 opened this issue Feb 1, 2024 · 0 comments
Open

Blocking ExtendLockLifetime might cause ThreadPool starvation #110

foriequal0 opened this issue Feb 1, 2024 · 0 comments

Comments

@foriequal0
Copy link

We hold per-user RedLock for every request. It works fine under a normal load.
But the problem starts when sometimes our server response time gets slower.

The request inbound rate remains the same for our scenario.
However the response time got longer, concurrent request count went higher.
Hence ExtendLockLifetime is called more.

https://github.com/samcook/RedLock.net/blob/master/RedLockNet.SERedis/RedLock.cs#L314
ExtendLockLifetime is called by Timer, and Timer callbacks are dispatched to ThreadPool.
ThreadPool doesn't like blocking tasks, and it is crucial to have performant ThreadPool for performant async C#.
In our case, all threads in the ThreadPool are blocked by the sudden surge of ExtendLockLifetime invocations, thus it causes ASP.NET server to almost stall.

.NET tries to spawn new threads to resolve starvation, but it cannot catch up with the rate of blocked threads with the default threading config. (and they do not recommend manually adjusting it)
https://learn.microsoft.com/en-us/dotnet/core/runtime-config/threading

Even if we change the config to spin up new threads quickly, it is not performant enough with that amount of threads due to the massive number of threads and context switches.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant