Performance issue with 2.x

[kimglory]

We had been using 1.2.6 for quite some time. 2 months ago, we decided to upgrade to 2.x. We did some testing, then ran into the performance issue. We have a service running in a cluster of 20 machines. For testing purpose, we upgraded and deployed 5 machines under 2.x. After running it for a while, we see that CPU of these 5 machines are higher than the other 15. Rough number is 70% vs 60%, which is pretty bad. Yesterday we did it again using the latest version 2.2.79, but the result is the same.
We did some performance testing using performance profiler in VS but didn't find anything useful. After reading the release note, we noticed that one of the biggest changes for 2.x is that it uses Pipelines.Sockets.Unofficial. So this is what we suspect that caused the perofrmance issue. However, we cannot find any proof.
What is so strange is that we replicate this issue in difference services and yet didn't find anyone reporting this issue over the internet. So we need some help here.

Some spec:

1. IIS 10.x. Tried both on win server 2016 and 2019
2. 500 QPS per machine
3. 50+ redis calls per search
4. All requests are StringGet

Some code:

// Constructor
 public RedisCacheClient(CacheClientConfig config)
        {
            _config = config as RedisClientConfig;
            _redisConnection = new RedisConnection(_config);

        }

...

// Connection init. Only one database will be created for a redis instance. 
private void InitConnection(RedisConnectionString redisConnectionString, out ConnectionMultiplexer connection, out IDatabase database, out IServer server)
        {
            connection = ConnectionMultiplexer.Connect(redisConnectionString.ConnectionString);
            server = connection.GetServer(redisConnectionString.ConnectionString.Split(',')[0], DefaultPort);
            connection.ErrorMessage += Connection_ErrorMessage;
            connection.InternalError += Connection_InternalError;

            database = connection.GetDatabase(redisConnectionString.OperateDb);
        }

...

// Get string values
public async Task<List<TEntity>> GetListAsync<TEntity>(List<string> keys, CacheGetOptions<TEntity> options = null)
        {
            var redisValues = await _redisConnection.ReadDatabase.StringGetAsync(keys.Select(k => (RedisKey)FormatKey<TEntity>(k, _redisConnection.ReadKeyType)).ToArray()).ConfigureAwait(false);

            return redisValues.Select(redisValue => TryAutoConvertToValue<TEntity>(redisValue, _config.RedisValueType).Item2).Where(v => v != null).ToList();
        }

[Ohad29]

Hi,
i am also facing performance issues ( not necessary CPU ) but more time take to get add object to REDIS and timeouts since i have started working with it ...

[kimglory]

Some updates. After reading some documents, I suspect it has something to do with the threads. So I have been tweaking around the SocketManager settings. Turns out it does affect the CPU usage. The way we create the instance is, for every redis instance, we will create a connection. By looking at the source code, I noticed that it will create a dedicated threadpool with 10 workers by default for every connection. In our case, we have 10 redis instances, so it will create 100 threads by default. So I tried a couple of ways of creating our own SocketManger. Tried setting the default worker to 5. Tried using the system thread pool instead of the dedicated one. They both reduces the CPU work load. But no matter how I do, It is still higer than 1.2.6.
I really don't know what to do next.

[NickCraver]

@kimglory To clarify here: it will not create 10 per connection - it creates 10 total in a shared pool (all multiplexers default to the 1 manager in an app instance). However, this can not be the ideal for either very large or very small workloads (or very small machines). A change we're considering for .NET 6 given the sync-over-async improved thread explosion handling is to default to the general thread pool overall.

You can try it out like this:

var config = ConfigurationOptions.Parse(redisConnectionString.ConnectionString);
config.SocketManager = SocketManager.ThreadPool;
var conn = ConnectionMultiplexer.Connect(config);

or inline:

var conn = ConnectionMultiplexer.Connect(redisConnectionString.ConnectionString, options => options.SocketManager = SocketManager.ThreadPool);

[kikaragyozov]

A change we're considering for .NET 6 given the sync-over-async improved thread explosion handling is to default to the general thread pool overall.

Could you please share a link to the improvement of sync-over-async thread explosion handling?
Is it perhaps the following?
Thank you so much.

[NickCraver]

@spiritbob absolutely - that's the change, there are more details in Stephen's post here: https://devblogs.microsoft.com/dotnet/performance-improvements-in-net-6/#threading

[EvgenyGrishnov]

Facing the same performance problems: VERY high CPU usage

.

[godwish]

I used the latest version from the beginning.
I made a webapi server, but the performance was not as good as I thought.
I tried to improve the performance, but I failed.
Looking at this article, it seems to be a similar situation to me.
The cpu usage is abnormal.
I don't know what modification I made, but I hope it will be revised.

[pianoman4873]

Maybe a little digression - but I saw that in your example you wrote - var redisValues = await _redisConnection.ReadDatabase.StringGetAsync(keys.Select(k => (RedisKey)FormatKey<TEntity>(k, _redisConnection.ReadKeyType)).ToArray()).ConfigureAwait(false); Isn't there an issue with calling the MGET command when we're in the cluster mode or does the library know to separate the calls to *multiple nodes* in the cluster ? I peeked at the code and from my understanding an exception would be thrown if the keys map to more than one "slot" - @mgravell your input / confirmation would be welcome :)

…

On Fri, Jan 28, 2022 at 8:14 AM Godwish ***@***.***> wrote: I used the latest version from the beginning. I made a webapi server, but the performance was not as good as I thought. I tried to improve the performance, but I failed. Looking at this article, it seems to be a similar situation to me. The cpu usage is abnormal. I don't know what modification I made, but I hope it will be revised. — Reply to this email directly, view it on GitHub <#1901 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AF745KF2UD5T72M2SGY5KZDUYIX25ANCNFSM5G3WWEPA> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>. You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[godwish]

I was originally using my own pooling system.
The performance was not good.

I searched and changed it to the way others use it.
Every time I use the command, I get Get Database.
Among the pooled redis, GetCounts().totalOutstanding was in the form of commanding redis with less total outstanding.

There is no change in performance.

We're working hard to improve the performance,
I can't get the result I want.

[NickCraver]

Facing the same performance problems: VERY high CPU usage .

For context, a profiler is not showing you at this is high CPU usage. It's showing that from captured stack frames, these were present often. That's expected as it's the reader loop, but they are waiting, not eating CPU.

[NickCraver]

@godwish Can you provide any detail at all? Highly subjective terms like "not that good" doesn't tell us much of anything - absolute circumstances and numbers would help inform if what you're seeing is expected. There are a lot of variables in play with client, server, latency, bandwidth, CPU power at both ends, etc.

[godwish]

@NickCraver The api server I'm building now doesn't do heavy work.
I am using redis and mariadb.
I think the tps should exceed 10,000.
However, there were about 1,500 cases.
When I checked the performance with the tool, the redis call does not come out properly.
Now we are approaching the other way.
I am using dotnet5. Task.delay -> Task.yield
There was an interesting change. (Not redis.)
tps increased to 1,800 cases. (but the cpu usage remains the same)
When I stress test, the CPU usage is always close to 100%.
I'm thinking it might not be 100% redis problem.
I think I'll have to look at the async await or the web client part one by one.
I think light work is being handled heavily.
The reason I said vaguely above is because there was no change in tps.
Anyway, I'm trying to see the whole thing. This is a problem I am experiencing while making a new dotnet5, but I am using redis a lot, so I wrote it here.

I don't think my experience will help. But I left it. It was because even a small clue was needed to increase the performance.

[kimglory]

The socketmanager supports 3 modes. 1. Dedicated, 2. Shared, 3. HighPriority. What we observe is for mode 1, it will create 10 threads per instance. Only for mode 2, it will share the threads in the thread pool.
Anyway, we tried your code, but it didn't improve the performance.
I start to think maybe it is because we are still using .net framework. Maybe something is fundamentally different in .net core. And that is the reason why we are this performance issue.