[BUG] Problems with the KeySpace if the number of keys is high.

[KhDanial]

Describe the bug
when you subscribe on huge number of keys with KeySpace, redis will block.
when i check procedure i found that in keyspace, redis will store all patterns and iterate on patterns to check client subscribed on key or not. when we subscribed on huge number of keys , the list size will be to large and iterate on list take long time.

To reproduce

enable keyspace notification and subscribed on huge number of keys and update them.

Expected behavior

redis works normaly and did not block.

[sundb]

Some of my thoughts.
In practice, when keyspace notify is used, most clients should be using the same pattern to listen, so it would be too much cpu overhead to iterate through all users' patterns.
Maybe we should add a new pubsubtype (e.g pubSubNotifyType) to send keyspace notifications by traversing the serverPubSubChannels by pattern, instead of traversing all users.
Misunderstanding, please ignore me.

[sundb]

@KhDanial Each of your clients is using a different pattern for keyspace?
Which version of redis are you using?

[KhDanial]

4.0.14

they can use same pattern, for example both of user subscribe on key "SampleKey".

imagine a situation that single client subscribe on 1e7 key then redis will be block and does not work properly,

[sundb]

@KhDanial The same patern optimization for pubsub has been implemented at dfb12f0.
But it is after version 6, maybe you can upgrade redis to a newer version.

[KhDanial]

@sundb
thanks
Was I able to clarify the problem correctly?

[sundb]

@KhDanial You mean you use a client that subscribes to a large number of patterns?
Can you tell us about your usage scenario? (or the pattern you use)

[KhDanial]

@sundb If you mean a lot of patterns, a lot of keys, yes, that's my problem.
i tested this dfb12f0 and it was not ok.

In a scenario imagine just one user subscribe on a large number of keys with KeySpace. as number of keys incresed, insert new keys will be so slow, because after each insertion redis will iterate on all keys(patterns) and it is not okay.
Did the problem clear up?

[sundb]

If you use keyspace notifications with a large number of patterns, it should be inevitable to iterate through all of them.
I'm not sure why you're using so many patterns, can you tell us what patterns you're using?

[KhDanial]

@sundb
psubscribe --keyspace@3--:key1
psubscribe --keyspace@3--:key2
psubscribe --keyspace@3--:key3
.....
psubscribe --keyspace@3--:key_100000

if you subscribe on these keys, redis will be to slow

[KhDanial]

It's important for me to understand what happened to some of the keys.
To further clarify the issue, keep in mind that two clients may make changes to a database and shared keys that both parties need to be aware of.

[sundb]

@KhDanial Using pattern matching a lot is bound to consume a lot of cpu time, just as we recommend using scan instead of keys to traverse all keys.
I'm surprised that you subscribed to 100000 patterns, why not use __keyspace@3__:key*?
If you have N clients subscribed to 100000 patterns, if one of them changes the key, it will cause the key to be matched by 100000 patterns, and then send a notification to (N-1) clients.

[KhDanial]

what does this do?
keyspace@3:key*

[sundb]

psubscribe __keyspace@3__:key*

[KhDanial]

i use your pattern, but i cant write in same format as you write :))))

[sundb]

i use your pattern, but i cant write in same format as you write :))))

I cant follow you, can't use it anymore?