How to get multiple sets by passing set key list in a single call

[rupendra-sharma]

Hi
I have stored 10k sets in Redis in a way like:
UserId1:connection1:connection2:connection3
UserId2:connection1:connection2:connection3
UserId3:connection1:connection2:connection3
.
.
UserId100000:connection1:connection2:connection3

Now I want to get selective user sets in single call by passing an redisKey array like passing User1,User100,User150,User1000
Is it possible with StackExchange.Redis? Redis provides a SSCAN command which is non-blocker to the server.

[mgravell]

SSCAN iterates a single set, and is intended to allow non-invasive access to a large set of data (by splitting what would have been a single SMEMBERS query into multiple paged queries)

What you want to do is to query multiple sets. So: before we ask "can SE.Redis do this?", we need to ask: "can redis do this?".

Now; redis does expose SUNION, but: it seems to me that asking a SUNION of 10k sets is a terrible, terrible idea, and will cripple the server while it runs.

Frankly, I wonder if you should be doing:

var keys = ...
var allContents = keys.SelectMany(key => db.SetScan(key));

this uses C# LINQ to turn an IEnumerable<> of the keys into an expanded IEnumerable<> of the flattened results - comparable to:

List<...> result = ...
foreach(var key in keys)
    foreach(var val in db.SetScan(key))
        result.Add(val);

except it isn't a List<T>, but rather: an open sequence - so it doesn't buffer everything first (which means: if you want to iterate it more than once - which includes things like .Count() - make sure you use .ToList() or similar)

Any use?

[rupendra-sharma]

@mgravell , You're right about the SSCAN command. The Redis documentation confused me about the use of the SSCAN. I have tried SUNION using redis-cli and it worked well to get 500 sets data in one call. Although the time taken was higher than my expectations(~0.56 s)

About your code suggestion:

List<...> result = ...
foreach(var key in keys)
foreach(var val in db.SetScan(key))
result.Add(val);

I feel this piece of code would hit Redis multiple times say 500 times for 500 sets.
I have the use case in which I have stored User Id and List of ConnectionIds in sets on Redis, now at the time of getting the sets back, i need to get multiple sets. The number could be between 1 to 5000. I am trying to reduce the round-trips to Redis and trying to get the desired number of sets in single call.
Any views?

[mgravell]

my thought: add an additional data structure. If you need to be able to get the data back in one go, double it all up! have the individual sets, but also have a meta-set; add to both instead of just one.

[rupendra-sharma]

I also thought of keeping the meta-data in a separate data structure but the problem is still not solved. My code can not group the user Ids at the time of adding it to Redis. But at the time of getting it back I would have a list of selected user ids that have to be fetched in a single call. And even if i have the user ids in a separate data structure on Redis, I will not be able to get these in a single call.
I am planing to use LUA script to execute SUNION using the LUA provider in the StackExchange.Redis:
I have a RedisKeys[] array which have UserIds as array items and I am trying to run this:

var script = LuaScript.Prepare("redis.call('sunion', 'user0', 'user1')");
var values= db.ScriptEvaluate(script);

But it always reruns NULL . I am new to LUA, is there anything you can suggest around this?

[mgravell]

you're missing a return. also, SE.Redis already exposes SUNION, no? probably as SetUnion? On 15 Apr 2017 7:17 p.m., "Riky" <notifications@github.com> wrote: I also thought of keeping the meta-data in a separate data structure but the problem is still not solved. My code can not group the user Ids at the time of adding it to Redis. But at the time of getting it back I would have a list of selected user ids that have to be fetched in a single call. And even if i have the user ids in a separate data structure on Redis, I will not be able to get these in a single call. I am planing to use LUA script to execute SUNION using the LUA provider in the StackExchange.Redis: I have a RedisKeys[] array which have UserIds as array items and I am trying to run this: var script = LuaScript.Prepare("redis.call('sunion', 'user0', 'user1')"); var values= db.ScriptEvaluate(script); But it always reruns NULL . I am new to LUA, is there anything you can suggest around this? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#613 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AABDsIXZbwEEAP76H6hRvgG3h4Vi7rJ2ks5rwQnFgaJpZM4M8Y2V> .

[rupendra-sharma]

Hi @mgravell

Thanks for the suggestion. i was able to write the C# equivalent using SE.redis as:

var values = _cache.SetCombine(SetOperation.Union, keyList);

where _Cache is the IDatabase instance and the KeyList is RedisKey[] array. The Values has all the SET members which can be enumerated using Foreach or LINQ.

The performance testing results are following where i have 10k Sets on Redis:
• 100 keys in KeyList: 69 MS
• 500 keys in KeyList: 150 MS
• 1000 keys in KeyList : 156 MS
• 2000 keys in KeyList: 221 MS
• 5000 keys in KeyList: 156 MS
• 10000 keys in KeyList: 215 MS
Which seems fine to me for now as the feature is a background PUB/SUB notifications. Although i have plan to have the testing again on the production.

The complete solution is on Stackexchange here:
http://stackoverflow.com/a/43433268/797473
Any thoughts?

Thanks

[mgravell]

as I said at the start, I still think that SUNION over 10k keys is a terrible idea. if you're ok with it: fine

…

On 16 Apr 2017 4:14 a.m., "Riky" ***@***.***> wrote: Hi @mgravell <https://github.com/mgravell> Thanks for the suggestion. i was able to write the C# equivalent using SE.redis as: var values = _cache.SetCombine(SetOperation.Union, keyList); where _Cache is the IDatabase instance and the KeyList is RedisKey[] array. The Values has all the SET members which can be enumerated using Foreach or LINQ. The performance testing results are following where i have 10k Sets on Redis: • 100 keys in KeyList: 69 MS • 500 keys in KeyList: 150 MS • 1000 keys in KeyList : 156 MS • 2000 keys in KeyList: 221 MS • 5000 keys in KeyList: 156 MS • 10000 keys in KeyList: 215 MS Which seems fine to me for now as the feature is a background PUB/SUB notifications. Although i have plan to have the testing again on the production. The complete solution is on Stackexchange here: http://stackoverflow.com/a/43433268/797473 Any thoughts? Thanks — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#613 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AABDsC9RoyM--x_aQOBn-DxmClHyfk_Kks5rwYeEgaJpZM4M8Y2V> .

[rupendra-sharma]

@mgravell you point is valid but I guess calling Redis for 500 times or more for one feature is also not a good idea. Anyways I would like to test it on my live environment after putting the real load than decide if I should keep this code or move to another approach like moving this data to database using Redis PUB/SUB and querying the data there.
thanks for the help offered by you.

[NickCraver]

Closing this out to cleanup.