Prefilter polled queues

[neob91-close]

This is a PoC for reducing the Redis load caused by each worker frequently polling the entire set of subqueues.

SSCAN lets us prefilter the queues we pull out of Redis, and seems pretty fast too:

production In [49]: r.scard("t:scheduled")
production Out[49]: 3919

production In [50]: timeit.timeit(lambda: r.smembers("t:scheduled"), number=5)
production Out[50]: 0.06490249998751096

production In [51]: timeit.timeit(lambda: r.sscan("t:scheduled", match="zoom.*", count=10000), number=5)
production Out[51]: 0.01752800800022669

[vtclose]

A couple nitpicks but I'm not very knowledgeable with TT internals to comment on correctness.

[thomasst]

Looks good except the comment.

[thomasst]

The relevant characters to be escaped are *, ?, [, and \ (https://github.com/redis/redis/blob/155acef51ac7826ed294a4b61d891f1c7a9a40ac/src/util.c#L58). You additionally have ] (which is fine but irrelevant), but you should also escape \, so this should be re.compile(r"([?*\[\]\\])") (or re.compile(r"([?*\[\\])")).

[neob91-close]

I don't know how I overlooked the \ character, thanks!

I am aware that it is not necessary to escape ], but to my knowledge Redis docs do not mention that fact and I prefer to avoid depending on undocumented behaviour.

[thomasst]

Which docs btw? I was trying to find Redis docs on globs but couldn't find any, so then I just looked at the source code.

[vtclose]

This has some examples https://redis.io/commands/keys/ but doesn't seem like a formal spec