Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature request: Blocking zRangeByScore to implement long-polling for scalable SQS-like queue system #5897

Open
tomekit opened this issue Mar 6, 2019 · 2 comments

Comments

@tomekit
Copy link

tomekit commented Mar 6, 2019

Is there any possibility to make the zRangeByScore blocking?

I've implemented the AWS SQS-like queue equivalent with: zRangeByScore, zAdd, zRem, setex and `get. Unfortunately my workers needs to query Redis, e.g. each 100ms to get the results as soon as possible. This causes unnecessary stress to both application and Redis. Especially that I am running tens of queues. This solution doesn't scale. I would like to reduce time to retrieve item down to ~10ms and support hundreds of queues. This would case huge stress on both Redis and my application.

The zRangeByScore is the function used to retrieve the items that needs to be processed. If I manage to make it blocking, that would solve the problem.

Is there any possibility / way to introduce blocking zRangeByScore?

This Gist might be useful for anyone that wants to have SQS-like equivalent using Redis:
https://gist.github.com/tomekit/676e93e67c0fe45d2d7f576bc0672542

EDIT:
I think this was mentioned here: #2801 however the original author closed the issue since he found a solution / implemented Laravel

@tomekit tomekit changed the title Feature request: Blocking zRangeByScore for scalable queue system Feature request: Blocking zRangeByScore to implement long-polling for scalable SQS-like queue system Mar 6, 2019
@itamarhaber
Copy link
Member

Hello @tomekit

Have you considered using BZPOPMIN instead of polling?

@ghost
Copy link

ghost commented Sep 26, 2019

I also need the functionality of a blocking ZRANGEBYSCORE.

BZPOPMIN will not work as the score is a timestamp and the current min may not have occurred yet.. unless I could add a maximum value limit to BZPOPMIN of the current time.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants