Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Replace node_redis with ioredis #16

Closed
thelinuxlich opened this issue Sep 24, 2015 · 7 comments
Closed

Replace node_redis with ioredis #16

thelinuxlich opened this issue Sep 24, 2015 · 7 comments
Assignees
Milestone

Comments

@thelinuxlich
Copy link

For redis 3 cluster support

@LewisJEllis
Copy link
Member

I've had my eye on ioredis and will give this some consideration. When I started on this library it hadn't been released yet, but now it's looking quite nice. I'll get familiar with their API and give it a try sometime soon.

@thelinuxlich
Copy link
Author

I forked bee-queue to add ioredis and only two tests are failing(reconnection)

@LewisJEllis
Copy link
Member

Cool! Off the top of my head, I would expect those two tests to fail among all others; they're pretty specific to how node_redis deals with connections. I made an ioredis branch; feel free to open a PR to it and we can work from there.

@thelinuxlich
Copy link
Author

Actually there were a lot of tests failing and now there are only two: Queue "after each" hook and Queue Health Check reports a failed job

My fork thelinuxlich@795a465

@LewisJEllis
Copy link
Member

Moving forward with this in #21. I just want to leave a note here on cluster support in general;

Because of the nature of Redis and its lists and (mostly) its Lua scripting, bee-queue won't exactly be able to offer true cluster support. It will work on Redis cluster deployments, but it will really just run everything on one Redis instance. Otherwise, the Lua scripts won't be able to access all the keys they need to in one place.

Fortunately Redis cluster has a very nice way to do this:

Redis Cluster supports multiple key operations as long as all the keys involved into a single command execution (or whole transaction, or Lua script execution) all belong to the same hash slot. The user can force multiple keys to be part of the same hash slot by using a concept called hash tags.

Hash tags are documented in the Redis Cluster specification, but the gist is that if there is a substring between {} brackets in a key, only what is inside the string is hashed, so for example this{foo}key and another{foo}key are guaranteed to be in the same hash slot, and can be used together in a command with multiple keys as arguments.

but it should be clear that any one instance of bee-queue will still be running entirely on one instance of Redis. Of course, it will be possible to run multiple instances of bee-queue on multiple instances of Redis.

@hustcer
Copy link

hustcer commented Aug 24, 2016

+1
need a update on this, asap. Thanks.

@skeggse
Copy link
Member

skeggse commented Aug 9, 2017

From #21

What's the latest on ioredis vs redis? It sounds like the two are on a long-term collision course, and that performance is comparable, but I'm not certain. I don't have a strong opinion one way or the other, but this PR would definitely need to be updated for 1.0.0. Feel free to reopen if you think this is still worthwhile.

@skeggse skeggse closed this as completed Aug 9, 2017
@skeggse skeggse reopened this Aug 9, 2017
skeggse pushed a commit that referenced this issue Aug 11, 2017
This allows client sharing between multiple Queue instances, or other uses.

It also begins to add support for ioredis (cc #16 and #21). We might consider supporting both, though there are some major inconsistencies between the two libraries, such as the multi-exec callback form.
skeggse pushed a commit that referenced this issue Aug 11, 2017
This allows client sharing between multiple Queue instances, or other uses.

It also begins to add support for ioredis (cc #16 and #21). We might consider supporting both, though there are some major inconsistencies between the two libraries, such as the multi-exec callback form.
skeggse pushed a commit that referenced this issue Aug 11, 2017
This allows client sharing between multiple Queue instances, or other uses.

It also begins to add support for ioredis (cc #16 and #21). We might consider supporting both, though there are some major inconsistencies between the two libraries, such as the multi-exec callback form.
@skeggse skeggse added this to the 2.0.0 milestone Apr 16, 2020
This was referenced Apr 16, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

5 participants