Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Multiple DC support via availability zones replication #5

Open
antirez opened this issue Dec 12, 2019 · 0 comments
Open

Multiple DC support via availability zones replication #5

antirez opened this issue Dec 12, 2019 · 0 comments

Comments

@antirez
Copy link
Owner

antirez commented Dec 12, 2019

For now Disque will consider a job replicated once there are N copies in th cluster. In case of multiple DCs in order to improve jobs safety especially if persistence is switched off, it is very important to also make sure there is at least one copy in a minimum number of AZ.

This could be implemented by having an AZ id configured in each Disque instance, up to AS 63 so that we can store then job replication info in little space using a bitmap.

ADDJOB will take an additional optional argument AZ to specify the minimum number of AZs that should have a copy. The replication acknowledge from nodes will include the AZ number so the receiving node can set the jobs AZ bits and understand if the replication was reached. If the first responders to arrive to N nodes as specified by REPLICATE where not in enough AZs, more nodes are dynamically added.

As an optimization Disque may want to remember the last AZ reported by each node to send the message to different AZs faster without using more copies than needed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant