Join GitHub today
GitHub is home to over 20 million developers working together to host and review code, manage projects, and build software together.
Redis Cluster: Publish propagation optimization #122
Redis Cluster is able to propagate all the Publish operations to all the connected nodes.
This way you can PUBLISH to every node, and SUBSCRIBE to any other node. All the messages are duplicated across nodes.
However if you have a very bigger cluster, with for instance 1000 nodes, it's not good I/O wise to copy the message to all the nodes. So the idea is to implement a Bloom filter based approach to deliver messages only to interested nodes.
Nodes already use the "Cluster Bus" to propagate PUBLISH operations (our cluster bus is just a node-to-node TCP socket using a binary protocol). We can use that bus to also update the bloom filter when there is a new subscriber. From time to time the node will reset the bloom filter (to avoid pollution on unsubscribes) and will broadcast it again.
This works great for channels. For patterns it is possible to use a prefix approach, that is, we can have two bloom filters, one for channels, and one for patterns. The prefix of every pattern up to a given length (for instance 8 bytes) is used to set the bloom filter. For instance if node A has a client subscribed to "foo*" it sets the bloom filter to prefix "f", "fo", and "foo". This way it is possible to also handle the patterns case.
A simpler alternative is to set the filter to all ones if there is at least one subscriber to a pattern channel (PSUBSCRIBE).
This is just an initial idea, we'll have time to improve this in release two of Redis Cluster. Likely the first release will just have the current "brute force" support.
(as sketched during a private email exchange)
Every node takes a bloom filter of its subscribers, from time to time
Now the first technical problem is that this filters must be updated,
Cool so far, apart from the problem, that will be documented with red
Now what about PSUBSCRIBE? The easy way is to say, if there is at
For example a given node may have two psubscribes to the following two patters:
What about if we have another bloom filter (and two CRC64s in the
Now another node gets two PUBLISH messages:
The target node subscribers filter does not match, but it may have
So we check "u", "us", "use", "user", "user1", ... up to "user1:foo".
It is computationally not exactly a joke (but we are not going to use
I think the bloom filters are just one way, you can't remove items from a bloom filter. But I think your idea can be modified to work this way: a node takes an internal counter about mismatches. Once N publish packets are received about non existing channels, then the bloom filter is cleared and rebuild.
Bloom filters only give you an assurance when an key is guaranteed not to be in there. That is to say, if the bloom filter has a hit, the key might or might now be in "database". If it misses, the key is for sure not existing.
If we want to use a bloom filter to control PUBLISH propagation, then can only stop sending to a specific node+topic when the filter misses. That's the only safe combination.
So we have to start with a full bloom filter where all node+topic combinations hit, and then remove the ones that we discover not to be subscribed.
Maybe I don't remember how bloom filters work, or I missing something obvious. I'll sleep on it.
Pedro: yes that what I did not understood your previous comment. I mean: you can't remove an item from a bloom filter. So the filter will gradually get more and more polluted. Finally when a given amount of pollution is reached, that a node can detect counting messages received for which it has no receivers, it needs to reset the filter again and rebuild it form the list of channels that are actually existing.
If instead it would be possible to remove items from bloom filters a mis-received message would just cause the bloom filter update and broadcast. But this requires a full list of channels scan unfortunately since in bloom filters we can only add channels but never remove them.
Then I don't see how a bloom filter would work. Think when you start, the bloom filter is empty. When you have a message that you need to forward to the other N-1 nodes, what query do you make to the bloom filter?
Basically what I'm not seeing is what is inside the bloom filter? All combinations on Node+Topic's each Node has? If so, then a new Node needs to ask of all other nodes the current subscribed topics, all of them, to init the filter.
This comment has been minimized.
This comment has been minimized.Show comment Hide comment
If you want to use a bit more memory, wouldn't it be possible to replace the 'bits' in bloom filter with 2 byte counters. Adding to bloom filter increases the counter bucket, removing from bloom filter decreases the counter bucket. Not technical a bloom, but still almost the same.