[ROCKETMQ-67] Consistent Hash allocate strategy #67

Jaskey · 2017-02-20T07:14:41Z

JIRA: https://issues.apache.org/jira/browse/ROCKETMQ-67

Consitent hash alogrithm to allocate message queue to prevent "thundering herd".

Core implementation:

Define a ConsitentHashRouter to hash clientId to a hash ring. And try to find the nearest clientId for every message queue.

High scalability considered, there are several new class to support as many scenarios as possible when user needs to use consitent hash

Any types of object can be routed as long as they implement Node interface.
Custom hash function can be specified
Numbers of virtual nodes can be specifed.

coveralls · 2017-02-20T07:29:37Z

Coverage increased (+0.08%) to 31.603% when pulling a56df6a on Jaskey:ROCKETMQ-67-Consistent-Hash into 573b22c on apache:master.

coveralls · 2017-02-20T07:29:37Z

Coverage increased (+0.08%) to 31.603% when pulling a56df6a on Jaskey:ROCKETMQ-67-Consistent-Hash into 573b22c on apache:master.

coveralls · 2017-02-20T07:29:37Z

Coverage increased (+0.08%) to 31.603% when pulling a56df6a on Jaskey:ROCKETMQ-67-Consistent-Hash into 573b22c on apache:master.

zhouxinyu · 2017-02-21T03:07:14Z

common/src/main/java/org/apache/rocketmq/common/consistenthash/Node.java

+package org.apache.rocketmq.common.consistenthash;
+
+/**
+ * Created by jaskeylin on 2017/2/20.


Hi, please remove the author info.

Sorry , i didn'it notice that. Just a minute

coveralls · 2017-02-21T04:04:26Z

Coverage increased (+0.2%) to 31.673% when pulling fa0d936 on Jaskey:ROCKETMQ-67-Consistent-Hash into 573b22c on apache:master.

coveralls · 2017-02-21T04:04:26Z

Coverage increased (+0.2%) to 31.673% when pulling fa0d936 on Jaskey:ROCKETMQ-67-Consistent-Hash into 573b22c on apache:master.

coveralls · 2017-02-21T04:04:26Z

Coverage increased (+0.2%) to 31.673% when pulling fa0d936 on Jaskey:ROCKETMQ-67-Consistent-Hash into 573b22c on apache:master.

coveralls · 2017-02-21T04:11:57Z

Coverage increased (+0.05%) to 31.571% when pulling d96a23f on Jaskey:ROCKETMQ-67-Consistent-Hash into 573b22c on apache:master.

coveralls · 2017-02-21T04:11:57Z

Coverage increased (+0.05%) to 31.571% when pulling d96a23f on Jaskey:ROCKETMQ-67-Consistent-Hash into 573b22c on apache:master.

coveralls · 2017-02-21T04:11:57Z

Coverage increased (+0.05%) to 31.571% when pulling d96a23f on Jaskey:ROCKETMQ-67-Consistent-Hash into 573b22c on apache:master.

coveralls · 2017-02-21T05:11:48Z

Coverage increased (+0.2%) to 31.677% when pulling 17379ca on Jaskey:ROCKETMQ-67-Consistent-Hash into 573b22c on apache:master.

coveralls · 2017-02-21T05:11:48Z

Coverage increased (+0.2%) to 31.677% when pulling 17379ca on Jaskey:ROCKETMQ-67-Consistent-Hash into 573b22c on apache:master.

coveralls · 2017-02-21T05:11:48Z

Coverage increased (+0.2%) to 31.677% when pulling 17379ca on Jaskey:ROCKETMQ-67-Consistent-Hash into 573b22c on apache:master.

Jaskey · 2017-03-09T10:01:56Z

@zhouxinyu @lizhanhui @shroman @vongosling
any advice on this pr?

Since I think consistent hash can be also applied to order message sharding, the classes in this pr can be reused. I will submit a new pr for that sharding message queue selector after this pr is merged : https://issues.apache.org/jira/browse/ROCKETMQ-136?jql=project%20%3D%20ROCKETMQ

shroman · 2017-03-10T02:16:56Z

@Jaskey Can you explain more about this feature? Having a better description on JIRA would help a lot review this PR -- especially for new functionalities. Since the description is very general, I have the following questions.

What is the advantage of introducing this strategy?
Consistent hashing could be useful if you have to move data between nodes on broker addition/removal. But since AllocateMessageQueueStrategy is used by consumers just to read from queues (if I get it right), on broker topology change a simple rehashing works well.
Of course, consistent hashing will glue consumers to the queues they are already consuming from as much as possible, even if topology is changed. Is this your concern?

Jaskey · 2017-03-10T03:23:46Z

@shroman

This is feature to give choice to users who care more about latency stabilization and messages duplication.

As you know, the default AllocateMessageQueueStrategy is averaging strategy which allocate queue to consumer as evenly as possible. But as you state, whenever queues numbers or consumer numbers changed, say a new consumer starts or an old consumer shutdowns, a rehashing will be triggered then almost all consumer suffered from this that they will rebalance to drop old queues and get new queues.

And that will cause

message latency from producer to consumer increases at the moment when consumer/queue numbers change, even when they scale up.
messages will be duplicated significantly since the offset may not be persisted to broker and that queue is assigned to another consumer to pull messages from.

This is especially significant when they have tens of consumer instances and scale-up or deployment is often.

Consistent Hash strategy to allocate queue is a good choice for these users.

shroman · 2017-03-10T03:38:05Z

@Jaskey This is a very good description, and now reviewers will understand well your intentions :) If you clearly state your intentions in JIRA, or discuss in the ml, it saves much time, and good for the record.

Sorry, it's not a code review yet, just wanted understand your intentions.

Jaskey · 2017-03-10T04:18:54Z

@shroman
The detail descriptions has been updated to JIRA. I will try described as detail as possible when I create an issue in JIRA in the future.

Jaskey · 2017-04-06T07:15:24Z

@shroman @zhouxinyu @lizhanhui @vongosling

I heard that consistent hash strategy will be accomplished in 4.1.x , while this pr has been open for more than a month, can we accelerate it, would you please help review the implementations?

dhchao11 · 2017-05-16T03:57:09Z

common/src/main/java/org/apache/rocketmq/common/consistenthash/ConsistentHashRouter.java

+            for (int i = 0; i < 4; i++) {
+                h <<= 8;
+                h |= ((int) digest[i]) & 0xFF;
+            }


digest is 128 bits， but the generated h only use 32 bits， other 96 bits are ignored, will it be better use ^ method

https://github.com/alibaba/RocketMQ/issues/270

* codeCov of protocol.mqtt and protocol.rpc for #22 * remove redundant 'TestConnectHandler' Co-authored-by: AhaThinking <ahathinking@AhaThinkingdeMacBook-Pro.local>

zhouxinyu reviewed Feb 21, 2017

View reviewed changes

ROCKETMQ-67 : Consistent Hash allocate strategy

17379ca

dhchao11 reviewed May 16, 2017

View reviewed changes

dongeforever pushed a commit to dongeforever/apache-rocketmq that referenced this pull request May 27, 2017

[ROCKETMQ-67] Consistent Hash allocate strategy closes apache#67

adae162

asfgit closed this in 787d128 Jun 6, 2017

lizhanhui pushed a commit to lizhanhui/rocketmq that referenced this pull request Jun 25, 2019

Issue apache#67 抛出连接异常时，携带对端网络地址。

677cb29

lizhanhui pushed a commit to lizhanhui/rocketmq that referenced this pull request Jun 25, 2019

Issue apache#67 连接超时异常优化

0aa3375

https://github.com/alibaba/RocketMQ/issues/270

lizhanhui pushed a commit to lizhanhui/rocketmq that referenced this pull request Jun 25, 2019

Issue apache#67 去掉Channel isWritable

f3f8a8b

lizhanhui pushed a commit to lizhanhui/rocketmq that referenced this pull request Jun 25, 2019

Issue apache#67 com.rocketmq.remoting.frameMaxLength 由8M改为16M

8942e7d

lizhanhui pushed a commit to lizhanhui/rocketmq that referenced this pull request Jun 25, 2019

Issue apache#67 发送消息失败，日志进行优化

e4a8171

Jaskey mentioned this pull request Jul 19, 2019

Support Consistent Hash with bounds allocate strategy #1333

Closed

JiaMingLiu93 pushed a commit to JiaMingLiu93/rocketmq that referenced this pull request May 28, 2020

[ROCKETMQ-67] Consistent Hash allocate strategy closes apache#67

4ae9750

JiaMingLiu93 pushed a commit to JiaMingLiu93/rocketmq that referenced this pull request May 28, 2020

[ROCKETMQ-67] Consistent Hash allocate strategy closes apache#67

9c95cf5

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[ROCKETMQ-67] Consistent Hash allocate strategy #67

[ROCKETMQ-67] Consistent Hash allocate strategy #67

Jaskey commented Feb 20, 2017 •

edited

Loading

coveralls commented Feb 20, 2017

coveralls commented Feb 20, 2017

coveralls commented Feb 20, 2017

zhouxinyu Feb 21, 2017

Jaskey Feb 21, 2017

coveralls commented Feb 21, 2017

coveralls commented Feb 21, 2017

coveralls commented Feb 21, 2017

coveralls commented Feb 21, 2017

coveralls commented Feb 21, 2017

coveralls commented Feb 21, 2017

coveralls commented Feb 21, 2017

coveralls commented Feb 21, 2017

coveralls commented Feb 21, 2017

Jaskey commented Mar 9, 2017 •

edited

Loading

shroman commented Mar 10, 2017

Jaskey commented Mar 10, 2017 •

edited

Loading

shroman commented Mar 10, 2017

Jaskey commented Mar 10, 2017

Jaskey commented Apr 6, 2017 •

edited

Loading

dhchao11 May 16, 2017

[ROCKETMQ-67] Consistent Hash allocate strategy #67

[ROCKETMQ-67] Consistent Hash allocate strategy #67

Conversation

Jaskey commented Feb 20, 2017 • edited Loading

coveralls commented Feb 20, 2017

coveralls commented Feb 20, 2017

coveralls commented Feb 20, 2017

zhouxinyu Feb 21, 2017

Choose a reason for hiding this comment

Jaskey Feb 21, 2017

Choose a reason for hiding this comment

coveralls commented Feb 21, 2017

coveralls commented Feb 21, 2017

coveralls commented Feb 21, 2017

coveralls commented Feb 21, 2017

coveralls commented Feb 21, 2017

coveralls commented Feb 21, 2017

coveralls commented Feb 21, 2017

coveralls commented Feb 21, 2017

coveralls commented Feb 21, 2017

Jaskey commented Mar 9, 2017 • edited Loading

shroman commented Mar 10, 2017

Jaskey commented Mar 10, 2017 • edited Loading

shroman commented Mar 10, 2017

Jaskey commented Mar 10, 2017

Jaskey commented Apr 6, 2017 • edited Loading

dhchao11 May 16, 2017

Choose a reason for hiding this comment

Jaskey commented Feb 20, 2017 •

edited

Loading

Jaskey commented Mar 9, 2017 •

edited

Loading

Jaskey commented Mar 10, 2017 •

edited

Loading

Jaskey commented Apr 6, 2017 •

edited

Loading