Adjust concurrent consume queue order #259

rudy2steiner · 2020-06-02T12:41:15Z

在并发消费时，重试队列和普通队列应当是一样的概率
高优先级、重试队列没有消息时，继续尝试读普通队列

liyue2008 · 2020-06-03T03:35:50Z

...ue-broker-core/src/main/java/org/joyqueue/broker/consumer/SlideWindowConcurrentConsumer.java

            listIndex = partitionManager.selectPartitionIndex(partitionSize, listIndex + i, accessTimes);
            short partition = partitionList.get(listIndex);
+            if(partition==Partition.RETRY_PARTITION_ID){


重试分区与普通分区有同样的概率被选中，会不会导致重试库压力过大并且拖慢并行消费的性能？

某次请求的轮询partition列表是否包含重试分区还有另外一个参数可以控制retry.random.bound.*，也即是说拉取重试消息的概率是可控制，理论上不会拖慢并行消费。新的改动可以解决的场景：

用户开启了并行，积压消息少或为零，重试消息多的场景，手动调大重试概率，保证轮询队列列表（大概率）包含重试队列

积压消息为零，快速快速消费大量的重试消息

突然有大量消息入队，大概率会拉取积压消息（普通队列数远大于重试队列数）

原实现在这种情况下，会一直(大概率)去消费重试队列，导致消息积压；
重试的压力主要表现在写上，读重试压力可以用缓存等方式解决

Adjust consume order

357ada3

liyue2008 suggested changes Jun 3, 2020

View reviewed changes

liyue2008 approved these changes Jun 3, 2020

View reviewed changes

chengzl90 merged commit d18bc6f into jd-opensource:master Jun 10, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Adjust concurrent consume queue order #259

Adjust concurrent consume queue order #259

rudy2steiner commented Jun 2, 2020 •

edited

Loading

liyue2008 Jun 3, 2020

rudy2steiner Jun 3, 2020 •

edited

Loading

liyue2008 Jun 3, 2020

rudy2steiner Jun 3, 2020

Adjust concurrent consume queue order #259

Adjust concurrent consume queue order #259

Conversation

rudy2steiner commented Jun 2, 2020 • edited Loading

liyue2008 Jun 3, 2020

Choose a reason for hiding this comment

rudy2steiner Jun 3, 2020 • edited Loading

Choose a reason for hiding this comment

用户开启了并行，积压消息少或为零，重试消息多的场景，手动调大重试概率，保证轮询队列列表（大概率）包含重试队列

liyue2008 Jun 3, 2020

Choose a reason for hiding this comment

rudy2steiner Jun 3, 2020

Choose a reason for hiding this comment

rudy2steiner commented Jun 2, 2020 •

edited

Loading

rudy2steiner Jun 3, 2020 •

edited

Loading