RMQ 4.6.0 [TIMEOUT_CLEAN_QUEUE]

[kayyellow]

1. Hardware configuration
   8C - 32G - aliyun 高效云盘 1T

2. The configuration for broker：
   `brokerClusterName=DefaultCluster
   brokerName=broker-b
   brokerId=0
   deleteWhen=04
   fileReservedTime=48
   brokerRole=SYNC_MASTER
   flushDiskType=ASYNC_FLUSH
   #发送消息的最大线程数
   sendMessageThreadPoolNums: 24
   #发送消息是否使用可重入锁
   useReentrantLockWhenPutMessage: true
   waitTimeMillsInSendQueue=1000
   mapedFileSizeCommitLog=1073741824
   storePathRootDir=/usr/local/rocketmq/store
   storePathCommitLog=/usr/local/rocketmq/store/commitlog
   storePathConsumerQueue=/usr/local/rocketmq/store/consumequeue
   storePathIndex=/usr/local/rocketmq/store/index

messageDelayLevel=1s 5s 10s 30s 1m 2m 3m 4m 5m 6m 7m 8m 9m 10m 20m 30m 1h 2h 6h 12h 1d 3d 15d

enablePropertyFilter=true

autoCreateTopicEnable=false`

3. Problem phenomenon：
   A lot of flow control log information often occurs，By tracking the code, it is found that the synchronous replication between the master node and the slave node has timed out：
   do sync transfer other node, wait return, but failed, topic
   putMessage not in lock eclipse time(ms)=6194

log warn 1

org.apache.rocketmq.client.exception.MQBrokerException: CODE: 2 DESC: [TIMEOUT_CLEAN_QUEUE]broker busy, start flow control for a while, period in queue: 209ms, size of queue: 56

log warn 2

2020-07-01 17:52:29 WARN SendMessageThread_8 - putMessage not in lock elapsed time(ms)=501, bodyLength=67
2020-07-01 17:52:29 WARN SendMessageThread_18 - putMessage not in lock elapsed time(ms)=501, bodyLength=67
2020-07-01 17:58:28 WARN SendMessageThread_3 - [NOTIFYME]putMessage in lock cost time(ms)=592, bodyLength=67 AppendMessageResult=AppendMessageResult{status=PUT_OK, wroteOffset=262888849194, wroteBytes=992, msgId='0A6F0C2D00002A9F0000003D35657F2A', storeTimestamp=1593597507430, logicsOffset=255311, pagecacheRT=592, msgNum=1}
2020-07-01 17:58:28 WARN SendMessageThread_19 - putMessage not in lock elapsed time(ms)=592, bodyLength=67
2020-07-01 17:58:28 WARN SendMessageThread_3 - putMessage not in lock elapsed time(ms)=593, bodyLength=67
2020-07-01 17:58:28 WARN SendMessageThread_16 - putMessage not in lock elapsed time(ms)=590, bodyLength=67
2020-07-01 17:58:28 WARN SendMessageThread_22 - putMessage not in lock elapsed time(ms)=589, bodyLength=67

4. How to troubleshoot and solve such problems？Should I upgrade to 4.7.x? [improvement] SYNC_MASTER could be change into pipeline manner #1515

[francisoliverlee]

@kayyellow
maybe brokerRole cause that

if ASYNC_MASTER
send to master broker, return send-result to client , async to slave broker

if SYNC_MASTER and Property key isWaitStoreMsgOK == true (isWaitStoreMsgOK default value is true)
send to master broker and sync to slave broker, return send-result to client (tps is very low)

public enum BrokerRole { ASYNC_MASTER, SYNC_MASTER, SLAVE; }

[kayyellow]

@kayyellow
maybe brokerRole cause that

if ASYNC_MASTER
send to master broker, return send-result to client , async to slave broker

if SYNC_MASTER and Property key isWaitStoreMsgOK == true (isWaitStoreMsgOK default value is true)
send to master broker and sync to slave broker, return send-result to client (tps is very low)

public enum BrokerRole { ASYNC_MASTER, SYNC_MASTER, SLAVE; }

OK， thx, I will try it.

##1515

According to the issue above， upgrade to 4.7.x seems can alse improve the performance of SYNC_MASTER. Should I change the brokerRole to ASYNC_MASTER, alse upgrade to 4.7.x to improve the performance？

[francisoliverlee]

it depends on how reliable you want rmq is. see the "code"