Broker-a busy but broker-b is normal

[syncwt]

BUG REPORT

1. Please describe the issue you observed:
   From the monitor of broker, I found broker's tps/qps always fluctuate at 1:00 am to 6:00 am（not disk clear time[deleteWhen]）. And Always broker-a decline and broker-b rise. It seems broker-a is wrong at that time. It would cause broker-b's load too high then produce message failed in sometimes .I checked broker log and founds nothing error about that.
   
   

• What did you do (The steps to reproduce)?
  It happens every day. I prepared to build a broker-c to reduce pressure for trouble time.

• What is expected to see?
  I expected to everything well.

• What did you see instead?

2. Please tell us about your environment:
   rocketmq version :4.8.0-bin-release ,deployed by private.
   2m-2s-async cluster mode . And master node is 4C12G.

3. Other information (e.g. detailed explanation, logs, related issues, suggestions on how to fix, etc):
   broker's conf as list:

brokerClusterName=DefaultCluster
brokerName=broker-a
brokerId=0

deleteWhen=06

fileReservedTime=168

waitTimeMillsInSendQueue=1000
#transientStorePoolEnable=true
transferMsgByHeap=false
sendMessageThreadPoolNums=16
useReentrantLockWhenPutMessage=true

brokerRole=ASYNC_MASTER
flushDiskType=ASYNC_FLUSH

And producer's client receive broker busy log as list:

[PCBUSY_CLEAN_QUEUE]broker busy, start flow control for a while, period in queue: 753ms, size of queue: 116

or

[REJECTREQUEST]system busy, start flow control for a while

or

[TIMEOUT_CLEAN_QUEUE]broker busy, start flow control for a while, period in queue: 1000ms, size of queue: 395

The broker's machine measures is all normal.

broker's storeerror.log

2023-02-07 01:51:15 WARN PullMessageThread_3 - Offset not matched. Request offset: 14474039918209, firstOffset: 25777319968768, lastOffset: 26379689132032, mappedFileSize: 1073741824, mappedFiles count: 561
2023-02-07 01:51:15 WARN PullMessageThread_16 - Offset not matched. Request offset: 14474039946505, firstOffset: 25777319968768, lastOffset: 26379689132032, mappedFileSize: 1073741824, mappedFiles count: 561
2023-02-07 01:51:15 WARN PullMessageThread_17 - Offset not matched. Request offset: 14474039961997, firstOffset: 25777319968768, lastOffset: 26379689132032, mappedFileSize: 1073741824, mappedFiles count: 561
...
2023-02-07 01:51:16 WARN PullMessageThread_18 - Offset not matched. Request offset: 13348880, firstOffset: 30000000, lastOffset: 36000000, mappedFileSize: 6000000, mappedFiles count: 1
2023-02-07 01:51:16 WARN PullMessageThread_19 - Offset not matched. Request offset: 13349180, firstOffset: 30000000, lastOffset: 36000000, mappedFileSize: 6000000, mappedFiles count: 1
2023-02-07 01:51:16 WARN PullMessageThread_20 - Offset not matched. Request offset: 13347020, firstOffset: 30000000, lastOffset: 36000000, mappedFileSize: 6000000, mappedFiles count: 1
...

[syncwt]

Through a period of investigation and analysis, It seems the different performance of machines's disk cause the problem. We are ongoing ensure and analyse. Maybe change a disk for broker-a is a right way to solve the problem.

[syncwt]

The issue is be confirmed. Tencent cbs cluster's load is too high that cause disk‘s io wait fluctuated. Then broker start flow control. We solved the problem by migrating cloud disk instance to a stable cluster finally.