[Bug] The I/O operation is blocked

[KunioHaha]

Before Creating the Bug Report

• I found a bug, not just asking a question, which should be created in GitHub Discussions.

• I have searched the GitHub Issues and GitHub Discussions of this repository and believe that this is not a duplicate.

• I have confirmed that this bug belongs to the current repository, not other repositories of RocketMQ.

Runtime platform environment

Centos release 7.9.2009

RocketMQ version

5.1.0

JDK Version

1.8.0_202

Describe the Bug

BrokerControllerScheduledThread1 Blocked in slave broker, The broker will not appear in the cluster. java IO opration createNewFile or FileInputStream will be failed, and the Thread will be blocked. there are some logs in different slave broker, They have similar errors :

Steps to Reproduce

no steps

What Did You Expect to See?

The I/O operation should be released regardless of success or failure

What Did You See Instead?

Thread blocked

Additional Context

No response

[KunioHaha]

TimerMetrics.java

ConfigManager.java

all happened in persist function

[KunioHaha]

Its trigger time is 4 a.m., and it is speculated that this occurs when cleaning up expired commitLogs. In the production environment, the disk occupancy rate has been consistently between 84% and 85%. I wonder if this is related to that.

[yx9o]

Hi, do you have any relevant logs? Can you post them?

[KunioHaha]

Hi, do you have any relevant logs? Can you post them?

complete logical min offset slowly

High IO at 4 am

Which part of the logs do you need， store or broker?

[KunioHaha]

Hi, do you have any relevant logs? Can you post them?
Sometimes, executing the deletion process takes nearly 2 minutes.

[KunioHaha]

I currently suspect that the high IO operations caused by the deletion process lead to the blockage of other IO tasks, which in turn results in the Broker going offline.

Hi, do you have any relevant logs? Can you post them?
Sometimes, executing the deletion process takes nearly 2 minutes.

[KunioHaha]

Hi, do you have any relevant logs? Can you post them?

complete logical min offset slowly

High IO at 4 am

Which part of the logs do you need， store or broker?

However, I'm not sure if correctMinOffset is consuming a large amount of read IO. I don't think it should take up that much, but I can't find any other reasons either.