[Bug] MessageRocksDBStorage accumulates timer-WAL flush tasks after reload

[LostSnowfluff]

Before Creating the Bug Report

• I found a bug, not just asking a question, which should be created in GitHub Discussions.

• I have searched the GitHub Issues and GitHub Discussions of this repository and believe that this is not a duplicate.

• I have confirmed that this bug belongs to the current repository, not other repositories of RocketMQ.

Runtime platform environment

Ubuntu 22.04.5 LTS, Linux x86_64.

RocketMQ version

Branch: develop
Version: 5.5.1
Git commit id: e90303810a14

JDK Version

OpenJDK 8u502.

Describe the Bug

MessageRocksDBStorage creates a ScheduledExecutorService and registers a
periodic timer-WAL flush task in postLoad():

private final ScheduledExecutorService scheduler =
    Executors.newScheduledThreadPool(1);

scheduler.scheduleAtFixedRate(this::flushTimerWal, 5, 5, TimeUnit.MINUTES);

However, preShutdown() does not cancel the scheduled task or shut down the
scheduler. AbstractRocksDBStorage.reloadRocksdb() uses the existing
shutdown() followed by start() lifecycle. Because the same scheduler
remains active, every successful reload registers another periodic flush task
on the same executor.

This also leaves a non-daemon scheduler and its delayed task alive after the
RocksDB resources have been closed.

Steps to Reproduce

1. Create a MessageRocksDBStorage with a temporary MessageStoreConfig.
2. Inspect its private scheduler as a ScheduledThreadPoolExecutor using a
   focused test or debugger. After construction, one periodic task is queued.
3. Call storage.shutdown().
4. Inspect the scheduler again and observe that it is not shut down and that
   its queue still contains one periodic task.
5. Call storage.start() to simulate the lifecycle used by
   AbstractRocksDBStorage.reloadRocksdb().
6. Call storage.shutdown() and storage.start() once more.
7. Inspect the scheduler after each start.

The affected code shows the following stable state transition:

Lifecycle state	Scheduler state	Queued periodic tasks
After construction	active	1
After first shutdown	not shut down	1
After first start	active	2
After second shutdown	not shut down	2
After second start	active	3

What Did You Expect to See?

• storage.shutdown() should cancel the periodic timer-WAL flush task and
  shut down its scheduler.
• Each subsequent start() should create one scheduler with exactly one
  periodic flush task.
• A scheduled task should not access RocksDB resources after they have been
  closed.

What Did You See Instead?

• The scheduler remains active after storage.shutdown().
• The delayed periodic task remains queued after the RocksDB instance and its
  options have been closed.
• Each shutdown/start reload adds another periodic flush task to the same
  executor.
• The scheduled runnable directly accesses db, flushOptions, and
  timerCFHandle without a lifecycle guard. If it runs during shutdown, it can
  access resources that are being closed or have already been cleared.

Additional Context

No response

[RockteMQ-AI]

Issue Evaluation

Category: type/bug | Status: Confirmed

The reported issue has been verified against the current codebase (develop branch).

Root Cause: In MessageRocksDBStorage, the ScheduledExecutorService scheduler (line 86) registers a periodic flushTimerWal task in postLoad() (line 117). However, preShutdown() (line 144) does not cancel the scheduled task or shut down the scheduler. When AbstractRocksDBStorage.reloadRocksdb() (line 685) calls shutdown() followed by start(), each reload registers another periodic flush task on the same executor without removing the previous one.

Impact:

• Timer-WAL flush tasks accumulate with each reload, causing unnecessary resource consumption
• Non-daemon scheduler threads remain alive after RocksDB resources are closed, potentially preventing clean JVM shutdown

Affected Components: store module — MessageRocksDBStorage, AbstractRocksDBStorage

Severity: Medium — resource leak that manifests under repeated reload scenarios

Suggested Fix: Store the ScheduledFuture returned by scheduleAtFixedRate() and cancel it in preShutdown(). Alternatively, shut down and recreate the scheduler in the shutdown()/start() lifecycle.

An automated fix proposal can be generated. Reply /approve to proceed with PR generation.

---

Automated evaluation by RockteMQ-AI