[fix] [broker] Prevent long deduplication cursor backlog so that topic loading wouldn't timeout

[poorbarcode]

Motivation

Background 1

• When a topic is loading up, it will wait for the deduplication recovery to be finished if deduplication is enabled.
  • users can enable deduplication on the broker level, namespace level or topic level.
• There is a scheduled task to acknowledge messages for the deduplication cursor.

Issue 1
(Highlight) the scheduled task that acknowledges messages for the deduplication cursor can only be created when the broker level deduplication is enabled, leading to an infinite backlog of deduplication cursor; the next deduplication recovery after a topic unloading will take a long time, which leads to topic load timeout.

---

Background 2

• MessageDeduplication maintains a counter that indicates how many messages have not been taken snapshot yet, it increases after per add entry. Broker triggers a snapshot building once the counter is greater than the config brokerDeduplicationEntriesInterval(default value is 1000).

Issue 2, thanks for @mattisonchao 's reminding ❤️

• send 999 messages, the counter increases to 999, does not reach brokerDeduplicationEntriesInterval(1000) now.
• unload the topic, the counter will be reset to 0,
• send 999 messages, the counter increases to 999, does not reach brokerDeduplicationEntriesInterval(1000) now.
• ...
• So the snapshot will never be built, resulting the same issue as the scenario 1.

A topic internal stats
topic-internal-stats.txt

Modifications

• Always start the scheduled task.

Documentation

• doc
• doc-required
• doc-not-needed
• doc-complete

Matching PR in forked repository

PR in forked repository: x

[lhotari]

The title "Deduplication cursor has infinite backlog causing topic load timeout" is misleading since the backlog isn't infinite.
This fix doesn't really address the case when there's a long backlog at the time of topic loading. It's trying to prevent the long backlog from building up. Perhaps "Prevent long deduplication cursor backlog so that topic loading wouldn't timeout" would be a better title?

[poorbarcode]

The title "Deduplication cursor has infinite backlog causing topic load timeout" is misleading since the backlog isn't infinite.
This fix doesn't really address the case when there's a long backlog at the time of topic loading. It's trying to prevent the long backlog from building up. Perhaps "Prevent long deduplication cursor backlog so that topic loading wouldn't timeout" would be a better title?

Done

[kristgpt]

👍

[poorbarcode]

This may leave BrokerDeduplicationEnabled not working, and we also have admin-API for getting/setting this value

Do not worry, it only affects the scheduled task.

[lhotari]

@poorbarcode DeduplicationDisabledBrokerLevelTest.testSnapshotCounterAfterUnload fails in branch-3.2 and blocks 3.2.3 release. do you have a chance to fix the problem in branch-3.2 ?

[poorbarcode]

@poorbarcode DeduplicationDisabledBrokerLevelTest.testSnapshotCounterAfterUnload fails in branch-3.2 and blocks 3.2.3 release. do you have a chance to fix the problem in branch-3.2 ?

@lhotari found the issue. PR 22034 exists in branch-3.2 so the test is wrong. resolved with commit c2532b9