Before Creating the Bug Report
Runtime platform environment
- OS: Linux
- Component: Broker (
ConsumerOffsetManager)
RocketMQ version
- branch: develop
- Git commit id: e348efa
JDK Version
JDK 8
Describe the Bug
ConsumerOffsetManager#queryMinOffsetInAllGroup(topic, filterGroups) iterates the live offsetTable.keySet() and removes the filtered groups' entries from it:
Set<String> topicGroups = this.offsetTable.keySet(); // live view of offsetTable
if (!UtilAll.isBlank(filterGroups)) {
for (String group : filterGroups.split(",")) {
Iterator<String> it = topicGroups.iterator();
while (it.hasNext()) {
String topicAtGroup = it.next();
if (group.equals(topicAtGroup.split(TOPIC_GROUP_SEPARATOR)[1])) {
it.remove(); // deletes the entry from offsetTable itself
removeConsumerOffset(topicAtGroup);
}
}
}
}
ConcurrentHashMap.keySet() is a live view, so it.remove() deletes every topic@group entry of the filtered groups from the real offset table. This method is called by AdminBrokerProcessor#queryCorrectionOffset (RequestCode.QUERY_CORRECTION_OFFSET), i.e. by running the read-only admin/diagnostic operation DefaultMQAdminExt#queryCorrectionOffset(topic, compareGroup, filterGroups).
Consequences:
- All consumer offsets of the filtered groups (for every topic on this broker) are wiped from memory; the next
persist() makes the deletion permanent (consumers.json no longer contains the keys). With RocksDBConsumerOffsetManager the removeConsumerOffset hook deletes the rows from RocksDB immediately.
- When the consumers of the filtered group commit/look up their offsets afterwards,
queryOffset returns -1 and consumption restarts according to consumeFromWhere — mass duplicate consumption or consumption skipping to the latest offset.
topicAtGroup.split(TOPIC_GROUP_SEPARATOR)[1] also throws ArrayIndexOutOfBoundsException if a malformed key without @ is present.
This looks like a copy-paste from the real cleanup methods (cleanOffset / removeOffset): a query method must never mutate the table; the filter groups were only meant to be excluded from the min-offset computation.
Steps to Reproduce
- Start a broker, let groups
G1 and G2 consume topic T so both groups have committed offsets.
- Run the admin operation
queryCorrectionOffset(T, G1, filterGroups="G2") once.
- Check
consumerOffset.json / getConsumerStatus: every T@G2 offset entry is gone and gets persisted that way.
A unit test asserting offsetTable still contains the filtered group's entry after calling queryMinOffsetInAllGroup fails on current develop.
What Did You Expect to See?
The query returns the min offsets excluding the filtered groups, without modifying offsetTable at all.
What Did You See Instead?
The query deletes the filtered groups' offsets from the live table (and from RocksDB/persisted JSON), causing silent offset loss.
Additional Context
Fix: compute on a snapshot of the key set and exclude the filter groups there, leaving offsetTable untouched. I will submit a PR with a regression test.
Before Creating the Bug Report
Runtime platform environment
ConsumerOffsetManager)RocketMQ version
JDK Version
JDK 8
Describe the Bug
ConsumerOffsetManager#queryMinOffsetInAllGroup(topic, filterGroups)iterates the liveoffsetTable.keySet()and removes the filtered groups' entries from it:ConcurrentHashMap.keySet()is a live view, soit.remove()deletes everytopic@groupentry of the filtered groups from the real offset table. This method is called byAdminBrokerProcessor#queryCorrectionOffset(RequestCode.QUERY_CORRECTION_OFFSET), i.e. by running the read-only admin/diagnostic operationDefaultMQAdminExt#queryCorrectionOffset(topic, compareGroup, filterGroups).Consequences:
persist()makes the deletion permanent (consumers.jsonno longer contains the keys). WithRocksDBConsumerOffsetManagertheremoveConsumerOffsethook deletes the rows from RocksDB immediately.queryOffsetreturns-1and consumption restarts according toconsumeFromWhere— mass duplicate consumption or consumption skipping to the latest offset.topicAtGroup.split(TOPIC_GROUP_SEPARATOR)[1]also throwsArrayIndexOutOfBoundsExceptionif a malformed key without@is present.This looks like a copy-paste from the real cleanup methods (
cleanOffset/removeOffset): a query method must never mutate the table; the filter groups were only meant to be excluded from the min-offset computation.Steps to Reproduce
G1andG2consume topicTso both groups have committed offsets.queryCorrectionOffset(T, G1, filterGroups="G2")once.consumerOffset.json/getConsumerStatus: everyT@G2offset entry is gone and gets persisted that way.A unit test asserting
offsetTablestill contains the filtered group's entry after callingqueryMinOffsetInAllGroupfails on current develop.What Did You Expect to See?
The query returns the min offsets excluding the filtered groups, without modifying
offsetTableat all.What Did You See Instead?
The query deletes the filtered groups' offsets from the live table (and from RocksDB/persisted JSON), causing silent offset loss.
Additional Context
Fix: compute on a snapshot of the key set and exclude the filter groups there, leaving
offsetTableuntouched. I will submit a PR with a regression test.