[Improve][TubeMQ] The storage space used by the offset of TubeMQ broker is too large

[FvvSuu]

Description

The storage space used by the offset of TubeMQ broker is too large.

There are 5~6 consumer groups in the picture

InLong Component

InLong TubeMQ

Are you willing to submit PR?

• Yes, I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[gosonzhang]

Thanks @FvvSuu, got it

[gosonzhang]

It should be caused by too much duplicate data in the offset records of the storage consumption group: in order to make the records more readable[1], for example, each partition consumption record contains the consumption group name, and each partition repeatedly records the maximum and minimum offsets, as a result, the total data record is bloated too much.

Considering the traversal of the text display, I plan to still use the text to represent the saved historical offset, but perform field key abbreviation and deduplication processing to reduce the length of the total record.

1. https://github.com/apache/inlong/blob/master/inlong-tubemq/tubemq-server/src/main/java/org/apache/inlong/tubemq/server/broker/offset/OffsetRecordInfo.java#L68