[Feature] Adding consume type that allows partition assign from the client

[gosonzhang]

Description

At present, before the consumer consumption data of TubeMQ, the Master must first perform the modulo distribution based on the partition set subscribed by the consumer group and the number of clients in the consumer group, and then the client can obtain the partitions that it is allowed to consume. This is controlled by the Master, which existing some shortcomings in the partition assign model:

1. Data consumption waiting time is too long: the client has a relatively long time from startup to final consumption to data. Because the Master allocates tasks in the consumption partition according to a fixed cycle, and the process involves the release of resources by the client of the allocated partition. In bad cases, the client may take more than a few minutes to get the partition for data consumption;

2. The partition allocation plan is not rich enough: The current service side performs Hash modulo allocation based on the total set of Topic partitions subscribed by the client and the total number of clients when the consumer group is allocated. However, some services require special allocation plans. , The current distribution plan on the service side is not friendly enough and cannot be changed at any time according to business needs;
3. Specified partition consumption is not supported: During the use of the version, the business feedback that the current service side management and control are not flexible enough, for example, the client needs to bind the consumption relationship between the consumer and the partition, or when you only want to consume certain partitions at a certain startup, The current service-side consumption control is not supported.

Recently, TubeMQ has been improved to address these shortcomings in our environment, adding consumers that allow partition allocation from the client, and this feature will be improved by this issue.

InLong Component

InLong TubeMQ

Are you willing to submit PR?

• Yes, I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

• Define the PB messages used in client partition-assigned #1775

• Adjust the logic of the consumer group on the Master side #1789

• Add consumer message processing logic #1794

• Add the client interface that supports partition allocation #1803

• [INLONG-1805]Add ClientBalanceConsumer class implementation #1806

• Add the principle introduction document of client partition assign #1825

[gosonzhang]

Completed, close the issue