[PIP-51] Introduce sticky consumer

[codelipenghui]

Motivation

Introduce sticky consumer, users can enable it by

client.newConsumer()
         .keySharedPolicy(KeySharedPolicy.sticky().ranges(Range.of(0, 10)))
         .subscribe();

Modifications

Add a new consumer selector named HashRangeExclusiveStickyKeyConsumerSelector to support sticky consumer.

This change added tests and can be verified as follows:

Add new unit tests.

Does this pull request potentially affect one of the following parts:

If yes was chosen, please highlight the changes

• Dependencies (does it add or upgrade a dependency): (no)
• The public API: (yes)
• The schema: (no)
• The default values of configurations: (no)
• The wire protocol: (yes)
• The rest endpoints: (no)
• The admin cli options: (no)
• Anything that affects deployment: (no)

Documentation

• Does this pull request introduce a new feature? (yes)

[codelipenghui]

run java8 tests
run Integration Tests

[codelipenghui]

run java8 tests
run Integration Tests

[aahmed-se]

run java8 tests
run integration tests.

[codelipenghui]

retest this please

[codelipenghui]

retest this please

[codelipenghui]

retest this please

[codelipenghui]

rerun cpp tests

[codelipenghui]

rerun cpp tests

[codelipenghui]

run java8 tests
run integration tests

[codelipenghui]

run java8 tests
run integration tests

[codelipenghui]

run java8 tests

[codelipenghui]

run integration tests

[codelipenghui]

run integration tests

[codelipenghui]

run integration tests

[sijie]

@codelipenghui this needs to be PIP-47?

[codelipenghui]

@sijie Yes, it should be PIP-47

[codelipenghui]

run java8 tests

[codelipenghui]

run java8 tests
run integration tests

[codelipenghui]

run cpp tests

[sijie]

@codelipenghui implementation looks good to me in general. however I don't think we should let user specify the total hash range. we should pick a fixed hash range (e.g. 0~66536), and provide util functions for consumers to split the hash range based on their requirements. please take a look at my detailed comment below.

[sijie]

probably just call it STICKY?

[codelipenghui]

Yes, STICKY is more simpler

[sijie]

I don't think we should allow users to configure thee totalHashRange. The hash range should be fixed (e.g. 0.0 ~ 1.0, Long.MIN_VALUE ~ Long.MAX_VALUE, or Integer.MIN_VALUE to Integer.MAX_VALUE, or 0 ~ 66536). Just pick up a range, and let consumer specify the hash range.

Allowing user configuring the totalHashRange introduces inconsistent across consumers of a subscription. people can make mistakes.

[codelipenghui]

Ok, i will fix by using a fixed range(0-65536), current total range for key shared subscription is also (0-65536)

[sijie]

Guava has a Range class. You might consider reusing its class.

[codelipenghui]

The pulsar client api module do not add guava dependency, shall we add the guava dependency?

[codelipenghui]

@sijie I have addressed you comments except the new Range class in pulsar client api module, because currently the pulsar client api module do not add guava dependency, considering to keep api module thin, so i add the new Range class.

[sijie]

@codelipenghui lgtm. this is a great addon to key_shared subscription.

I left one minor comment.

[sijie]

can you use upper case for enums?

AUTO_SPLIT = 0;
STICKY = 1;

[codelipenghui]

@sijie I have added a task tracker #5586 to track the reset cursor approach for sticky consumer.

[sijie]

@codelipenghui can you update the enum?

[codelipenghui]

@sijie I can update it tomorrow, my mac book has some problems with docker environment, can not build the proto. need to build them on another machine 😊

[codelipenghui]

run java8 tests

[codelipenghui]

@sijie I have updated the enum in PulsarApi.proto