[improve][broker] Explicitly close LB internal topics when playing a follower (ExtensibleLoadManagerImpl only)

[heesung-sn]

Motivation

Internal topics ownership conflict is a concern here when the leadership changes frequently - we should make sure only the current leader owns these internal topics.

The persistent internal topic can self-resolve this topic ownership conflict by the ledger fencing logic.

However, I think the non-persistent internal topics can be a problem because the fencing logic doesn't apply to non-persistent topics.

To gracefully release the previous ownership of these internal topics, I think we better make followers explicitly close the internal topics - when leadership changes, there might be a chance that some clients reconnect to the old leader if the connected zk and the old leader's zk are slow. Then, when the old leader zk catches up(notifies a follower signal to the old leader), the old leader should make sure to close any internal topics and its client connections, if any.

Modifications

• call closeInternalTopics() inside playFollower()

Verifying this change

• Make sure that the change passes the CI checks.

Does this pull request potentially affect one of the following parts:

If the box was checked, please highlight the changes

• Dependencies (add or upgrade a dependency)
• The public API
• The schema
• The default values of configurations
• The threading model
• The binary protocol
• The REST endpoints
• The admin CLI options
• The metrics
• Anything that affects deployment

Documentation

• doc
• doc-required
• doc-not-needed
• doc-complete

Matching PR in forked repository

PR in forked repository:

[github-actions]

@heesung-sn Please add the following content to your PR description and select a checkbox:

- [ ] `doc` <!-- Your PR contains doc changes -->
- [ ] `doc-required` <!-- Your PR changes impact docs and you will update later -->
- [ ] `doc-not-needed` <!-- Your PR changes do not impact docs -->
- [ ] `doc-complete` <!-- Docs have been already added -->

[lhotari]

so there could be fewer internal topic conflicts.

What is an internal topic conflict? What is the impact of it? How does it get resolved if the topic wouldn't be closed (before this PR)?

[codecov-commenter]

Codecov Report

Attention: Patch coverage is 81.81818% with 2 lines in your changes missing coverage. Please review.

Project coverage is 74.54%. Comparing base (bbc6224) to head (80d1187).
Report is 504 commits behind head on master.

Files	Patch %	Lines
...dbalance/extensions/ExtensibleLoadManagerImpl.java	81.81%	2 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##             master   #23144      +/-   ##
============================================
+ Coverage     73.57%   74.54%   +0.96%     
- Complexity    32624    34097    +1473     
============================================
  Files          1877     1919      +42     
  Lines        139502   144252    +4750     
  Branches      15299    15773     +474     
============================================
+ Hits         102638   107532    +4894     
+ Misses        28908    28485     -423     
- Partials       7956     8235     +279     

Flag	Coverage Δ
inttests	27.78% <45.45%> (+3.19%)	⬆️
systests	24.77% <9.09%> (+0.44%)	⬆️
unittests	73.90% <81.81%> (+1.05%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files	Coverage Δ
...dbalance/extensions/ExtensibleLoadManagerImpl.java	80.03% <81.81%> (-0.05%)	⬇️

... and 499 files with indirect coverage changes

[heesung-sn]

Internal topics ownership conflict is concern here when the leadership changes frequently - we should make sure only the current leader owns these internal topics.

The persistent internal topic can self-resolve this topic ownership conflict by the ledger fencing logic.

However, I think the non-persistent internal topics can be problem because the fencing logic doesn't apply to non-persistent topics.

To gracefully release the previous ownership of these internal topics, I think we better make followers explicitly close the internal topics - when leadership changes, there might be a chance that some clients reconnect to the old leader if the connected zk and the old leader's zk are slow. Then, when the old leader zk catches up(notifies a follower signal to the old leader), the old leader should make sure to close any internal topics and its client connections, if any.

[lhotari]

LGTM