[improve][broker] PIP-192: Write the child ownership to `ServiceUnitStateChannel` instead of ZK when handling bundle split #18858

Demogorgon314 · 2022-12-09T14:46:00Z

Master Issue: #16691

Motivation

When the channel handles the split event, we should write the child ownership to ServiceUnitStateChannel instead of zookeeper, since we are using the ServiceUnitStateChannel to store the bundle ownership.

Modifications

Write the child ownership to ServiceUnitStateChannel instead of ZK when handling bundle split.
Retry when split bundle has BadVersion exception.
Add unit test to cover this change.

Documentation

doc
doc-required
doc-not-needed
doc-complete

Matching PR in forked repository

PR in forked repository: Demogorgon314#8

...ava/org/apache/pulsar/broker/loadbalance/extensions/channel/ServiceUnitStateChannelImpl.java

...ava/org/apache/pulsar/broker/loadbalance/extensions/channel/ServiceUnitStateChannelTest.java

...ava/org/apache/pulsar/broker/loadbalance/extensions/channel/ServiceUnitStateChannelImpl.java

codecov-commenter · 2023-01-04T03:41:29Z

Codecov Report

Merging #18858 (e97ca14) into master (9ec1d07) will decrease coverage by 0.24%.
The diff coverage is 0.00%.

@@             Coverage Diff              @@
##             master   #18858      +/-   ##
============================================
- Coverage     47.43%   47.19%   -0.25%     
- Complexity     9535    10685    +1150     
============================================
  Files           632      712      +80     
  Lines         59839    69722    +9883     
  Branches       6234     7485    +1251     
============================================
+ Hits          28384    32904    +4520     
- Misses        28375    33098    +4723     
- Partials       3080     3720     +640

Flag	Coverage Δ
unittests	`47.19% <0.00%> (-0.25%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
...xtensions/channel/ServiceUnitStateChannelImpl.java	`0.00% <0.00%> (ø)`
...ache/pulsar/broker/namespace/NamespaceService.java	`56.38% <0.00%> (-1.84%)`	⬇️
.../pulsar/broker/service/SharedConsumerAssignor.java	`3.70% <0.00%> (-64.82%)`	⬇️
...apache/pulsar/broker/service/EntryAndMetadata.java	`0.00% <0.00%> (-40.75%)`	⬇️
...e/pulsar/broker/service/EntryBatchIndexesAcks.java	`82.14% <0.00%> (-10.72%)`	⬇️
...ersistentStickyKeyDispatcherMultipleConsumers.java	`51.96% <0.00%> (-8.34%)`	⬇️
...sistent/PersistentDispatcherMultipleConsumers.java	`51.96% <0.00%> (-7.18%)`	⬇️
...r/service/AbstractDispatcherMultipleConsumers.java	`66.12% <0.00%> (-4.84%)`	⬇️
...oker/service/schema/SchemaRegistryServiceImpl.java	`62.13% <0.00%> (-4.05%)`	⬇️
...e/HashRangeExclusiveStickyKeyConsumerSelector.java	`36.92% <0.00%> (-3.08%)`	⬇️
... and 120 more

codelipenghui · 2023-01-17T04:46:44Z

When the channel handles the split event, we should write the split bundle to zookeeper instead of ServiceUnitStateChannel,

@Demogorgon314 I think it's incorrect in the PR's description, right?

Demogorgon314 · 2023-01-17T04:52:17Z

I think it's incorrect in the PR's description, right?

@codelipenghui Thanks for reminding me, I have fixed it.

codelipenghui

Hmmm, I think the right solution is to improve the NamespaceService to decide to write to Zookeeper or service unit channel? We also have Admin API to split the bundle, it will be a problem if the load manager write the bundle split to service unit state channel, but the Admin API can't write to the service unit channel?

Demogorgon314 · 2023-01-17T05:15:05Z

Hmmm, I think the right solution is to improve the NamespaceService to decide to write to Zookeeper or service unit channel? We also have Admin API to split the bundle, it will be a problem if the load manager write the bundle split to service unit state channel, but the Admin API can't write to the service unit channel?

Yes, then we need to wait for this PR merge first. #19102

# Conflicts: # pulsar-broker/src/test/java/org/apache/pulsar/broker/loadbalance/extensions/channel/ServiceUnitStateChannelTest.java

pulsar-broker/src/main/java/org/apache/pulsar/broker/namespace/NamespaceService.java

...ava/org/apache/pulsar/broker/loadbalance/extensions/channel/ServiceUnitStateChannelImpl.java

# Conflicts: # pulsar-broker/src/test/java/org/apache/pulsar/broker/loadbalance/extensions/channel/ServiceUnitStateChannelTest.java

Demogorgon314 · 2023-02-07T07:10:38Z

Hmmm, I think the right solution is to improve the NamespaceService to decide to write to Zookeeper or service unit channel? We also have Admin API to split the bundle, it will be a problem if the load manager write the bundle split to service unit state channel, but the Admin API can't write to the service unit channel?

@codelipenghui I've had a second thought, Actually, we should keep this method in ServiceUnitStateChannel . The split operation should call by channel when receiving the Splitting state, like the unload operation. We can support the admin API later in other PRs.

eolivelli · 2023-02-08T07:56:48Z

...ava/org/apache/pulsar/broker/loadbalance/extensions/channel/ServiceUnitStateChannelImpl.java

+                        bundleFactory.invalidateBundleCache(bundle.getNamespaceObject());
+                        updateFuture.complete(splitBundles);
+                    }).exceptionally(e -> {
+                        // Clean the new bundle when has exception.


what about logging the error here ?

When the updateFuture.completeExceptionally(e); is called, the log will print in this place. https://github.com/apache/pulsar/pull/18858/files#diff-191ed1df4f5804c8fd6cdaf909cca01d071a110c3e701bdc98352f99e7a92e8eR534

eolivelli · 2023-02-08T07:57:39Z

...ava/org/apache/pulsar/broker/loadbalance/extensions/channel/ServiceUnitStateChannelImpl.java

+                        FutureUtil.waitForAll(futureList)
+                                .whenComplete((__, ex) -> {
+                                    if (ex != null) {
+                                        log.warn("Clean new bundles failed,", ex);


what are the consequences of this failure ?

can you please point me out to where we handle this inconsistent state?

There are two cases to entry this failure.

When pubAsync has an exception.

When updateNamespaceBundles, updateNamespaceBundlesForPolicies has exception.

In these two cases, we won’t write the new bundle to ZK, so when the client tries to lookup, it will still find the old bundle, but some of the new bundle will be left in table-view, and when the broker is down, the unused bundle will be deleted in table-view.

eolivelli · 2023-02-08T07:59:28Z

...ava/org/apache/pulsar/broker/loadbalance/extensions/channel/ServiceUnitStateChannelImpl.java

+                    && (counter.incrementAndGet() < NamespaceService.BUNDLE_SPLIT_RETRY_LIMIT)) {
+                pulsar.getExecutor().schedule(() -> splitServiceUnitOnceAndRetry(namespaceService, bundleFactory,
+                                bundle, serviceUnit, data, counter, startTime, completionFuture), 100, MILLISECONDS);
+            } else if (ex instanceof IllegalArgumentException) {


why we are doing the test on "ex" here and on "ex.getCause()" above ?

and why this case is so special ? we should at least log that we are passing from this branch

Good catch. It should use FutureUtil.unwrapCompletionException(ex) here.

# Conflicts: # pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/extensions/channel/ServiceUnitStateChannelImpl.java

...ava/org/apache/pulsar/broker/loadbalance/extensions/channel/ServiceUnitStateChannelImpl.java

eolivelli

+1

Demogorgon314 self-assigned this Dec 9, 2022

github-actions bot added the doc-not-needed Your PR changes do not impact docs label Dec 9, 2022

Demogorgon314 mentioned this pull request Dec 12, 2022

PIP-192: New Pulsar Broker Load Balancer #16691

Closed

heesung-sn reviewed Dec 14, 2022

View reviewed changes

eolivelli reviewed Dec 16, 2022

View reviewed changes

...ava/org/apache/pulsar/broker/loadbalance/extensions/channel/ServiceUnitStateChannelImpl.java Outdated Show resolved Hide resolved

heesung-sn approved these changes Dec 16, 2022

View reviewed changes

Demogorgon314 changed the title ~~[improve][broker] Write the child ownership to ServiceUnitStateChannel instead of ZK when handling bundle split~~ [improve][broker] PIP-192: Write the child ownership to ServiceUnitStateChannel instead of ZK when handling bundle split Dec 18, 2022

Demogorgon314 requested a review from eolivelli December 26, 2022 02:12

codelipenghui added this to the 2.12.0 milestone Dec 30, 2022

Demogorgon314 requested review from heesung-sn, Jason918, BewareMyPower and codelipenghui January 3, 2023 01:46

Demogorgon314 added the ready-to-test label Jan 3, 2023

heesung-sn reviewed Jan 3, 2023

View reviewed changes

...ava/org/apache/pulsar/broker/loadbalance/extensions/channel/ServiceUnitStateChannelImpl.java Outdated Show resolved Hide resolved

Demogorgon314 force-pushed the Demogorgon314/PIP-192-improve-split branch from 369c5d4 to e97ca14 Compare January 4, 2023 03:24

codelipenghui reviewed Jan 17, 2023

View reviewed changes

Demogorgon314 requested a review from shibd January 19, 2023 01:58

Demogorgon314 added 5 commits January 31, 2023 17:00

Write the child ownerships to BSC instead of ZK when split bundle

93fc19a

Retry when split bundle has BadVersion exception

55d7a62

Rename the msg field

4a0e2e4

Reduce unused logs

c9f132f

Address reviewer's comments

ca2878e

Demogorgon314 added 6 commits January 31, 2023 17:02

Cleanup the new bundle when has exception.

e596489

Clean the child bundles when test completed

732fd42

Use owned state

19e322c

Use scheduler direct

922186f

Revert handle exception

c99681d

Merge master into current branch

568a39b

Demogorgon314 force-pushed the Demogorgon314/PIP-192-improve-split branch from e97ca14 to 568a39b Compare January 31, 2023 09:27

Demogorgon314 added 2 commits February 3, 2023 15:50

Merge branch 'master' into Demogorgon314/PIP-192-improve-split

02223c2

# Conflicts: # pulsar-broker/src/test/java/org/apache/pulsar/broker/loadbalance/extensions/channel/ServiceUnitStateChannelTest.java

Merge master to current branch

38c019b

gaoran10 reviewed Feb 6, 2023

View reviewed changes

Demogorgon314 marked this pull request as draft February 6, 2023 09:25

Demogorgon314 added 3 commits February 6, 2023 18:33

Refactor getSplitBoundary method

563da58

Merge branch 'master' into Demogorgon314/PIP-192-improve-split

a81653a

# Conflicts: # pulsar-broker/src/test/java/org/apache/pulsar/broker/loadbalance/extensions/channel/ServiceUnitStateChannelTest.java

Fix test

a13d4b3

gaoran10 approved these changes Feb 6, 2023

View reviewed changes

Demogorgon314 marked this pull request as ready for review February 7, 2023 07:10

Demogorgon314 closed this Feb 7, 2023

Demogorgon314 reopened this Feb 7, 2023

eolivelli requested changes Feb 8, 2023

View reviewed changes

Use unwrap completion exception when has exception

3a7b073

Demogorgon314 requested a review from eolivelli February 11, 2023 06:48

Merge branch 'master' into Demogorgon314/PIP-192-improve-split

6e95835

# Conflicts: # pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/extensions/channel/ServiceUnitStateChannelImpl.java

codelipenghui reviewed Feb 13, 2023

View reviewed changes

Demogorgon314 added 2 commits February 13, 2023 16:34

Address review's comment

137f893

Use unmodifiable list

5d440f4

codelipenghui approved these changes Feb 13, 2023

View reviewed changes

eolivelli approved these changes Feb 13, 2023

View reviewed changes

eolivelli merged commit 950ff44 into apache:master Feb 13, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[improve][broker] PIP-192: Write the child ownership to `ServiceUnitStateChannel` instead of ZK when handling bundle split #18858

[improve][broker] PIP-192: Write the child ownership to `ServiceUnitStateChannel` instead of ZK when handling bundle split #18858

Demogorgon314 commented Dec 9, 2022 •

edited

codecov-commenter commented Jan 4, 2023 •

edited

codelipenghui commented Jan 17, 2023

Demogorgon314 commented Jan 17, 2023

codelipenghui left a comment

Demogorgon314 commented Jan 17, 2023

Demogorgon314 commented Feb 7, 2023

eolivelli Feb 8, 2023

Demogorgon314 Feb 8, 2023

eolivelli Feb 8, 2023

Demogorgon314 Feb 8, 2023

eolivelli Feb 8, 2023

Demogorgon314 Feb 8, 2023

eolivelli left a comment

[improve][broker] PIP-192: Write the child ownership to ServiceUnitStateChannel instead of ZK when handling bundle split #18858

[improve][broker] PIP-192: Write the child ownership to ServiceUnitStateChannel instead of ZK when handling bundle split #18858

Conversation

Demogorgon314 commented Dec 9, 2022 • edited

Motivation

Modifications

Documentation

Matching PR in forked repository

codecov-commenter commented Jan 4, 2023 • edited

Codecov Report

codelipenghui commented Jan 17, 2023

Demogorgon314 commented Jan 17, 2023

codelipenghui left a comment

Choose a reason for hiding this comment

Demogorgon314 commented Jan 17, 2023

Demogorgon314 commented Feb 7, 2023

eolivelli Feb 8, 2023

Choose a reason for hiding this comment

Demogorgon314 Feb 8, 2023

Choose a reason for hiding this comment

eolivelli Feb 8, 2023

Choose a reason for hiding this comment

Demogorgon314 Feb 8, 2023

Choose a reason for hiding this comment

eolivelli Feb 8, 2023

Choose a reason for hiding this comment

Demogorgon314 Feb 8, 2023

Choose a reason for hiding this comment

eolivelli left a comment

Choose a reason for hiding this comment

[improve][broker] PIP-192: Write the child ownership to `ServiceUnitStateChannel` instead of ZK when handling bundle split #18858

[improve][broker] PIP-192: Write the child ownership to `ServiceUnitStateChannel` instead of ZK when handling bundle split #18858

Demogorgon314 commented Dec 9, 2022 •

edited

codecov-commenter commented Jan 4, 2023 •

edited