[feat][cpp] Support multiple brokers in service URL#17162
Merged
BewareMyPower merged 3 commits intoapache:masterfrom Aug 30, 2022
Merged
[feat][cpp] Support multiple brokers in service URL#17162BewareMyPower merged 3 commits intoapache:masterfrom
BewareMyPower merged 3 commits intoapache:masterfrom
Conversation
Contributor
Author
|
The |
b7cb675 to
43b166e
Compare
Contributor
Author
|
Draft this PR until the Python tests are fixed. |
### Motivation It's a catchup for apache#3249. Currently C++ client doesn't have strict validation on the service URL. If multiple brokers are specified, only the 1st broker will be used. ### Modifications - Add `ServiceURI` to support configuring multiple brokers in the service URL. See `ServiceURITest` for how to configure it. - Add `ServiceNameResolver` whose `resolveHost` method selects the next broker in a round robin way. - Since the broker's address for topic lookup is no longer the service URL itself, to handle the case when proxy is enabled, a `getBroker` method (like the same method in Java's `LookupService`), which returns the future of a pair of logical and physical addresses, is added to replace `lookupAsync`. - Apply `ServiceNameResolver` into `ClientImpl` and the `LookupService` implementations. - Rename `BinaryLookupServiceTest` to `LookupServiceTest` and add `testMultiAddresses` to test both `BinaryProtoLookupService` and `HTTPLookupService` that all available brokers will be accessed in a round robin way. ### TODO This is the 1st part to support multiple brokers. Even with this patch, if one of these hosts is not available, the topic lookup will fail immediately instead of trying other broker. Since this patch already includes many code changes, I will push another patch to fix it.
62d9fc3 to
f2ddba8
Compare
Demogorgon314
approved these changes
Aug 24, 2022
Contributor
Author
|
@RobertIndie @merlimat @rdhabalia @gaoran10 Could any of you take a second look? |
BewareMyPower
commented
Aug 30, 2022
RobertIndie
approved these changes
Aug 30, 2022
BewareMyPower
added a commit
that referenced
this pull request
Sep 15, 2022
### Motivation It's a catchup for #3249. Currently C++ client doesn't have strict validation on the service URL. If multiple brokers are specified, only the 1st broker will be used. ### Modifications - Add `ServiceURI` to support configuring multiple brokers in the service URL. See `ServiceURITest` for how to configure it. - Add `ServiceNameResolver` whose `resolveHost` method selects the next broker in a round robin way. - Since the broker's address for topic lookup is no longer the service URL itself, to handle the case when proxy is enabled, a `getBroker` method (like the same method in Java's `LookupService`), which returns the future of a pair of logical and physical addresses, is added to replace `lookupAsync`. - Apply `ServiceNameResolver` into `ClientImpl` and the `LookupService` implementations. - Rename `BinaryLookupServiceTest` to `LookupServiceTest` and add `testMultiAddresses` to test both `BinaryProtoLookupService` and `HTTPLookupService` that all available brokers will be accessed in a round robin way. ### TODO This is the 1st part to support multiple brokers. Even with this patch, if one of these hosts is not available, the topic lookup will fail immediately instead of trying other broker. Since this patch already includes many code changes, I will push another patch to fix it. (cherry picked from commit 786ab30)
7 tasks
|
Hi @BewareMyPower : this change supports multiple Broker URLs for brokers present in same cluster or it even supports brokers from different clusters? As this is not clearly mentioned in documentation. thanks, ... |
Contributor
Author
|
See my comment #3249 (comment) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
It's a catchup for #3249.
Currently C++ client doesn't have strict validation on the service URL.
If multiple brokers are specified, only the 1st broker will be used.
Modifications
ServiceURIto support configuring multiple brokers in theservice URL. See
ServiceURITestfor how to configure it.ServiceNameResolverwhoseresolveHostmethod selects thenext broker in a round robin way.
URL itself, to handle the case when proxy is enabled, a
getBrokermethod (like the same method in Java's
LookupService), which returnsthe future of a pair of logical and physical addresses, is added to
replace
lookupAsync.ServiceNameResolverintoClientImpland theLookupServiceimplementations.BinaryLookupServiceTesttoLookupServiceTestand addtestMultiAddressesto test bothBinaryProtoLookupServiceandHTTPLookupServicethat all available brokers will be accessed in around robin way.
Behavior Change
This PR also changes the behavior when the user passed service URL.
Before this PR, creating a
Clientobject will succeed but the methods tocreate a producer or consumer will return a failed
Result. After this PR,an
invalid_argumentexception will be thrown.TODO
This is the 1st part to support multiple brokers. Even with this patch,
if one of these hosts is not available, the topic lookup will fail
immediately instead of trying other broker. Since this patch already
includes many code changes, I will push another patch to fix it.
Documentation
Check the box below or label this PR directly.
Need to update docs?
doc-required(Your PR needs to update docs and you will update later)
doc-not-needed(Please explain why)
doc(Your PR contains doc changes)
doc-complete(Docs have been already added)