[improve][pip] PIP-433: Optimize the conflicts of the replication and automatic creation mechanisms, including the automatic creation of topics and schemas #24485
+172
−0
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
github-actions
bot
added
PIP
doc-required
|
The PIP also solves the issue #24417 |
3 tasks
Comment on lines
+14
to
+24
| #### Topic auto-creation by clients if you have enabled Geo Replication. | ||
| - `Client of source cluster`: start a consumer/producer for a topic | ||
| - `Client of source cluster`: Get partitions for topic | ||
| - At this step, the client will try to get the partitioned metadata of the topic. If the topic does not exist, the broker will create the partitioned metadata automatically. | ||
| - `Client of source cluster`: Start internal consumers/producers for each sub-topic | ||
| - At this step, the client will try to connect to the sub-topic. If the sub-topic does not exist, the broker will create the partitioned metadata automatically. | ||
| - `Source broker`: starts the geo replicator when a sub-topic is loading up. | ||
| - The geo replicator maintains an internal producer for the topic under the remote cluster. | ||
| - The internal producer is a single sub-topic producer; it will not trigger a partitioned metadata auto-creation. | ||
| - When starting the internal producer, it confirms that the target topic under the remote cluster should be a non-partitioned topic or a sub-topic. | ||
| - **(Highlight)** Otherwise, prints error logs and stops to start. |
There was a problem hiding this comment.
If the topic creation is triggered by the clients, the default partitions of topic auto-creation will be used in this case?
Comment on lines
+18
to
+19
| - `Client of source cluster`: Start internal consumers/producers for each sub-topic | ||
| - At this step, the client will try to connect to the sub-topic. If the sub-topic does not exist, the broker will create the partitioned metadata automatically. |
There was a problem hiding this comment.
What does sub topic means here? internal partition of a partitioned topic?
| -e, --enable Enable schema validation enforced | ||
| ``` | ||
|
|
||
| To add a new param `--always-allow-for-replicator-if-compatible`, which means that always allow the replicator to register a new schema if compatible. The default value is `true`. |
There was a problem hiding this comment.
Do we need this option in broker.conf?
| Default: non-partitioned | ||
| ``` | ||
|
|
||
| To add a new param `--always-allow-for-replicator`, which means that always allow the replicator to trigger a topic auto-creation, including partitioned topics. The default value is `true`. |
There was a problem hiding this comment.
Do we need this option in broker.conf?
Comment on lines
+58
to
+61
| 3. Checks compatibility between two clusters when enabling namespace-level replication, which includes the following | ||
| a. All topics’ partitions that have been created should be the same, including `__change_events` | ||
| b. Auto-creation policies should be the same | ||
| c. Support for adjusting policies automatically when enabling namespace-level policies, and provide an API to review the changes that will be applied. By the way, this feature will be an option param of enabling namespace-level geo replication |
There was a problem hiding this comment.
I think the geo-replication should get rid of the topic auto-creation. So we don't need to sync the policies.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Motivation & Modifications
Documentation
docdoc-requireddoc-not-neededdoc-completeMatching PR in forked repository
PR in forked repository: x