[PIP-136] Sync Pulsar metadata across multiple clouds #16425

rdhabalia · 2022-07-07T00:27:57Z

Motivation

Implementation of PIP-136
Sometimes it's not possible to share metadata-store (global zookeeper) between pulsar clusters deployed on separate cloud provider platforms, and synchronizing configuration metadata (policies) can be a critical path to share tenant/namespace/topic policies between clusters and administrate pulsar policies uniformly across all clusters. Therefore, we need a mechanism to sync configuration metadata between clusters deployed on the different cloud platforms.

PIP-136 also has a section Rejected alternative with use of system-topic along with metadata-synchronizer to replicate and persist local topic policies with it.

doc-complete

github-actions · 2022-07-07T00:28:22Z

@rdhabalia Please provide a correct documentation label for your PR.
Instructions see Pulsar Documentation Label Guide.

rdhabalia · 2022-07-26T06:17:37Z

@codelipenghui why did we change PR release to 2.12 ?
This PR is already ready for review for few weeks now.

eolivelli · 2022-07-26T19:33:41Z

@rdhabalia I think that it has been a bulk update.

I am reviewing the patch

eolivelli · 2022-07-26T19:36:16Z

pulsar-metadata/src/main/java/org/apache/pulsar/metadata/api/MetadataEvent.java

+@NoArgsConstructor
+@ToString
+public class MetadataEvent {
+    private String path;


Should we add some 'version' field in order to better support protocol enhancements?

I think we could have added version later on without any schema incompatibility issue. However, there is not much harm to add. I have added it.

eolivelli · 2022-07-26T19:41:14Z

...r-broker/src/main/java/org/apache/pulsar/broker/service/PulsarMetadataEventSynchronizer.java

+        if (consumer != null) {
+            return;
+        }
+        ConsumerBuilder<MetadataEvent> consumerBuilder = client.newConsumer(AvroSchema.of(MetadataEvent.class))


Nit:Please use Schema.AVRO()

Using AVRO with a Pojo that is not defined with the avro plugin leads to bad performance because it is going to use reflection.

We could generate the AVRO poco using the Maven plugin. I am 100% that it is worth (but not to be done in this patch)

I can change to Schema.AVRO() but it eventually calls AvroSchema.of. Also, it is called only once at server startup so, it should not have a performance issue.

as I said, it won't make much of a difference but I still have made the change so, we can progress for this PR.

eolivelli · 2022-07-26T19:42:26Z

pulsar-broker/src/test/java/org/apache/pulsar/client/api/SimpleProducerConsumerTest.java

@@ -221,7 +221,7 @@ public long millis() {
        }
    }

-    @Test(timeOut = 100000)
+    @Test//(timeOut = 100000)


Please revert

eolivelli · 2022-07-26T19:45:27Z

pulsar-metadata/src/main/java/org/apache/pulsar/metadata/impl/AbstractMetadataStore.java

+            MetadataEvent event = new MetadataEvent(path, null, new HashSet<>(),
+                    expectedVersion.orElse(null), Instant.now().toEpochMilli(),
+                    getMetadataEventSynchronizer().get().getClusterName(), NotificationType.Deleted);
+            return getMetadataEventSynchronizer().get().notify(event)


So we could write to the topic and then fail the write.

I understand this is a trade off.

How can we recover?
Like if I am deleting a topic and the delete fails, it may happen that on the remote cluster the topic is eventually deleted.

I wonder if we should use the topic as commit log and read from the topic from all the brokers and apply the changes coming only from the log

Yes, that was the idea to use Topic is a WAL and it's doing the same thing along with managing API compatibility.
We are building a generic solution for metadata-store which is used for local metadata store as well and these APIs expect response in the same session the and can't depend on message processing by the another broker. It also has to maintain the API compatibility. Therefore, it processes the request and also publish a message which will be eventually be processed and consistent across all the clusters. So, it gives guarantee to have eventual consistency as all the clusters including local cluster who is going to publish will also process the message.
We have gone through the life cycle of this behavior and it should work without doing any compromise or having inconsistency. In fact, this behavior is also documented in PIP.

add unit test add findbug

rdhabalia · 2022-07-31T05:26:25Z

@eolivelli thanks for reviewing it. I have addressed all the comments.

eolivelli

+1

codelipenghui · 2022-08-01T01:27:00Z

@rdhabalia, I have moved it back to 2.10.0. There are so many old PRs and in-progress PRs, I try to move them all to 2.12.0 first and move them back after they are completed.

* [PIP-136] Sync Pulsar policies across multiple clouds add unit test add findbug * address comments

mattisonchao · 2022-12-07T04:55:37Z

pulsar-broker/src/main/java/org/apache/pulsar/broker/PulsarService.java

@@ -350,7 +353,8 @@ public PulsarService(ServiceConfiguration config,
        this.offloaderStats = LedgerOffloaderStats.create(false, false, null, 0);
    }

-    public MetadataStore createConfigurationMetadataStore() throws MetadataStoreException {
+    public MetadataStore createConfigurationMetadataStore(PulsarMetadataEventSynchronizer synchronizer)


This broke the broker's public API.

rdhabalia added the area/broker label Jul 7, 2022

rdhabalia added this to the 2.11.0 milestone Jul 7, 2022

rdhabalia requested review from merlimat, lhotari, sijie, Jason918, jiazhai and eolivelli July 7, 2022 00:27

rdhabalia self-assigned this Jul 7, 2022

rdhabalia mentioned this pull request Jul 7, 2022

PIP-136: Sync Pulsar policies across multiple clouds #16424

Open

github-actions bot added the doc-label-missing label Jul 7, 2022

rdhabalia mentioned this pull request Jul 7, 2022

[PIP-136]: Sync Pulsar policies across multiple clouds #15223

Closed

rdhabalia added doc-complete Your PR changes impact docs and the related docs have been already added. and removed doc-label-missing labels Jul 7, 2022

rdhabalia changed the title ~~[PIP-136] Sync Pulsar policies across multiple clouds~~ [PIP-136] Sync Pulsar metadata across multiple clouds Jul 7, 2022

rdhabalia force-pushed the metadata_sync3 branch 2 times, most recently from 2388202 to 3c9b0b5 Compare July 21, 2022 02:11

codelipenghui modified the milestones: 2.11.0, 2.12.0 Jul 26, 2022

eolivelli reviewed Jul 26, 2022

View reviewed changes

[PIP-136] Sync Pulsar policies across multiple clouds

4c7c764

add unit test add findbug

rdhabalia force-pushed the metadata_sync3 branch from 3c9b0b5 to 4c7c764 Compare July 31, 2022 05:20

eolivelli approved these changes Jul 31, 2022

View reviewed changes

address comments

350ff16

rdhabalia force-pushed the metadata_sync3 branch from 3c89211 to 350ff16 Compare July 31, 2022 06:00

rdhabalia merged commit a38f74d into apache:master Jul 31, 2022

codelipenghui modified the milestones: 2.12.0, 2.11.0 Aug 1, 2022

Gleiphir2769 pushed a commit to Gleiphir2769/pulsar that referenced this pull request Aug 4, 2022

[PIP-136] Sync Pulsar metadata across multiple clouds (apache#16425)

f0b3b23

* [PIP-136] Sync Pulsar policies across multiple clouds add unit test add findbug * address comments

sijie mentioned this pull request Aug 12, 2022

ISSUE-16424: PIP-136: Sync Pulsar policies across multiple clouds streamnative/pulsar-archived#4485

Open

mattisonchao reviewed Dec 7, 2022

View reviewed changes

This was referenced May 9, 2024

[fix] [broker] Fix configurationMetadataSyncEventTopic is marked supporting dynamic setting, but not implemented #22684

Merged

[fix] [broker] local metadata sync topic contains configuration events causing all operations stuck #22695

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[PIP-136] Sync Pulsar metadata across multiple clouds #16425

[PIP-136] Sync Pulsar metadata across multiple clouds #16425

rdhabalia commented Jul 7, 2022 •

edited by github-actions bot

Loading

github-actions bot commented Jul 7, 2022

rdhabalia commented Jul 26, 2022

eolivelli commented Jul 26, 2022

eolivelli Jul 26, 2022

rdhabalia Jul 31, 2022

eolivelli Jul 26, 2022

rdhabalia Jul 31, 2022

rdhabalia Jul 31, 2022

eolivelli Jul 26, 2022

eolivelli Jul 26, 2022

rdhabalia Jul 31, 2022

rdhabalia commented Jul 31, 2022

eolivelli left a comment

codelipenghui commented Aug 1, 2022

mattisonchao Dec 7, 2022

[PIP-136] Sync Pulsar metadata across multiple clouds #16425

[PIP-136] Sync Pulsar metadata across multiple clouds #16425

Conversation

rdhabalia commented Jul 7, 2022 • edited by github-actions bot Loading

Motivation

github-actions bot commented Jul 7, 2022

rdhabalia commented Jul 26, 2022

eolivelli commented Jul 26, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rdhabalia commented Jul 31, 2022

eolivelli left a comment

Choose a reason for hiding this comment

codelipenghui commented Aug 1, 2022

Choose a reason for hiding this comment

rdhabalia commented Jul 7, 2022 •

edited by github-actions bot

Loading