[pulsar-broker] Implement AutoSubscriptionCreation by namespace override #6637

murong00 · 2020-03-30T11:46:43Z

Motivation

Add a new namespace policy autoSubscriptionCreationOverride which will enable an override of broker autoSubscriptionCreation settings on the namespace level. Users can keep autoSubscriptionCreation disabled for the broker and allow it on a specific namespace using this feature.

Modifications

Add a new namespace policy: autoSubscriptionCreationOverride and associated API / CLI interface for setting and removing.
When checking autoSubscriptionCreation configuration, the broker first retrieves namespace policies from zookeeper. If not set, it falls back to broker configuration.
Some minor improvement on autoTopicCreationOverride (e.g. v1 Namespaces API & cli md)

jiazhai · 2020-03-31T01:22:48Z

@murong00 thanks for this feature. Please check the CI failures.

jiazhai · 2020-04-02T00:52:40Z

/pulsarbot run-failure-checks

jiazhai · 2020-04-02T14:56:08Z

/pulsarbot run-failure-checks

codelipenghui · 2020-04-06T11:36:43Z

pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/impl/NamespacesBase.java

+                            globalZk().setData(path(POLICIES, namespaceName.toString()),
+                                    jsonMapper().writeValueAsBytes(policiesNode.getKey()), policiesNode.getValue().getVersion());


It's better to call setData() asynchronously.

codelipenghui · 2020-04-06T11:37:23Z

pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/impl/NamespacesBase.java

+                            globalZk().setData(path(POLICIES, namespaceName.toString()),
+                                    jsonMapper().writeValueAsBytes(policiesNode.getKey()), policiesNode.getValue().getVersion());


Same as the above comment.

murong00 · 2020-04-07T03:14:12Z

@codelipenghui I hava addressed your comment, please help to take a look, thanks.

murong00 · 2020-04-07T10:55:32Z

/pulsarbot run-failure-checks

murong00 · 2020-04-08T01:24:32Z

/pulsarbot run-failure-checks

codelipenghui · 2020-04-08T08:51:57Z

/pulsarbot run-failure-checks

codelipenghui · 2020-04-08T10:50:39Z

pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/impl/NamespacesBase.java

+                            log.info("[{}] Successfully removed autoTopicCreation override on namespace {}", clientAppId(), namespaceName);
+                            asyncResponse.resume(Response.noContent().build());


The response should complete in the zookeeper callback. Otherwise, the broker always returns success.

@codelipenghui Thanks for pointing out this, have fixed.

codelipenghui · 2020-04-08T10:52:49Z

pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/impl/NamespacesBase.java

+                                    jsonMapper().writeValueAsBytes(policiesNode.getKey()), policiesNode.getValue().getVersion(),
+                                    (rc, path1, ctx, stat) -> {
+                                        if (rc == KeeperException.Code.OK.intValue()) {
+                                            policiesCache().invalidate(path(POLICIES, namespaceName.toString()));


Looks should complete the asyncResponse here.

codelipenghui · 2020-04-08T10:53:54Z

pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/impl/NamespacesBase.java

+                            String autoOverride = autoTopicCreationOverride.allowAutoTopicCreation ? "enabled" : "disabled";
+                            log.info("[{}] Successfully {} autoTopicCreation on namespace {}", clientAppId(), autoOverride, namespaceName);


Should move to the zookeeper callback.

codelipenghui · 2020-04-08T10:55:17Z

pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/impl/NamespacesBase.java

+                            String autoOverride = autoSubscriptionCreationOverride.allowAutoSubscriptionCreation ? "enabled" : "disabled";
+                            log.info("[{}] Successfully {} autoSubscriptionCreation on namespace {}", clientAppId(), autoOverride, namespaceName);
                            asyncResponse.resume(Response.noContent().build());


Should move to the zookeeper callback.

codelipenghui · 2020-04-08T10:55:44Z

pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/impl/NamespacesBase.java

+                            log.info("[{}] Successfully removed autoSubscriptionCreation override on namespace {}", clientAppId(), namespaceName);
+                            asyncResponse.resume(Response.noContent().build());


Should move to the zookeeper callback.

murong00 · 2020-04-09T04:24:41Z

/pulsarbot run-failure-checks

murong00 · 2020-04-09T06:49:18Z

/pulsarbot run-failure-checks

murong00 · 2020-04-09T08:57:47Z

/pulsarbot run-failure-checks

sijie · 2020-04-09T19:23:28Z

/pulsarbot run-failure-checks

sijie · 2020-04-12T21:10:13Z

/pulsarbot run-failure-checks

codelipenghui · 2020-04-13T08:46:13Z

@murong00 Could you please rebase your branch since some test-related fixes merged into the master branch.

Fix checkstyle.

murong00 · 2020-04-14T01:47:37Z

/pulsarbot run-failure-checks

Anonymitaet · 2020-06-10T06:31:31Z

Docs have been added: https://pulsar.apache.org/docs/en/next/pulsar-admin/#set-auto-topic-creation

…ide (apache#6637) ### Motivation Add a new namespace policy `autoSubscriptionCreationOverride` which will enable an override of broker `autoSubscriptionCreation` settings on the namespace level. Users can keep `autoSubscriptionCreation` disabled for the broker and allow it on a specific namespace using this feature. ### Modifications * Add a new namespace policy: `autoSubscriptionCreationOverride` and associated API / CLI interface for setting and removing. * When checking `autoSubscriptionCreation` configuration, the broker first retrieves namespace policies from zookeeper. If not set, it falls back to broker configuration. * Some minor improvement on `autoTopicCreationOverride` (e.g. v1 Namespaces API & cli md)

sijie approved these changes Apr 5, 2020

View reviewed changes

sijie assigned murong00 Apr 5, 2020

sijie added area/admin area/broker labels Apr 5, 2020

sijie added this to the 2.6.0 milestone Apr 5, 2020

codelipenghui reviewed Apr 6, 2020

View reviewed changes

murong00 force-pushed the branch-6637 branch from 4df8bad to be660fb Compare April 7, 2020 03:10

murong00 force-pushed the branch-6637 branch from be660fb to 79a3a5d Compare April 7, 2020 09:26

codelipenghui approved these changes Apr 8, 2020

View reviewed changes

codelipenghui self-requested a review April 8, 2020 10:56

codelipenghui requested changes Apr 8, 2020

View reviewed changes

codelipenghui approved these changes Apr 9, 2020

View reviewed changes

sijie added the doc-required Your PR changes impact docs and you will update later. label Apr 9, 2020

murong00 added 3 commits April 13, 2020 19:20

Implement AutoSubscriptionCreation by namespace override.

c096fb0

Fix checkstyle.

Call setData asynchronously

c0bfa24

Complete the asyncResponse correctly.

9c6e7db

murong00 force-pushed the branch-6637 branch from dbf3ca7 to 9c6e7db Compare April 13, 2020 11:21

sijie merged commit d8be7c5 into apache:master Apr 14, 2020

sijie mentioned this pull request May 20, 2020

[discussion] Pulsar release 2.6.0 #5819

Closed

Anonymitaet removed the doc-required Your PR changes impact docs and you will update later. label Jun 10, 2020

Jason918 mentioned this pull request Nov 1, 2022

[feature][broker] Support autoSubscriptionCreation on topic level #17720

Merged

1 task

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[pulsar-broker] Implement AutoSubscriptionCreation by namespace override #6637

[pulsar-broker] Implement AutoSubscriptionCreation by namespace override #6637

murong00 commented Mar 30, 2020

jiazhai commented Mar 31, 2020

jiazhai commented Apr 2, 2020

jiazhai commented Apr 2, 2020

codelipenghui Apr 6, 2020

murong00 Apr 7, 2020

codelipenghui Apr 6, 2020

murong00 Apr 7, 2020

murong00 commented Apr 7, 2020

murong00 commented Apr 7, 2020

murong00 commented Apr 8, 2020

codelipenghui commented Apr 8, 2020

codelipenghui Apr 8, 2020

murong00 Apr 9, 2020 •

edited

Loading

codelipenghui Apr 8, 2020

murong00 Apr 9, 2020

codelipenghui Apr 8, 2020

murong00 Apr 9, 2020

codelipenghui Apr 8, 2020

murong00 Apr 9, 2020

codelipenghui Apr 8, 2020

murong00 Apr 9, 2020

murong00 commented Apr 9, 2020

murong00 commented Apr 9, 2020

murong00 commented Apr 9, 2020

sijie commented Apr 9, 2020

sijie commented Apr 12, 2020

codelipenghui commented Apr 13, 2020

murong00 commented Apr 14, 2020

Anonymitaet commented Jun 10, 2020

		globalZk().setData(path(POLICIES, namespaceName.toString()),
		jsonMapper().writeValueAsBytes(policiesNode.getKey()), policiesNode.getValue().getVersion());

		log.info("[{}] Successfully removed autoTopicCreation override on namespace {}", clientAppId(), namespaceName);
		asyncResponse.resume(Response.noContent().build());

		String autoOverride = autoTopicCreationOverride.allowAutoTopicCreation ? "enabled" : "disabled";
		log.info("[{}] Successfully {} autoTopicCreation on namespace {}", clientAppId(), autoOverride, namespaceName);

[pulsar-broker] Implement AutoSubscriptionCreation by namespace override #6637

[pulsar-broker] Implement AutoSubscriptionCreation by namespace override #6637

Conversation

murong00 commented Mar 30, 2020

Motivation

Modifications

jiazhai commented Mar 31, 2020

jiazhai commented Apr 2, 2020

jiazhai commented Apr 2, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

murong00 commented Apr 7, 2020

murong00 commented Apr 7, 2020

murong00 commented Apr 8, 2020

codelipenghui commented Apr 8, 2020

Choose a reason for hiding this comment

murong00 Apr 9, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

murong00 commented Apr 9, 2020

murong00 commented Apr 9, 2020

murong00 commented Apr 9, 2020

sijie commented Apr 9, 2020

sijie commented Apr 12, 2020

codelipenghui commented Apr 13, 2020

murong00 commented Apr 14, 2020

Anonymitaet commented Jun 10, 2020

murong00 Apr 9, 2020 •

edited

Loading