[Broker] Remove possible error case for system topic name checking #10254

michaeljmarshall · 2021-04-17T23:09:41Z

Motivation

In reading through the system topic code, I noticed that there are some edge cases where normal topics might be considered system topics. This PR should remove that possibility.

I have a few questions on implementation. I will put comments on the lines where I have questions.

Modifications

Updated the isSystemTopic method to just check for presence the system topic prefix (__). This change could be problematic if end users are already using topic names with the system topic prefix.
Use a topic's localName to check for equality instead of endsWith. In the current implementation, a user could have a topic with the local name my__change_events, and because it "ends with" __change_events, it would be considered a system topic.
Update the documentation on topic names to warn about the system topic prefix.

Verifying this change

This change is a trivial rework / code cleanup without any test coverage.

Documentation

System topics were introduced in pulsar 2.6.0. I updated the documentation for all versions since then to indicate that users should avoid creating topics that begin with __. This is important because if there are to be additional system topics, we want to make sure there are not accidentally naming collisions.

michaeljmarshall · 2021-04-17T23:11:03Z

pulsar-broker/src/main/java/org/apache/pulsar/broker/service/BrokerService.java

@@ -2523,7 +2523,7 @@ public boolean isAllowAutoTopicCreation(final String topic) {

    public boolean isAllowAutoTopicCreation(final TopicName topicName) {
        //System topic can always be created automatically
-        if (EventsTopicNames.NAMESPACE_EVENTS_LOCAL_NAME.equals(topicName.getLocalName())
+        if (SystemTopicClient.isSystemTopic(topicName)


TRANSACTION_BUFFER_SNAPSHOT was previously not included in this check. I think it is likely preferred to allow all system topics to be auto created.

michaeljmarshall · 2021-04-17T23:11:53Z

pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentTopic.java

@@ -643,7 +643,8 @@ protected void handleProducerRemoved(Producer producer) {
        }

        try {
-            if (!topic.endsWith(EventsTopicNames.NAMESPACE_EVENTS_LOCAL_NAME)
+            // TODO can all system topics have any type of subscription?
+            if (!TopicName.get(topic).getLocalName().equals(EventsTopicNames.NAMESPACE_EVENTS_LOCAL_NAME)


TRANSACTION_BUFFER_SNAPSHOT was not included in this check. Should it be?

TRANSACTION_BUFFER_SNAPSHOT does not need to include this check

michaeljmarshall · 2021-04-17T23:14:03Z

pulsar-broker/src/main/java/org/apache/pulsar/broker/systopic/SystemTopicClient.java

    static boolean isSystemTopic(TopicName topicName) {
-        return EventsTopicNames.NAMESPACE_EVENTS_LOCAL_NAME.equals(topicName.getLocalName());
+        return topicName.getLocalName().startsWith(EventsTopicNames.SYSTEM_TOPIC_LOCAL_NAME_PREFIX);


The one major consequence of this implementation is the all topics created or existing with __ would be considered system topics. Perhaps that is too much of a breaking change.

I updated it this way because the old implementation ignored the TRANSACTION_BUFFER_SNAPSHOT topic, which is a system topic. It seems easier to maintain if we just follow the paradigm instead of adding every new system topic name as we create new ones.

i don't think __ would be considered system topics is a good idea. It will cause problem when broker upgrade .

That is a great point. It does seem like there is already a bit of a trend where system topics begin with __, but you're right that it would need to be introduced more intentionally than this PR.

I'll look at changing this PR accordingly.

michaeljmarshall · 2021-04-17T23:15:34Z

@codelipenghui and @congbobo184 - would you mind taking a look at this PR? I believe both of you have worked on the different system topics features. Thanks!

congbobo184

good work! left some comment.

congbobo184 · 2021-04-19T13:36:36Z

pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentTopic.java

@@ -643,7 +643,8 @@ protected void handleProducerRemoved(Producer producer) {
        }

        try {
-            if (!topic.endsWith(EventsTopicNames.NAMESPACE_EVENTS_LOCAL_NAME)
+            // TODO can all system topics have any type of subscription?
+            if (!TopicName.get(topic).getLocalName().equals(EventsTopicNames.NAMESPACE_EVENTS_LOCAL_NAME)


TRANSACTION_BUFFER_SNAPSHOT does not need to include this check

congbobo184 · 2021-04-19T13:56:17Z

pulsar-broker/src/main/java/org/apache/pulsar/broker/systopic/SystemTopicClient.java

    static boolean isSystemTopic(TopicName topicName) {
-        return EventsTopicNames.NAMESPACE_EVENTS_LOCAL_NAME.equals(topicName.getLocalName());
+        return topicName.getLocalName().startsWith(EventsTopicNames.SYSTEM_TOPIC_LOCAL_NAME_PREFIX);


i don't think __ would be considered system topics is a good idea. It will cause problem when broker upgrade .

jerrypeng · 2021-04-19T19:05:10Z

Why did the tests now actually run for this PR? There are code changes for this PR as well.

michaeljmarshall · 2021-04-20T00:00:35Z

Why did the tests now actually run for this PR? There are code changes for this PR as well.

Great question. I took a look and didn't see anything obviously wrong. Will look again later tonight (my time).

@lhotari - this PR skipped tests for some reason. The PR includes both documentation and code changes. Tagging you, as I know you're familiar with the logic used to skip tests. Thanks.

sijie · 2021-05-04T03:26:37Z

@michaeljmarshall have you gotten a chance to address the comments?

michaeljmarshall · 2021-05-04T05:57:05Z

@sijie and @congbobo184 - I just added a commit addressing the comments. However, I am wondering if this is the right change. While rebasing, I noticed that #10334 shows that there are possibly more system topics than I realized. When initially writing the PR, I had assumed that the EventsTopicNames class contained all system topics, however, that might not be true. Let me know if this PR needs more changes.

codelipenghui · 2021-05-22T01:12:50Z

Hi @michaeljmarshall, Could you please help confirm if #10529 has fixed the problem? Sorry, sorry we missed your PR.

michaeljmarshall · 2021-06-09T17:10:47Z

Hi @michaeljmarshall, Could you please help confirm if #10529 has fixed the problem? Sorry, sorry we missed your PR.

@codelipenghui - based on a quick look through that PR, I don't think it fixes the problem. My main concern is that checking that a topic's suffix matches a string is not the same as checking that a topic's name matches a string. I would like to see stricter rules for pulsar system topics. Sorry, I haven't had time to come back to this PR to address some of the comments. Hopefully I'll be able to get back to this PR soon.

lhotari · 2021-06-09T17:11:48Z

pulsar-common/src/main/java/org/apache/pulsar/common/events/EventsTopicNames.java

+     * The set of all events topic names.
+     */
+    public static final Set<String> EVENTS_TOPIC_NAMES =
+            new HashSet<>(Arrays.asList(NAMESPACE_EVENTS_LOCAL_NAME, TRANSACTION_BUFFER_SNAPSHOT));


this should be wrapped with Collections.unmodifiableSet to make the Set immutable.

michaeljmarshall · 2021-06-17T18:43:11Z

I'd like to see better definition around system topics. I don't think this PR will achieve that though, so I am going to close this for now.

michaeljmarshall commented Apr 17, 2021

View reviewed changes

michaeljmarshall force-pushed the minor-changes-to-system-topic-internals branch 2 times, most recently from 9302eeb to 11f26ae Compare April 18, 2021 07:28

codelipenghui assigned michaeljmarshall Apr 19, 2021

codelipenghui added this to the 2.8.0 milestone Apr 19, 2021

congbobo184 requested changes Apr 19, 2021

View reviewed changes

michaeljmarshall mentioned this pull request Apr 20, 2021

[GitHub Actions] Fix logic that was incorrectly skipping tests #10276

Merged

[Broker] Remove possible error case for system topic name checking

768c59b

michaeljmarshall force-pushed the minor-changes-to-system-topic-internals branch from 11f26ae to 89a0043 Compare May 4, 2021 05:38

Minimize potential impact of changes based on code review feedback

1e68628

michaeljmarshall force-pushed the minor-changes-to-system-topic-internals branch from 89a0043 to 1e68628 Compare May 4, 2021 05:40

codelipenghui removed this from the 2.8.0 milestone May 22, 2021

michaeljmarshall mentioned this pull request Jun 9, 2021

[Transactions] Problems in case of transactionCoordinatorEnabled=true and allowAutoTopicCreation=false #10871

Closed

lhotari reviewed Jun 9, 2021

View reviewed changes

michaeljmarshall mentioned this pull request Jun 10, 2021

maxTopicsPerNamespace check should exclude system topic. #10850

Merged

michaeljmarshall closed this Jun 17, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Broker] Remove possible error case for system topic name checking #10254

[Broker] Remove possible error case for system topic name checking #10254

michaeljmarshall commented Apr 17, 2021

michaeljmarshall Apr 17, 2021

michaeljmarshall Apr 17, 2021

congbobo184 Apr 19, 2021

michaeljmarshall Apr 17, 2021

congbobo184 Apr 19, 2021 •

edited

Loading

michaeljmarshall Apr 29, 2021

michaeljmarshall commented Apr 17, 2021

congbobo184 left a comment •

edited

Loading

congbobo184 Apr 19, 2021

congbobo184 Apr 19, 2021 •

edited

Loading

jerrypeng commented Apr 19, 2021

michaeljmarshall commented Apr 20, 2021

sijie commented May 4, 2021 •

edited

Loading

michaeljmarshall commented May 4, 2021

codelipenghui commented May 22, 2021

michaeljmarshall commented Jun 9, 2021

lhotari Jun 9, 2021

michaeljmarshall commented Jun 17, 2021

[Broker] Remove possible error case for system topic name checking #10254

[Broker] Remove possible error case for system topic name checking #10254

Conversation

michaeljmarshall commented Apr 17, 2021

Motivation

Modifications

Verifying this change

Documentation

michaeljmarshall Apr 17, 2021

Choose a reason for hiding this comment

michaeljmarshall Apr 17, 2021

Choose a reason for hiding this comment

congbobo184 Apr 19, 2021

Choose a reason for hiding this comment

michaeljmarshall Apr 17, 2021

Choose a reason for hiding this comment

congbobo184 Apr 19, 2021 • edited Loading

Choose a reason for hiding this comment

michaeljmarshall Apr 29, 2021

Choose a reason for hiding this comment

michaeljmarshall commented Apr 17, 2021

congbobo184 left a comment • edited Loading

Choose a reason for hiding this comment

congbobo184 Apr 19, 2021

Choose a reason for hiding this comment

congbobo184 Apr 19, 2021 • edited Loading

Choose a reason for hiding this comment

jerrypeng commented Apr 19, 2021

michaeljmarshall commented Apr 20, 2021

sijie commented May 4, 2021 • edited Loading

michaeljmarshall commented May 4, 2021

codelipenghui commented May 22, 2021

michaeljmarshall commented Jun 9, 2021

lhotari Jun 9, 2021

Choose a reason for hiding this comment

michaeljmarshall commented Jun 17, 2021

congbobo184 Apr 19, 2021 •

edited

Loading

congbobo184 left a comment •

edited

Loading

congbobo184 Apr 19, 2021 •

edited

Loading

sijie commented May 4, 2021 •

edited

Loading