Polices On Topic

[codelipenghui]

This is the master ticket for tracking all the tasks for polices on topic

• 1. MessageTTL [Get, Set, Remove] Support set message TTL on topic level. #7738
• 2. Retention Policy [Get, Set, Remove] [Issue 2688]Support set retention on topic level. #7747
• 3. Message Dispatch Rate [Get, Set, Remove] support message dispatch rate on topic level #7863
• 4. Backlog Quota [Get, Set, Remove] [Issue 2689] Support set backlog quota on topic level. #7646
• 5. Persistence Policy [Get, Set, Remove] [ISSUE 7757] Support Persistence Policies on topic level #7817
• 6. Max Producer [Get, Set, Remove] [ISSUE 7758] Support set Max Producer on topic level. #7914
• 7. Max Consumer [Get, Set, Remove] [Issue 7759] Support set Max Consumer on topic level. #7968
• 8. Compaction Threshold [Get, Set, Remove] add compaction threshold for topic level #7881
• 9. Deduplication [Get, Set, Remove] Support deduplication on topic level #7821
• 10. Offload Policy [Get, Set, Remove] Support topicl level offload #7883
• 11. MaxUnackMessagesPerConsumer [Get, Set, Remove] Support MaxUnackedMessagesOnConsumer on topic level #7818
• 12. MaxUnackMessagesPerSubscription [Get, Set, Remove] Support set MaxUnackMessagesPerSubscription on topic level #7802
• 13. DelayedDeliveryPolicies [Get, Set, Remove] support topic level delayed delivery policy #7784
• 14. Inactive topic policy Support topic-level inactiveTopicPolicies #7986

[ivankelly]

@codelipenghui

This change will cause more load on zookeeper. For each topic loaded, we would have to do a read for the policy of that topic when loading the topic policy, even if no topic policy exists.

How does the broker get notified of updated to the policy for a topic? If we add a watcher, that's a watcher per topic. We should consider if there's another way of doing this without placing more load on ZK, such as in a 'system' topic.

[codelipenghui]

@ivankelly

This change will cause more load on zookeeper. For each topic loaded, we would have to do a read for the policy of that topic when loading the topic policy, even if no topic policy exists.

How does the broker get notified of updated to the policy for a topic? If we add a watcher, that's a watcher per topic. We should consider if there's another way of doing this without placing more load on ZK, such as in a 'system' topic.

Yes, this is a very necessary concern. I think 'system' topic is an excellent solution.

1. For policy changed, send message to the 'system' topic and write to zookeeper, then 'system' topic consumer update policy cache in broker. So that just need to read the cache.
2. For topic owner changed, load the namespace policy and topic policy into broker cache.

For 'system' topic design. I'm not sure the 'system' topic border.

1. A 'system' topic in a cluster
2. Namespace 'system' topic / Topic 'system' topic
3. 'system' topic for each namespace / 'system' topic for each topic

I'm prone to 1 or 2, for 3 too much topic will be created.

@ivankelly Can you give me some idea more ? Thanks

[ivankelly]

I would put it at the namespace level, so 2. It would be good to have a generic model for system topics, as currently it would require you wiring up something yourself. Maybe it would be better to put it at the managed ledger layer also, since if we put it at the pulsar level we would have a circular dependency of sorts.

[sijie]

@ivankelly I am a bit confused about using managed ledger. managed ledger doesn't have tailing facility, how do you propagate the policy changes using managed ledger?

[ivankelly]

You shouldn't need to propagate. When you update the policy, you're hitting the broker that owns the topic.

[sijie]

@ivankelly

How does the broker get notified of updated to the policy for a topic? If we add a watcher, that's a watcher per topic.

You shouldn't need to propagate. When you update the policy, you're hitting the broker that owns the topic.

I am lost now. Isn't the existence of system topic to address the concern of zookeeper notifications? so we convert the zookeeper notifications, to consume from system topic. penghui's previous comment was using system topic, which is making sense, since pulsar already has producer and consumer. but in your last comment, you suggest using mledger to avoid circular dependency, then my question is "how is possible that the other brokers get notified with policy changes, because mledger doesn't have tailing facility".

[ivankelly]

Isn't the existence of system topic to address the concern of zookeeper notifications

From what I can tell, system topics don't actually exist as an actual concept in pulsar, they're just normal topics. But as a conceptual concept, they don't exist just to replace notifications. They also exist as a place to store metadata which can scale.

Ok, so, to recap where the suggestion to use managed ledger came from...

The first suggestion was to put topic policies in zookeeper. This was discarded because of zookeeper scale issues.

So then it was suggested to use a system topic in pulsar. I'm questioning whether we should use a topic for policies on topics. What happens if the topic that contains the policy so has a topic policy? Maybe we need to prevent system topics from having topic level policies.

Otherwise, we put the "system topics" a lower layer and we wouldn't even have this problem. I suggested managed ledger, but dlog would also work here if we want notifications.

But stepping back, what I really want is that we define what system topics are and make them a reusable concept within the system.

[sijie]

But as a conceptual concept, they don't exist just to replace notifications. They also exist as a place to store metadata which can scale.

okay. if system topic is to store metadata, isn't it overlapped with the goal of building table service for metadata storage. if you just use a pure log to store metadata, we still have too much to build and most of the work is overlapped with the goal of table service.

so I think we should first target system topic as a changelog of what's happening within namespace, rather than making it to replace zookeeper to store metadata.

So then it was suggested to use a system topic in pulsar. I'm questioning whether we should use a topic for policies on topics. What happens if the topic that contains the policy so has a topic policy? Maybe we need to prevent system topics from having topic level policies.

good point. I see it now.

Otherwise, we put the "system topics" a lower layer and we wouldn't even have this problem. I suggested managed ledger, but dlog would also work here if we want notifications.

I think we should just consolidate managed ledger and dlog into one library and use it across the place. The storage layer already provide tailing as notifications, which pulsar should leverage for "system topics" here, maybe "readonly broker" in future, and even the sql segment reader optimization.

[ivankelly]

I think we should just consolidate managed ledger and dlog into one library and use it across the place.

+1 to this.

[sijie]

@codelipenghui I moved this task to 2.4.0 since it seems that we need address system topic first.

[codelipenghui]

@sijie @merlimat Are you already working on system topic now? I want to add dispatch rate limitation for a topic, it seems depends on system topic.

[sijie]

@codelipenghui @merlimat @ivankelly

After talking to a lot of users, users would like to have the ability to override the policy at namespace level. Since Pulsar doesn't support this feature, it forces the users to create many namespaces just for the ability to configure the policy for different topics.

I think we should just go ahead to implement the topic level policy without waiting for the system topic. If the concern is about zk access, we can think of reducing the zk access or having a flag to control whether to enable this feature or not and give warnings to people to not over-use this feature.

Thoughts?

[codelipenghui]

All topic policies are ready for 2.7.0. Thanks to all contributors work together to complete the great feature.