-
Notifications
You must be signed in to change notification settings - Fork 3.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
PIP-124: Create init subscription before sending message to DLQ #13408
Comments
@RobertIndie the PIP id is duplicated this is the official list of PIPs I can add your PIP to the list and assign you a number |
I assigned PIP-137. |
This PIP was proposed before #13490. Can you change your PIP id in your PIP? |
@RobertIndie I am sorry for the confusion, we already discussed about this problem about assigning PIP IDs on dev@pulsar.apache.org and we decided to go with the Wiki. I will change my PIP id, no problem. For the next time please I suggesting you to ask for a PIP ID on the dev@ mailing list. |
@eolivelli Thanks for your suggestion. But I think I was following the process for PIP proposing: https://github.com/apache/pulsar/blob/master/.github/ISSUE_TEMPLATE/pip.md
I have opened the corresponding discussion on the mailing list and self assigned the PIP number as soon as I created this issue. Did I misunderstand it? If there is something wrong with this process, I think we need to update it. |
…#14458) Master Issue: #13408 ### Motivation In #13355, we added support for creating initial subscription when creating the producer. But the broker didn't check if the subscription can be created automatically. The initial subscription will be created even if the `allowAutoSubscriptionCreation` is disabled.
…#14458) Master Issue: #13408 ### Motivation In #13355, we added support for creating initial subscription when creating the producer. But the broker didn't check if the subscription can be created automatically. The initial subscription will be created even if the `allowAutoSubscriptionCreation` is disabled. (cherry picked from commit db25438)
…apache#14458) Master Issue: apache#13408 ### Motivation In apache#13355, we added support for creating initial subscription when creating the producer. But the broker didn't check if the subscription can be created automatically. The initial subscription will be created even if the `allowAutoSubscriptionCreation` is disabled.
Motivation
If we enable the DLQ when consuming messages. For some messages that can't be processed successfully, the messages will be moved to the DLQ, but if we do not specify the data retention for the namespace or create a subscription for the DLQ to retain the data, the data of the DLQ will be removed automatically. Therefore, we need to create the initial subscription before sending messages to the DLQ.
Goal
Users can set the initial subscription name in the DeadLetterPolicy. The consumer will create the initial subscription before sending messages to the DLQ. At this point, subsequent messages produced to the DLQ are not automatically deleted unexpectedly. If
allowAutoSubscriptionCreation
inbroker.conf
is false, the initial subscription won't be created automatically. Otherwise, it will confuse the user. Users can explicitly create that subscription to handle this case.This PIP needs to be compatible with the previous behavior. The initial subscription name in the DeadLetterPolicy is optional. The default behavior is not to create the initial subscription which is consistent with the original behavior.
API Changes
initSubscriptionName
to theDeadLetterPolicy
CommandProducer
. The broker will use this field to create the initial subscription.Implementation
When the deadLetterProducer is initialized, the consumer will set the initial subscription of the deadLetterProducer according to the DeadLetterPolicy.
When the broker creates a producer(after receiving the CommandProducer), if it finds that the topic does not exist, it will not only create the topic(if the topic automatically creation is enabled) but also create the initial subscription(if the initial subscription name is set). When creating an initial subscription, the user needs to have the
canConsume
permission and theallowAutoSubscriptionCreation
needs to be enabled.Reject Alternatives
Create the initial subscription using the consumer
Before the deadLetterProducer is initialized, the consumer first tries to create a deadLetterConsumer using the initial subscription name in the DeadLetterPolicy. When this subscription already exists, the ConsumerBusy exception will occur. In this case, we can ignore that exception and create the deadLetterProducer.
This is the original solution for this PIP. But this introduces too much workload for the client. In the subsequent discussion, we try to add a new command
CommandCreateSubscription
to avoid creating and closing the deadLetterConsumer, but again, this introduces workload as well as greater complexity.Use the retention policy to retent the data of the DLQ
Before creating the deadLetterProducer, an admin can create a retention policy for the topic or namespace. Then, consumers of the topic have the duration of the retention policy to discover the topic and create a subscription before messages are lost.
But users are not easy to set a different data retention policy or create a new subscription for a lazy created DLQ topic.
Add a new policy to retain data when the topic has no subscriptions
Adding a new policy for pulsar topics: a namespace or a topic level policy that makes it possible to retain messages indefinitely when a topic has no subscriptions.
For the auto-created topic, the subscription can only be determined at the time of creation. It may or may not create. If users are able to determine which consumers are, and these consumers need to receive any message sent by the producer, they should create the topic and subscription manually or use the consumer to trigger the topic auto-creation, not the producer.
Add the initial subscription name field to the metadata of the CommandProducer
Use producer metadata to carry the init subscription name so that we don't need to introduce a new field in CommandProducer, and the new field looks a little confusing.
While this does not change protobuf, it still implicitly changes the protocol. This solution has different advantages and disadvantages than the current one. Finally, we decided to go with the current solution.
In addition, there is a discussion about whether we need to create the initial subscription on the retryLetterTopic. The consumer has already subscribed to the retryLetterTopic and created the subscription before creating the retryLetterProducer, so we don't need to create the initial subscription for it.
Prototype implementation PR: #13355
The text was updated successfully, but these errors were encountered: