PIP-124: Create init subscription before sending message to DLQ

[RobertIndie]

Motivation

If we enable the DLQ when consuming messages. For some messages that can't be processed successfully, the messages will be moved to the DLQ, but if we do not specify the data retention for the namespace or create a subscription for the DLQ to retain the data, the data of the DLQ will be removed automatically. Therefore, we need to create the initial subscription before sending messages to the DLQ.

Goal

Users can set the initial subscription name in the DeadLetterPolicy. The consumer will create the initial subscription before sending messages to the DLQ. At this point, subsequent messages produced to the DLQ are not automatically deleted unexpectedly. If allowAutoSubscriptionCreation in broker.conf is false, the initial subscription won't be created automatically. Otherwise, it will confuse the user. Users can explicitly create that subscription to handle this case.

This PIP needs to be compatible with the previous behavior. The initial subscription name in the DeadLetterPolicy is optional. The default behavior is not to create the initial subscription which is consistent with the original behavior.

API Changes

• Add initSubscriptionName to the DeadLetterPolicy

/**
 * Name of the initial subscription name of the dead letter topic.
 * If this field is not set, the initial subscription will not be created.
 */
private String initSubscriptionName;

• Add a new field to the CommandProducer. The broker will use this field to create the initial subscription.

optional string initial_subscription_name

• Add a new config to the Producer Configuration. This allows the consumer to specify the initial subscription when creating the deadLetterProducer.

/**
 * Use this config to automatically create an initial subscription when creating the topic.
 * If this field is not set, the initial subscription will not be created.
 *
 * @param initialSubscriptionName
 *              Name of the initial subscription of the topic.
 * @return the producer builder instance
 */
ProducerBuilder<T> initialSubscriptionName(String initialSubscriptionName);

Implementation

When the deadLetterProducer is initialized, the consumer will set the initial subscription of the deadLetterProducer according to the DeadLetterPolicy.

When the broker creates a producer(after receiving the CommandProducer), if it finds that the topic does not exist, it will not only create the topic(if the topic automatically creation is enabled) but also create the initial subscription(if the initial subscription name is set). When creating an initial subscription, the user needs to have the canConsume permission and the allowAutoSubscriptionCreation needs to be enabled.

Reject Alternatives

Create the initial subscription using the consumer

Before the deadLetterProducer is initialized, the consumer first tries to create a deadLetterConsumer using the initial subscription name in the DeadLetterPolicy. When this subscription already exists, the ConsumerBusy exception will occur. In this case, we can ignore that exception and create the deadLetterProducer.

This is the original solution for this PIP. But this introduces too much workload for the client. In the subsequent discussion, we try to add a new command CommandCreateSubscription to avoid creating and closing the deadLetterConsumer, but again, this introduces workload as well as greater complexity.

Use the retention policy to retent the data of the DLQ

Before creating the deadLetterProducer, an admin can create a retention policy for the topic or namespace. Then, consumers of the topic have the duration of the retention policy to discover the topic and create a subscription before messages are lost.

But users are not easy to set a different data retention policy or create a new subscription for a lazy created DLQ topic.

Add a new policy to retain data when the topic has no subscriptions

Adding a new policy for pulsar topics: a namespace or a topic level policy that makes it possible to retain messages indefinitely when a topic has no subscriptions.

For the auto-created topic, the subscription can only be determined at the time of creation. It may or may not create. If users are able to determine which consumers are, and these consumers need to receive any message sent by the producer, they should create the topic and subscription manually or use the consumer to trigger the topic auto-creation, not the producer.

Add the initial subscription name field to the metadata of the CommandProducer

Use producer metadata to carry the init subscription name so that we don't need to introduce a new field in CommandProducer, and the new field looks a little confusing.

While this does not change protobuf, it still implicitly changes the protocol. This solution has different advantages and disadvantages than the current one. Finally, we decided to go with the current solution.

In addition, there is a discussion about whether we need to create the initial subscription on the retryLetterTopic. The consumer has already subscribed to the retryLetterTopic and created the subscription before creating the retryLetterProducer, so we don't need to create the initial subscription for it.

Prototype implementation PR: #13355

[eolivelli]

@RobertIndie the PIP id is duplicated
I am sorry for the confusion

this is the official list of PIPs
https://github.com/apache/pulsar/wiki

I can add your PIP to the list and assign you a number

[eolivelli]

I assigned PIP-137.
sorry for the confusion

[eolivelli]

https://github.com/apache/pulsar/wiki

[RobertIndie]

This PIP was proposed before #13490.
I think we should use pulsar issues to manage the PIP id(or a better way to automate it), not the wiki. Not all contributors can add their own PIP to the wiki.
This PIP has been hotly discussed in the mailing list, and changing the number of this PIP will cause a lot of inconveniences.

Can you change your PIP id in your PIP?

[eolivelli]

@RobertIndie I am sorry for the confusion, we already discussed about this problem about assigning PIP IDs on dev@pulsar.apache.org and we decided to go with the Wiki.
The only source of truth we can have is the Wiki as it is the central place where we list PIPs, it already happened many times that people just created the issues on github and no one noticed it.

I will change my PIP id, no problem.

For the next time please I suggesting you to ask for a PIP ID on the dev@ mailing list.

[RobertIndie]

For the next time please I suggesting you to ask for a PIP ID on the dev@ mailing list.

@eolivelli Thanks for your suggestion.

But I think I was following the process for PIP proposing: https://github.com/apache/pulsar/blob/master/.github/ISSUE_TEMPLATE/pip.md

2. The author(s) will send a note to the dev@pulsar.apache.org mailing list
   to start the discussion, using subject prefix [PIP] xxx. To determine the appropriate PIP
   number xxx, inspect the mailing list (https://lists.apache.org/list.html?dev@pulsar.apache.org)
   for the most recent PIP. Add 1 to that PIP's number to get your PIP's number.

I have opened the corresponding discussion on the mailing list and self assigned the PIP number as soon as I created this issue. Did I misunderstand it? If there is something wrong with this process, I think we need to update it.