Pending ack set managed ledger config true #11494

liangyepianzhou · 2021-07-29T07:00:58Z

Modivation

Fix the Issue of #11481
In standalone mode, pulsar 2.8.0 cannot be used normally when the transaction is started

CauseBy

getTopicwas executed twice when FunctionWorkService .
getTopicIfExistsmake createIfMissing = false When the execution ends.
PersistentSubscription will create a ledger for the subscription when transaction was turned on.
new MetadataNotFoundException("Managed ledger not found")was thrown when calling MetaStoreImpl::getManagedLedgerInfo

implement

Create a separate ManagerLedgerConfig for PendingAck

verify

Add testSubscriptionRecreateTopic in TransactionTest

Documentation

no-need-Documentation
This is just a bug fix

eolivelli · 2021-07-29T07:50:03Z

pulsar-broker/src/test/java/org/apache/pulsar/broker/transaction/TransactionTest.java

+        admin.topics().unload(topicName);
+        try {
+            admin.topics().createNonPartitionedTopic(topicName);
+        } catch (Exception e){


you can use catch (PulsarAdminException.ConflictException) and let other exceptions be not caught

pulsar-broker/src/test/java/org/apache/pulsar/broker/transaction/TransactionTest.java

eolivelli · 2021-07-29T07:51:23Z

...ain/java/org/apache/pulsar/broker/transaction/pendingack/impl/MLPendingAckStoreProvider.java

@@ -51,7 +52,8 @@
        PersistentTopic originPersistentTopic = (PersistentTopic) subscription.getTopic();
        String pendingAckTopicName = MLPendingAckStore
                .getTransactionPendingAckStoreSuffix(originPersistentTopic.getName(), subscription.getName());
-
+        ManagedLedgerConfig managedLedgerConfig = originPersistentTopic.getManagedLedger().getConfig();
+        managedLedgerConfig.setCreateIfMissing(true);


this is not good, we cannot alter the configuration of another component this way.

if "createIfMissing" is required we have to set it upstream

Thank you for pointing out.
I made some changes, please take a look again

eolivelli · 2021-07-29T08:57:55Z

...broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentSubscription.java

@@ -150,6 +151,8 @@ public PersistentSubscription(PersistentTopic topic, String subscriptionName, Ma
        this.setReplicated(replicated);
        if (topic.getBrokerService().getPulsar().getConfig().isTransactionCoordinatorEnabled()
                && !checkTopicIsEventsNames(TopicName.get(topicName))) {
+            ManagedLedgerConfig config = this.topic.getManagedLedger().getConfig();
+            config.setCreateIfMissing(true);


this is still not good.
you have to apply this setting elsewhere

maybe @congbobo184 or @codelipenghui can help more

Thank you very much for your advice. After discussing with Congbo, the changes have been made. Could you please take a look again?

codelipenghui · 2021-07-29T17:05:29Z

@liangyepianzhou Could you please provide the root cause of #11481? before fixing the issue, we'd better point out where the problem is so that the reviewers can get more context to provide more suggestions.

…ction was needed

liangyepianzhou · 2021-07-31T02:55:02Z

Thank you for pointing it @codelipenghui
I have given some context above.
If there is a problem, hope you can point it out,
it would be better if you can provide some help.

liangyepianzhou · 2021-08-03T00:20:58Z

/pulsarbot run-failure-checks

liangyepianzhou · 2021-08-03T00:56:37Z

/pulsarbot run-failure-checks

gaoran10 · 2021-08-03T03:08:21Z

pulsar-broker/src/main/java/org/apache/pulsar/broker/service/BrokerService.java

@@ -1232,6 +1232,7 @@ public void openLedgerComplete(ManagedLedger ledger, Object ctx) {
                                    preCreateSubForCompaction = ((SystemTopic) persistentTopic)
                                            .preCreateSubForCompactionIfNeeded();
                                }
+                                managedLedgerConfig.setCreateIfMissing(true);


Maybe set createIfMissing as true directly is not reasonable, this will break the service configuration?

This may depend on the original design intention of getTopicIfExists. Calling getTopicIfExists will directly set createIfMissing to false, which will affect the configuration. I don't know if this is intentional or ignored.

However, I did make a mistake and did not consider its initial configuration. I think I should keep the configuration first, and then restore the configuration

Thank you for pointing out.
Can you see if there are any problems now?

Sorry, Ran. The above is my ill-conceived.
I have another discussion with congbo.
After this topic has been created, the field will have no meaning. And each subscription of transaction needs createIfMsiing=true to create PendingAck. So it is also feasible to set it to true directly in Completely

Anonymitaet · 2021-08-03T04:30:45Z

Thanks for your contribution. For this PR, do we need to update docs?

(The PR template contains info about doc, which helps others know more about the changes. Can you provide doc-related info in this and future PR descriptions? Thanks)

liangyepianzhou · 2021-08-03T04:41:27Z

Thanks for your contribution. For this PR, do we need to update docs?

(The PR template contains info about doc, which helps others know more about the changes. Can you provide doc-related info in this and future PR descriptions? Thanks)

Thank you for your reminder, I will indicate it in the comment in the future.
This is just a bug fix. No documentation required

liangyepianzhou · 2021-08-03T06:27:48Z

/pulsarbot run-failure-checks

congbobo184

LGTM! left some comment. Could you please add more details? It might be better to give steps to reproduce

congbobo184 · 2021-08-04T12:23:32Z

pulsar-broker/src/test/java/org/apache/pulsar/broker/transaction/TransactionTest.java

+    ConcurrentHashMap<String, CompletableFuture<ManagedLedgerImpl>> ledgers =
+            (ConcurrentHashMap<String, CompletableFuture<ManagedLedgerImpl>>)field.get(managedLedgerFactory);
+    ledgers.remove(TopicName.get(topic).getPersistenceNamingEncoding());
+    admin.topics().unload(topic);


dont need unload.

congbobo184 · 2021-08-04T12:23:55Z

pulsar-broker/src/test/java/org/apache/pulsar/broker/transaction/TransactionTest.java

+        admin.topics().createNonPartitionedTopic(topic);
+        Assert.fail();
+    } catch (PulsarAdminException.ConflictException e){
+        log.info("Cann`t create topic again");


should Assert this exception

hangc0276 · 2021-08-06T08:14:07Z

@liangyepianzhou I am cutting 2.8.1, would you please address @codelipenghui @congbobo184 's comments?

liangyepianzhou · 2021-08-06T08:26:12Z

@liangyepianzhou I am cutting 2.8.1, would you please address @codelipenghui @congbobo184 's comments?

I am waiting for review from @codelipenghui @congbobo184

hangc0276 · 2021-08-06T08:35:28Z

/pulsarbot run-failure-checks

hangc0276 · 2021-08-06T08:47:18Z

@liangyepianzhou I am cutting 2.8.1, would you please address @codelipenghui @congbobo184 's comments?

I am waiting for review from @codelipenghui @congbobo184

OK，@codelipenghui @congbobo184 Please help review this PR, thanks.

…onfig used in Creating newPendingAckStore

…gedLedgerConfig_true

…gAck configuration to be lost

eolivelli · 2021-08-10T06:29:37Z

...ain/java/org/apache/pulsar/broker/transaction/pendingack/impl/MLPendingAckStoreProvider.java

+                                                        pendingAckStoreFuture
+                                                                .complete(new MLPendingAckStore(ledger, cursor,
+                                                                        subscription.getCursor()));
+                                                        log.info("{},{} open MLPendingAckStore cursor success",


probably this logger should be moved to level debug

eolivelli

Lgtm

congbobo184

LGTM! good work.

## Modivation Fix the Issue of #11481 In standalone mode, pulsar 2.8.0 cannot be used normally when the transaction is started ## CauseBy ```getTopic```was executed twice when FunctionWorkService . ```getTopicIfExists```make ```createIfMissing = false``` When the execution ends. ```PersistentSubscription``` will create a ledger for the subscription when transaction was turned on. ```new MetadataNotFoundException("Managed ledger not found")```was thrown when calling ```MetaStoreImpl::getManagedLedgerInfo``` ## implement Create a separate ManagerLedgerConfig for PendingAck ## verify Add testSubscriptionRecreateTopic in TransactionTest (cherry picked from commit daf457d)

## Modivation Fix the Issue of apache#11481 In standalone mode, pulsar 2.8.0 cannot be used normally when the transaction is started ## CauseBy ```getTopic```was executed twice when FunctionWorkService . ```getTopicIfExists```make ```createIfMissing = false``` When the execution ends. ```PersistentSubscription``` will create a ledger for the subscription when transaction was turned on. ```new MetadataNotFoundException("Managed ledger not found")```was thrown when calling ```MetaStoreImpl::getManagedLedgerInfo``` ## implement Create a separate ManagerLedgerConfig for PendingAck ## verify Add testSubscriptionRecreateTopic in TransactionTest

eolivelli requested changes Jul 29, 2021

View reviewed changes

codelipenghui assigned liangyepianzhou Jul 29, 2021

codelipenghui added area/transaction type/bug The PR fixed a bug or issue reported a bug labels Jul 29, 2021

codelipenghui added this to the 2.9.0 milestone Jul 29, 2021

codelipenghui added the release/2.8.1 label Jul 29, 2021

liangyepianzhou closed this Jul 30, 2021

liangyepianzhou added 2 commits July 31, 2021 10:41

Give permission to open ledger for PersistentSubscription when transa…

4409dcc

…ction was needed

Add test to verify the change

57a6fb3

liangyepianzhou reopened this Jul 31, 2021

liangyepianzhou requested a review from eolivelli August 1, 2021 04:20

gaoran10 reviewed Aug 3, 2021

View reviewed changes

liangyepianzhou requested a review from gaoran10 August 3, 2021 03:41

Anonymitaet added the doc-not-needed Your PR changes do not impact docs label Aug 3, 2021

liangyepianzhou mentioned this pull request Aug 3, 2021

Donot create transaction components for function work topic #11543

Closed

congbobo184 reviewed Aug 4, 2021

View reviewed changes

liangyepianzhou added 2 commits August 6, 2021 18:38

use BrokerService::getManagedLedgerConfig to get a new ManagerLedgerC…

8271c15

…onfig used in Creating newPendingAckStore

checkstyle

491888d

liangyepianzhou added 14 commits August 9, 2021 21:58

PendingAck uses singleton mode to create config

eef34ec

Merge remote-tracking branch 'apache/master' into PendingAck_Set_Mana…

c76700c

…gedLedgerConfig_true

Distinguish topic and ManagerLedger's Config

1ff6c23

checkstyle

3d65ee6

Distinguish topic and ManagerLedger's Config

c55a3a0

Distinguish topic and ManagerLedger's Config

1f6572e

checkstyle

e041319

checkstyle

b0b6eee

modify log

1b9cc8e

Optimize the test to ensure that the broker will not cause the Pendin…

ee2a339

…gAck configuration to be lost

checkStyle

9dc6bcd

licensed changed fixed

6b705dd

modify exceptionally

7f2f388

modify exceptionally

0ec52f3

eolivelli reviewed Aug 10, 2021

View reviewed changes

liangyepianzhou added 2 commits August 10, 2021 14:38

modify TransactionTest

7fcfdc7

modify log.debug

fd5fb75

eolivelli approved these changes Aug 10, 2021

View reviewed changes

congbobo184 approved these changes Aug 10, 2021

View reviewed changes

congbobo184 requested a review from codelipenghui August 10, 2021 10:00

codelipenghui approved these changes Aug 10, 2021

View reviewed changes

codelipenghui merged commit daf457d into apache:master Aug 10, 2021

hangc0276 added the cherry-picked/branch-2.8 Archived: 2.8 is end of life label Aug 12, 2021

tisonkun mentioned this pull request Nov 10, 2022

standalone模式下，开启事务时，pulsar2.8.0不能正常使用。急急急！ #11481

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Pending ack set managed ledger config true #11494

Pending ack set managed ledger config true #11494

liangyepianzhou commented Jul 29, 2021 •

edited

Loading

eolivelli Jul 29, 2021

eolivelli Jul 29, 2021

liangyepianzhou Jul 29, 2021

eolivelli Jul 29, 2021

liangyepianzhou Aug 3, 2021 •

edited

Loading

codelipenghui commented Jul 29, 2021

liangyepianzhou commented Jul 31, 2021

liangyepianzhou commented Aug 3, 2021

liangyepianzhou commented Aug 3, 2021

gaoran10 Aug 3, 2021

liangyepianzhou Aug 3, 2021

liangyepianzhou Aug 3, 2021

liangyepianzhou Aug 3, 2021

liangyepianzhou Aug 3, 2021 •

edited

Loading

Anonymitaet commented Aug 3, 2021

liangyepianzhou commented Aug 3, 2021

liangyepianzhou commented Aug 3, 2021

congbobo184 left a comment

congbobo184 Aug 4, 2021

congbobo184 Aug 4, 2021

hangc0276 commented Aug 6, 2021

liangyepianzhou commented Aug 6, 2021

hangc0276 commented Aug 6, 2021

hangc0276 commented Aug 6, 2021

eolivelli Aug 10, 2021

eolivelli left a comment

congbobo184 left a comment

Pending ack set managed ledger config true #11494

Pending ack set managed ledger config true #11494

Conversation

liangyepianzhou commented Jul 29, 2021 • edited Loading

Modivation

CauseBy

implement

verify

Documentation

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

liangyepianzhou Aug 3, 2021 • edited Loading

Choose a reason for hiding this comment

codelipenghui commented Jul 29, 2021

liangyepianzhou commented Jul 31, 2021

liangyepianzhou commented Aug 3, 2021

liangyepianzhou commented Aug 3, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

liangyepianzhou Aug 3, 2021 • edited Loading

Choose a reason for hiding this comment

Anonymitaet commented Aug 3, 2021

liangyepianzhou commented Aug 3, 2021

liangyepianzhou commented Aug 3, 2021

congbobo184 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hangc0276 commented Aug 6, 2021

liangyepianzhou commented Aug 6, 2021

hangc0276 commented Aug 6, 2021

hangc0276 commented Aug 6, 2021

Choose a reason for hiding this comment

eolivelli left a comment

Choose a reason for hiding this comment

congbobo184 left a comment

Choose a reason for hiding this comment

liangyepianzhou commented Jul 29, 2021 •

edited

Loading

liangyepianzhou Aug 3, 2021 •

edited

Loading

liangyepianzhou Aug 3, 2021 •

edited

Loading