[Transaction] Fix the transaction marker doe not deleted as expect. #11126

congbobo184 · 2021-06-28T04:00:57Z

Motivation

implement

Trigger transaction maker deletes again if there are new delete operations happen during the last delete task processing.

Verifying this change

Add the tests for it

Does this pull request potentially affect one of the following parts:
If yes was chosen, please highlight the changes

Dependencies (does it add or upgrade a dependency): (no)
The public API: (no)
The schema: (no)
The default values of configurations: (no)
The wire protocol: (no)
The rest endpoints: (no)
The admin cli options: (no)
Anything that affects deployment: (no)

eolivelli

Great work,
I left only one comment about shutting down the thread pool

eolivelli · 2021-06-28T06:53:01Z

pulsar-broker/src/main/java/org/apache/pulsar/broker/service/BrokerService.java

@@ -447,6 +448,13 @@ public void start() throws Exception {
            }
        }

+        if (pulsar.getConfiguration().isTransactionCoordinatorEnabled()) {
+            this.transactionBufferDeleteMarkerMonitor = Executors


can you please "shutdown" this Executor service on BrokerService close ?

I also cannot find the shutdown for these other threadpools: topicPublishRateLimiterMonitor, brokerPublishRateLimiterMonitor, deduplicationSnapshotMonitor

good catch! I will follow your advice.

codelipenghui

Looks like the regular task for checking the marker deletion can't fix #11002, if the check task have not triggered yet, we will peek a wrong message right?

I think the right fix is to skip the markers when peeking messages and we don't need a regular checker for the marker deletion, If we have many topics in a broker, the checker will waste more CPU cycles. Or we can say it's an expected behavior(Allow markers that have not been deleted within a period of time)

congbobo184 · 2021-06-28T15:44:08Z

...broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentSubscription.java

@@ -390,16 +391,7 @@ public void acknowledgeMessage(List<Position> positions, AckType ackType, Map<St
            }
        }

-        if (topic.getBrokerService().getPulsar().getConfig().isTransactionCoordinatorEnabled()
-                && this.pendingAckHandle.isTransactionAckPresent()) {


@codelipenghui #11002 cause by this code, because delete marker judge the pending ack state, if one sub don't use pending ack, it will not delete marker. if we don't start a monitor to check this, how we fix this race condition.

eolivelli · 2021-06-29T05:23:23Z

@codelipenghui the peekMessage problem is only a symptom (and it is good to fix it even by skipping messages)
The real problem IMHO is that the list of individually deleted messages will grow without bounds.

codelipenghui · 2021-06-29T05:34:33Z

@eolivelli I have a discussion with @congbobo184 , I think the correct fix is to trigger the marker deletion test properly. The root cause is due to the race condition the marker deletion task has not been triggered due to the previous marker deletion task does not completed, so we need to re-trigger the marker deletion task if there are acks that happen during the last marker deletion task processing.

eolivelli · 2021-06-29T05:45:30Z

@codelipenghui

the correct fix is to trigger the marker deletion test properly
makes sense

So IIUC we are not going to add an additional ThreadPool,
@congbobo184 regarding the shutdown of the executors I will be happy to send the patch
it is better to address the problem in a separate patch

codelipenghui · 2021-06-29T09:03:47Z

@eolivelli

So IIUC we are not going to add an additional ThreadPool,

I think we don't need to add an additional thread pool, we just need to add a check when we complete the last delete marker task.

eolivelli

great

I left one comment
PTAL

eolivelli · 2021-07-01T11:09:10Z

pulsar-broker/src/test/java/org/apache/pulsar/broker/service/TransactionMarkerDeleteTest.java

@@ -77,7 +81,7 @@ public void testTransactionMarkerDelete() throws Exception {
                managedLedger.openCursor("test"), false);

        Position position1 = managedLedger.addEntry("test".getBytes());
-        managedLedger.addEntry(Markers
+        Position position2 = managedLedger.addEntry(Markers


can we simulate a sequence with multiple transactions markers:

send

commit

send

commit
...
and verify that there is no hole ?

also

send (with tx)

commit

commit

commit

send (no tx)
....

congbobo184 · 2021-07-02T11:31:45Z

@eolivelli add some test like your comment. please review again! thanks.

eolivelli

looks good to me

@codelipenghui
you left some comments, please take a look again

codelipenghui · 2021-07-15T12:33:44Z

@congbobo184 Could you please help update the description of the PR? And add more context about how the root cause is and how the PR can fix the issue, this will make others can understand it easily.

eolivelli · 2021-07-16T10:22:08Z

@codelipenghui can you cherry pick to 2.8.1 ?

…11126) Trigger transaction maker deletes again if there are new delete operations happen during the last delete task processing. (cherry picked from commit 3fcfb22)

…pache#11126) Trigger transaction maker deletes again if there are new delete operations happen during the last delete task processing. (cherry picked from commit 3fcfb22)

…pache#11126) Trigger transaction maker deletes again if there are new delete operations happen during the last delete task processing.

[Transaction] Transaction delete marker monitor.

c802a2c

congbobo184 requested review from codelipenghui, eolivelli and gaoran10 June 28, 2021 04:01

fix transaction delete marker

bfa837d

eolivelli requested changes Jun 28, 2021

View reviewed changes

codelipenghui reviewed Jun 28, 2021

View reviewed changes

codelipenghui assigned congbobo184 Jun 28, 2021

codelipenghui added this to the 2.9.0 milestone Jun 28, 2021

codelipenghui added the release/2.8.1 label Jun 28, 2021

congbobo184 commented Jun 28, 2021

View reviewed changes

congbo added 3 commits July 1, 2021 12:59

Add delete marker state wait

71ed2ed

fix code style

5dd6f87

Fix code style

fccb332

eolivelli reviewed Jul 1, 2021

View reviewed changes

Fix some comment

b15aaae

eolivelli approved these changes Jul 2, 2021

View reviewed changes

codelipenghui approved these changes Jul 15, 2021

View reviewed changes

codelipenghui changed the title ~~[Transaction] Transaction delete marker monitor.~~ [Transaction] Fix the transaction marker doe not deleted as expect. Jul 16, 2021

codelipenghui merged commit 3fcfb22 into apache:master Jul 16, 2021

codelipenghui added the area/transaction label Jul 16, 2021

codelipenghui added the cherry-picked/branch-2.8 Archived: 2.8 is end of life label Jul 23, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Transaction] Fix the transaction marker doe not deleted as expect. #11126

[Transaction] Fix the transaction marker doe not deleted as expect. #11126

congbobo184 commented Jun 28, 2021 •

edited by codelipenghui

Loading

eolivelli left a comment

eolivelli Jun 28, 2021

congbobo184 Jun 28, 2021

codelipenghui left a comment

congbobo184 Jun 28, 2021 •

edited

Loading

eolivelli commented Jun 29, 2021

codelipenghui commented Jun 29, 2021 •

edited

Loading

eolivelli commented Jun 29, 2021

codelipenghui commented Jun 29, 2021

eolivelli left a comment

eolivelli Jul 1, 2021

congbobo184 commented Jul 2, 2021

eolivelli left a comment

codelipenghui commented Jul 15, 2021

eolivelli commented Jul 16, 2021

[Transaction] Fix the transaction marker doe not deleted as expect. #11126

[Transaction] Fix the transaction marker doe not deleted as expect. #11126

Conversation

congbobo184 commented Jun 28, 2021 • edited by codelipenghui Loading

Motivation

implement

Verifying this change

eolivelli left a comment

Choose a reason for hiding this comment

eolivelli Jun 28, 2021

Choose a reason for hiding this comment

congbobo184 Jun 28, 2021

Choose a reason for hiding this comment

codelipenghui left a comment

Choose a reason for hiding this comment

congbobo184 Jun 28, 2021 • edited Loading

Choose a reason for hiding this comment

eolivelli commented Jun 29, 2021

codelipenghui commented Jun 29, 2021 • edited Loading

eolivelli commented Jun 29, 2021

codelipenghui commented Jun 29, 2021

eolivelli left a comment

Choose a reason for hiding this comment

eolivelli Jul 1, 2021

Choose a reason for hiding this comment

congbobo184 commented Jul 2, 2021

eolivelli left a comment

Choose a reason for hiding this comment

codelipenghui commented Jul 15, 2021

eolivelli commented Jul 16, 2021

congbobo184 commented Jun 28, 2021 •

edited by codelipenghui

Loading

congbobo184 Jun 28, 2021 •

edited

Loading

codelipenghui commented Jun 29, 2021 •

edited

Loading