[ISSUE #6570] Optimizing the process of putting messages to improve performance #6571

RongtongJin · 2023-04-11T08:24:56Z

Make sure set the target branch to develop

What is the purpose of the change

Brief changelog

We will only increase the queueOffset if the append message is successful.

Verifying this change

XXXX

Follow this checklist to help us incorporate your contribution quickly and easily. Notice, it would be helpful if you could finish the following 5 checklist(the last one is not necessary)before request the community to review your PR.

Make sure there is a Github issue filed for the change (usually before you start working on it). Trivial changes like typos do not require a Github issue. Your pull request should address just this issue, without pulling in other changes - one PR resolves one issue.
Format the pull request title like [ISSUE #123] Fix UnknownException when host config not exist. Each commit in the pull request should have a meaningful subject line and body.
Write a pull request description that is detailed enough to understand what the pull request does, how, and why.
Write necessary unit-test(over 80% coverage) to verify your logic correction, more mock a little better when cross module dependency exist. If the new feature or significant change is committed, please remember to add integration-test in test module.
Run mvn -B clean apache-rat:check findbugs:findbugs checkstyle:checkstyle to make sure basic checks pass. Run mvn clean install -DskipITs to make sure unit-test pass. Run mvn clean test-compile failsafe:integration-test to make sure integration-test pass.
If this contribution is large, please file an Apache Individual Contributor License Agreement.

…et in consumeQueue build when the message is illegal

codecov-commenter · 2023-04-11T10:05:35Z

Codecov Report

Merging #6571 (8b48688) into develop (e3b8178) will increase coverage by 0.04%.
The diff coverage is 72.50%.

@@              Coverage Diff              @@
##             develop    #6571      +/-   ##
=============================================
+ Coverage      43.10%   43.15%   +0.04%     
- Complexity      8996     9002       +6     
=============================================
  Files           1107     1108       +1     
  Lines          78341    78361      +20     
  Branches       10216    10225       +9     
=============================================
+ Hits           33769    33815      +46     
+ Misses         40343    40312      -31     
- Partials        4229     4234       +5

Impacted Files	Coverage Δ
...rocketmq/common/message/MessageExtBrokerInner.java	`0.00% <0.00%> (ø)`
...org/apache/rocketmq/store/DefaultMessageStore.java	`46.79% <ø> (-0.15%)`	⬇️
...n/java/org/apache/rocketmq/store/MessageStore.java	`0.00% <ø> (ø)`
...ketmq/store/plugin/AbstractPluginMessageStore.java	`0.00% <ø> (ø)`
...main/java/org/apache/rocketmq/store/CommitLog.java	`66.03% <65.83%> (+0.85%)`	⬆️
...pache/rocketmq/store/dledger/DLedgerCommitLog.java	`76.31% <73.33%> (-0.60%)`	⬇️
...n/java/org/apache/rocketmq/store/ConsumeQueue.java	`66.26% <75.00%> (-0.24%)`	⬇️
...apache/rocketmq/store/queue/ConsumeQueueStore.java	`48.01% <77.77%> (+0.37%)`	⬆️
...a/org/apache/rocketmq/store/MessageExtEncoder.java	`86.59% <78.26%> (-3.48%)`	⬇️
.../java/org/apache/rocketmq/store/MultiDispatch.java	`89.65% <89.65%> (ø)`
... and 1 more

... and 19 files with indirect coverage changes

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

guyinyou

lgtm

store/src/main/java/org/apache/rocketmq/store/queue/BatchConsumeQueue.java

store/src/main/java/org/apache/rocketmq/store/CommitLog.java

lizhanhui · 2023-04-12T11:12:01Z

This fix is too complex and less performant.
How about this way: Before entering the critical region, validate and serialize the messages, with consume queue offset filled with place holders. In the critical region, allocate offsets as before and replace place-holders with them.
This way, critical region is minimized and CPU-intensive serialization can be parallel at the same time.

lizhanhui

Optimize solution

RongtongJin · 2023-04-12T11:34:58Z

This fix is too complex and less performant. How about this way: Before entering the critical region, validate and serialize the messages, with consume queue offset filled with place holders. In the critical region, allocate offsets as before and replace place-holders with them. This way, critical region is minimized and CPU-intensive serialization can be parallel at the same time.

Hi @lizhanhui, my fix do not change the logic of entering the critical region, but the operation of validate and serialize the messages can indeed be moved outside of topicQueueLock, I will try it.

lizhanhui · 2023-04-12T11:39:11Z

This fix is too complex and less performant. How about this way: Before entering the critical region, validate and serialize the messages, with consume queue offset filled with place holders. In the critical region, allocate offsets as before and replace place-holders with them. This way, critical region is minimized and CPU-intensive serialization can be parallel at the same time.

Hi @lizhanhui, my fix do not change the logic of entering the critical region, but the operation of validate and serialize the messages can indeed be moved outside of topicQueueLock, I will try it.

Okay. Considering the lock is global, any extra complication should be taken seriously. Looking forward to your optimized solution.

caigy

LGTM

store/src/main/java/org/apache/rocketmq/store/DefaultMessageStore.java

…icQueue lock

…t_match # Conflicts: # store/src/main/java/org/apache/rocketmq/store/DefaultMessageStore.java # store/src/main/java/org/apache/rocketmq/store/MessageStore.java # store/src/main/java/org/apache/rocketmq/store/plugin/AbstractPluginMessageStore.java # store/src/test/java/org/apache/rocketmq/store/DefaultMessageStoreCleanFilesTest.java # store/src/test/java/org/apache/rocketmq/store/MultiDispatchTest.java

RongtongJin · 2023-04-18T07:26:52Z

Hi @lizhanhui, I have completed the modification. Moving the serialization operation outside the lock for general messages (messages not related to BCQ and LMQ) is simple, and this is the solution implemented in version 4.x. However, since LMQ and BCQ were introduced after version 5.0, they need to overwrite the queueOffset-related properties during putting message, which caused the current changes. Therefore, for these two types of messages (BCQ and LMQ), when serializing outside the lock, the properties are not written, and properties are serialized and written after the queueOffset is truly determined in lock. This also makes the append process somewhat complicated. I have done some testing and it is currently running normally. Plz feel free to review.

RongtongJin · 2023-04-22T01:57:03Z

@lizhanhui Plz help review～

RongtongJin · 2023-04-24T00:40:53Z

@lizhanhui If we don't have enough time to review this PR currently, I suggest submitting another PR that fixes the bug in the original way first. That fix is minimally invasive and follows the current writing process. Then we can further review how to optimize performance in this PR.

RongtongJin · 2023-04-24T06:28:11Z

@lizhanhui If we don't have enough time to review this PR currently, I suggest submitting another PR that fixes the bug in the original way first. That fix is minimally invasive and follows the current writing process. Then we can further review how to optimize performance in this PR.

Original way to fix the issue #6641, the main purpose of this PR is to optimize the process of putting messages to improve performance.

mxsm

LGTM

# Conflicts: # store/src/main/java/org/apache/rocketmq/store/CommitLog.java # store/src/main/java/org/apache/rocketmq/store/ConsumeQueue.java # store/src/main/java/org/apache/rocketmq/store/DefaultMessageStore.java # store/src/main/java/org/apache/rocketmq/store/MessageStore.java # store/src/main/java/org/apache/rocketmq/store/dledger/DLedgerCommitLog.java # store/src/main/java/org/apache/rocketmq/store/plugin/AbstractPluginMessageStore.java # store/src/main/java/org/apache/rocketmq/store/queue/BatchConsumeQueue.java # store/src/main/java/org/apache/rocketmq/store/queue/ConsumeQueueInterface.java # store/src/main/java/org/apache/rocketmq/store/queue/ConsumeQueueStore.java # store/src/test/java/org/apache/rocketmq/store/MultiDispatchTest.java

ShadowySpirits · 2023-04-26T07:41:59Z

store/src/main/java/org/apache/rocketmq/store/CommitLog.java

@@ -815,89 +809,72 @@ public CompletableFuture<PutMessageResult> asyncPutMessage(final MessageExtBroke
            }
        }

-        topicQueueLock.lock(topicQueueKey);
-        try {
+        PutMessageThreadLocal putMessageThreadLocal = this.putMessageThreadLocal.get();


Is there a race issue? I notice you use thread local variable out of the critical region.

Yes, we serialize outside of the lock, so each thread will have a ByteBuf for storing serialized data.

lizhanhui · 2023-04-26T08:03:10Z

@lizhanhui If we don't have enough time to review this PR currently, I suggest submitting another PR that fixes the bug in the original way first. That fix is minimally invasive and follows the current writing process. Then we can further review how to optimize performance in this PR.

I am kind of busy recently and it is OK to create a new pull request for the optimization and dedicate this one to bugfix.

ni-ze · 2023-04-27T08:29:01Z

TGTM~

common/src/main/java/org/apache/rocketmq/common/message/MessageExtBrokerInner.java

store/src/main/java/org/apache/rocketmq/store/MultiDispatch.java

store/src/main/java/org/apache/rocketmq/store/CommitLog.java

store/src/main/java/org/apache/rocketmq/store/ConsumeQueue.java

RongtongJin · 2023-04-29T13:39:39Z

@caigy Completed the modification of all review comments.

lzkisok · 2023-10-30T09:04:18Z

大概什么时候能合并并发布新版本，谢谢

joeCarf

LGTM

Fix the issue that expectLogicOffset is greater than currentLogicOffs…

a18cad7

…et in consumeQueue build when the message is illegal

RongtongJin marked this pull request as draft April 11, 2023 08:25

RongtongJin added 2 commits April 11, 2023 17:20

Add new UT

b7d9bf5

Fix bug that UT can not pass

8fa3f39

RongtongJin marked this pull request as ready for review April 11, 2023 10:42

guyinyou previously approved these changes Apr 12, 2023

View reviewed changes

lizhanhui reviewed Apr 12, 2023

View reviewed changes

store/src/main/java/org/apache/rocketmq/store/queue/BatchConsumeQueue.java Outdated Show resolved Hide resolved

lizhanhui reviewed Apr 12, 2023

View reviewed changes

store/src/main/java/org/apache/rocketmq/store/CommitLog.java Outdated Show resolved Hide resolved

lizhanhui requested changes Apr 12, 2023

View reviewed changes

Fix the comment

3c865e2

RongtongJin dismissed guyinyou’s stale review via 3c865e2 April 12, 2023 11:44

caigy previously approved these changes Apr 13, 2023

View reviewed changes

fuyou001 reviewed Apr 14, 2023

View reviewed changes

store/src/main/java/org/apache/rocketmq/store/DefaultMessageStore.java Outdated Show resolved Hide resolved

RongtongJin added 3 commits April 17, 2023 15:07

Refactor the relevant logic so that encode message is outside the top…

783c62b

…icQueue lock

Pass the checkstyle

2172daf

RongtongJin dismissed caigy’s stale review via 2172daf April 17, 2023 07:40

RongtongJin added 3 commits April 17, 2023 16:38

Pass the BCQ UTs

18f2dcc

Pass the checkstyle

a01b961

Pass BatchPutMessageTest.testPutMessages UT

3509fbb

RongtongJin requested review from fuyou001, lizhanhui, guyinyou and caigy April 18, 2023 07:27

RongtongJin changed the title ~~[ISSUE #6570] Fix the issue that expectLogicOffset is greater than currentLogicOffset in consumeQueue build when the message is illegal~~ [ISSUE #6570] Optimizing the process of putting messages to improve performance Apr 24, 2023

mxsm previously approved these changes Apr 24, 2023

View reviewed changes

RongtongJin dismissed mxsm’s stale review via 8b48688 April 26, 2023 06:20

ShadowySpirits reviewed Apr 26, 2023

View reviewed changes