[improve](txn insert) Txn load support cloud mode #34721
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Thank you for your contribution to Apache Doris. Since 2024-03-18, the Document has been moved to doris-website. |
|
run buildall |
|
clang-tidy review says "All clean, LGTM! 👍" |
TPC-H: Total hot run time: 40555 ms |
TPC-DS: Total hot run time: 185951 ms |
gavinchou
reviewed
May 11, 2024
mymeiyi
force-pushed
the
txn-insert-cloud-2
branch
2 times, most recently
from
8078ba3
to
296f6a1
Compare
|
clang-tidy review says "All clean, LGTM! 👍" |
|
run buildall |
|
clang-tidy review says "All clean, LGTM! 👍" |
TPC-H: Total hot run time: 41166 ms |
TPC-DS: Total hot run time: 188753 ms |
mymeiyi
force-pushed
the
txn-insert-cloud-2
branch
from
296f6a1
to
1fc1a2b
Compare
mymeiyi
force-pushed
the
txn-insert-cloud-2
branch
from
1fc1a2b
to
a493ddf
Compare
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
| @@ -1332,6 +1332,313 @@ TEST(MetaServiceTest, CommitTxnExpiredTest) { | |||
| } | |||
| } | |||
|
|
|||
| TEST(MetaServiceTest, CommitTxnWithSubTxnTest) { | |||
There was a problem hiding this comment.
warning: function 'TEST' exceeds recommended size/complexity thresholds [readability-function-size]
TEST(MetaServiceTest, CommitTxnWithSubTxnTest) {
^Additional context
cloud/test/meta_service_test.cpp:1334: 185 lines including whitespace and comments (threshold 80)
TEST(MetaServiceTest, CommitTxnWithSubTxnTest) {
^| } | ||
| } | ||
|
|
||
| TEST(MetaServiceTest, BeginAndAbortSubTxnTest) { |
There was a problem hiding this comment.
warning: function 'TEST' exceeds recommended size/complexity thresholds [readability-function-size]
TEST(MetaServiceTest, BeginAndAbortSubTxnTest) {
^Additional context
cloud/test/meta_service_test.cpp:1521: 118 lines including whitespace and comments (threshold 80)
TEST(MetaServiceTest, BeginAndAbortSubTxnTest) {
^
|
run buildall |
TPC-H: Total hot run time: 41841 ms |
TPC-DS: Total hot run time: 187484 ms |
ClickBench: Total hot run time: 30.74 s |
| } | ||
| } | ||
|
|
||
| TEST(MetaServiceTest, BeginAndAbortSubTxnTest) { |
There was a problem hiding this comment.
warning: function 'TEST' exceeds recommended size/complexity thresholds [readability-function-size]
TEST(MetaServiceTest, BeginAndAbortSubTxnTest) {
^Additional context
cloud/test/meta_service_test.cpp:1521: 119 lines including whitespace and comments (threshold 80)
TEST(MetaServiceTest, BeginAndAbortSubTxnTest) {
^
|
run buildall |
mymeiyi
force-pushed
the
txn-insert-cloud-2
branch
from
4c9df64
to
80a107e
Compare
|
run buildall |
mymeiyi
force-pushed
the
txn-insert-cloud-2
branch
2 times, most recently
from
94bc14a
to
16a2691
Compare
|
run buildall |
TPC-DS: Total hot run time: 182574 ms |
|
run buildall |
|
run fe_ut |
|
run performance |
mymeiyi
force-pushed
the
txn-insert-cloud-2
branch
from
8981044
to
afd5858
Compare
|
run buildall |
TPC-H: Total hot run time: 41029 ms |
TPC-DS: Total hot run time: 172961 ms |
ClickBench: Total hot run time: 30.23 s |
|
run cloud_p0 |
|
PR approved by at least one committer and no changes requested. |
github-actions
bot
added
the
approved
mymeiyi
force-pushed
the
txn-insert-cloud-2
branch
from
afd5858
to
029385e
Compare
github-actions
bot
removed
the
approved
|
run buildall |
TPC-H: Total hot run time: 41093 ms |
TPC-DS: Total hot run time: 172097 ms |
ClickBench: Total hot run time: 30.84 s |
|
run cloud_ut |
|
run p0 |
|
run cloudut |
github-actions
bot
added
the
approved
|
PR approved by at least one committer and no changes requested. |
dataroaring
pushed a commit
that referenced
this pull request
Jun 7, 2024
## Proposed changes ### Purpose The user doc: https://doris.apache.org/zh-CN/docs/dev/data-operate/import/transaction-load-manual We have supported insert into select(#31666), update(#33034) and delete(#33100) in transaction load. #32980 implements one txn write to one partition more than one rowsets. This pr implements to cloud mode of #32980 ### Implementation #### sub_txn_id see #32980 #### Meta service supports commit txn This process is generally the same as commit_txn, the difference is that he partitions version will plus 1 in multi sub txns. One example: Suppose the table, partition, tablet and version info is: ``` -------------------------------------------- | table | partition | tablet | version | -------------------------------------------- | t1 | t1_p1 | t1_p1.1 | 1 | | t1 | t1_p1 | t1_p1.2 | 1 | | t1 | t1_p2 | t1_p2.1 | 2 | | t2 | t2_p3 | t2_p3.1 | 3 | | t2 | t2_p4 | t2_p4.1 | 4 | -------------------------------------------- ``` Now we commit a txn with 3 sub txns and the tablets are: * sub_txn1: t1_p1.1, t1_p1.2, t1_p2.1 * sub_txn2: t2_p3.1 * sub_txn3: t1_p1.1, t1_p1.2 When commit, the partitions version will be: * sub_txn1: t1_p1(1 -> 2), t1_p2(2 -> 3) * sub_txn2: t2_p3(3 -> 4) * sub_txn3: t1_p1(2 -> 3) After commit, the partitions version will be: * t1: t1_p1(3), t1_p2(3) * t2: t2_p3(4), t2_p4(4) #### Meta service support generate sub_txn_id by `begin_sub_txn`
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Proposed changes
Purpose
The user doc: https://doris.apache.org/zh-CN/docs/dev/data-operate/import/transaction-load-manual
We have supported insert into select(#31666), update(#33034) and delete(#33100) in transaction load.
#32980 implements one txn write to one partition more than one rowsets.
This pr implements to cloud mode of #32980
Implementation
sub_txn_id
see #32980
Meta service supports commit txn
This process is generally the same as commit_txn, the difference is that he partitions version will plus 1 in multi sub txns.
One example:
Suppose the table, partition, tablet and version info is:
Now we commit a txn with 3 sub txns and the tablets are:
When commit, the partitions version will be:
After commit, the partitions version will be:
Meta service support generate sub_txn_id by
begin_sub_txnFurther comments
If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...