[feature](txn insert) txn insert support insert into select #31666
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Thank you for your contribution to Apache Doris. |
mymeiyi
force-pushed
the
txn-insert-tables
branch
from
3becfa8
to
e40a1ab
Compare
|
run buildall |
1 similar comment
|
run buildall |
TPC-H: Total hot run time: 32116 ms |
TPC-DS: Total hot run time: 187669 ms |
ClickBench: Total hot run time: 27.03 s |
mymeiyi
force-pushed
the
txn-insert-tables
branch
from
f18dccc
to
659ca52
Compare
|
run buildall |
morrySnow
reviewed
Mar 4, 2024
| @@ -531,7 +531,8 @@ public LogicalPlan visitInsertTable(InsertTableContext ctx) { | |||
| if (isOverwrite) { | |||
| command = new InsertOverwriteTableCommand(sink, labelName); | |||
| } else { | |||
| if (ConnectContext.get() != null && ConnectContext.get().isTxnModel()) { | |||
| if (ConnectContext.get() != null && ConnectContext.get().isTxnModel() | |||
| && sink.child() instanceof LogicalInlineTable) { | |||
There was a problem hiding this comment.
In legacy planner, we support insert into t select 1. So if we only support LogicalInlineTable here means we have diff with legacy planner
There was a problem hiding this comment.
The nereids does not support insert into t select 1? It fallback to legacy
|
|
||
| if (physicalSink instanceof PhysicalOlapTableSink) { | ||
| if (GroupCommitInserter.groupCommit(ctx, sink, physicalSink)) { | ||
| return; | ||
| } | ||
| OlapTable olapTable = (OlapTable) targetTableIf; | ||
| String label = this.labelName.orElse( | ||
| ctx.isTxnModel() ? null : String.format("label_%x_%x", ctx.queryId().hi, ctx.queryId().lo)); |
There was a problem hiding this comment.
could u add some comment to explain why lable should be null when we in txn model
| @@ -162,6 +162,10 @@ private static InternalService.PDataRow getRowStringValue(List<NamedExpression> | |||
| private static void beginBatchInsertTransaction(ConnectContext ctx, | |||
| String dbName, String tblName, List<Column> columns) { | |||
| TransactionEntry txnEntry = ctx.getTxnEntry(); | |||
| if (txnEntry.isTransactionBegan()) { | |||
| throw new AnalysisException( | |||
| "Transaction insert can not insert into values and insert into select at the same time"); | |||
There was a problem hiding this comment.
why isTransactionBegan means use insert into values with insert into select at same time?
There was a problem hiding this comment.
isTransactionBegan means the insert into select already began a txn.
This function will begin a txn for insert into values.
Currently we forbid it, maybe support it later.
There was a problem hiding this comment.
add a TODO coment here, we'll support mix usage of insert into values and insert into select in future.
| boolean isGroupCommit = (parsedStmt != null | ||
| && parsedStmt instanceof LogicalPlanAdapter | ||
| && ((LogicalPlanAdapter) parsedStmt).getLogicalPlan() instanceof InsertIntoTableCommand) | ||
| && !context.isTxnModel(); |
There was a problem hiding this comment.
so all insert into without txn model is group commit?
There was a problem hiding this comment.
The logic should be
boolean isGroupCommit = (Config.wait_internal_group_commit_finish
|| context.sessionVariable.isEnableInsertGroupCommit()) && parsedStmt != null
&& parsedStmt instanceof LogicalPlanAdapter
&& ((LogicalPlanAdapter) parsedStmt).getLogicalPlan() instanceof InsertIntoTableCommand;
but some nereids case can not pass.
I add !context.isTxnModel() here becase the regression cases expect txn insert does not force fallback
There was a problem hiding this comment.
so the variable name now should not be isGroupCommit? please use a better variable name, and add comments to explain it here
zhannngchen
reviewed
Mar 5, 2024
zhannngchen
reviewed
Mar 5, 2024
fe/fe-core/src/main/java/org/apache/doris/transaction/TransactionEntry.java
Show resolved
Hide resolved
mymeiyi
force-pushed
the
txn-insert-tables
branch
from
659ca52
to
96f04b2
Compare
|
run buildall |
TPC-H: Total hot run time: 37666 ms |
TPC-DS: Total hot run time: 178038 ms |
ClickBench: Total hot run time: 30.64 s |
|
Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G' |
github-actions
bot
added
the
approved
|
PR approved by at least one committer and no changes requested. |
|
PR approved by anyone and no changes requested. |
mymeiyi
force-pushed
the
txn-insert-tables
branch
from
96f04b2
to
2bf8bd0
Compare
|
run buildall |
|
run p0 |
morningman
pushed a commit
that referenced
this pull request
Apr 12, 2024
dataroaring
pushed a commit
that referenced
this pull request
Jun 5, 2024
## Proposed changes ### Purpose The user doc: https://doris.apache.org/zh-CN/docs/dev/data-operate/import/transaction-load-manual We have supported insert into select(#31666), update(#33034) and delete(#33100) in transaction load. #32980 implements one txn write to one partition more than one rowsets. This pr implements to cloud mode of #32980 ### Implementation #### sub_txn_id see #32980 #### Meta service supports commit txn This process is generally the same as commit_txn, the difference is that he partitions version will plus 1 in multi sub txns. One example: Suppose the table, partition, tablet and version info is: ``` -------------------------------------------- | table | partition | tablet | version | -------------------------------------------- | t1 | t1_p1 | t1_p1.1 | 1 | | t1 | t1_p1 | t1_p1.2 | 1 | | t1 | t1_p2 | t1_p2.1 | 2 | | t2 | t2_p3 | t2_p3.1 | 3 | | t2 | t2_p4 | t2_p4.1 | 4 | -------------------------------------------- ``` Now we commit a txn with 3 sub txns and the tablets are: * sub_txn1: t1_p1.1, t1_p1.2, t1_p2.1 * sub_txn2: t2_p3.1 * sub_txn3: t1_p1.1, t1_p1.2 When commit, the partitions version will be: * sub_txn1: t1_p1(1 -> 2), t1_p2(2 -> 3) * sub_txn2: t2_p3(3 -> 4) * sub_txn3: t1_p1(2 -> 3) After commit, the partitions version will be: * t1: t1_p1(3), t1_p2(3) * t2: t2_p3(4), t2_p4(4) #### Meta service support generate sub_txn_id by `begin_sub_txn`
dataroaring
pushed a commit
that referenced
this pull request
Jun 7, 2024
## Proposed changes ### Purpose The user doc: https://doris.apache.org/zh-CN/docs/dev/data-operate/import/transaction-load-manual We have supported insert into select(#31666), update(#33034) and delete(#33100) in transaction load. #32980 implements one txn write to one partition more than one rowsets. This pr implements to cloud mode of #32980 ### Implementation #### sub_txn_id see #32980 #### Meta service supports commit txn This process is generally the same as commit_txn, the difference is that he partitions version will plus 1 in multi sub txns. One example: Suppose the table, partition, tablet and version info is: ``` -------------------------------------------- | table | partition | tablet | version | -------------------------------------------- | t1 | t1_p1 | t1_p1.1 | 1 | | t1 | t1_p1 | t1_p1.2 | 1 | | t1 | t1_p2 | t1_p2.1 | 2 | | t2 | t2_p3 | t2_p3.1 | 3 | | t2 | t2_p4 | t2_p4.1 | 4 | -------------------------------------------- ``` Now we commit a txn with 3 sub txns and the tablets are: * sub_txn1: t1_p1.1, t1_p1.2, t1_p2.1 * sub_txn2: t2_p3.1 * sub_txn3: t1_p1.1, t1_p1.2 When commit, the partitions version will be: * sub_txn1: t1_p1(1 -> 2), t1_p2(2 -> 3) * sub_txn2: t2_p3(3 -> 4) * sub_txn3: t1_p1(2 -> 3) After commit, the partitions version will be: * t1: t1_p1(3), t1_p2(3) * t2: t2_p3(4), t2_p4(4) #### Meta service support generate sub_txn_id by `begin_sub_txn`
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Proposed changes
Transaction insert support:
Limit:
insert into selectandinsert into valuescan be used at one transaction.Will support in later pr.
Further comments
If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...