branch-4.1: pick some group commit prs by mymeiyi · Pull Request #61487 · apache/doris

mymeiyi · 2026-03-18T10:36:47Z

…mit_data_bytes to 0 (apache#61034)

…61242) ### What problem does this PR solve? Issue Number: close #xxx Related PR: #xxx Problem Summary: 1. Create table support set `group_commit_mode` table property ``` CREATE TABLE ... PROPERTIES( "group_commit_mode" = "async_mode" ); ``` 2. Support alter this property ``` ALTER TABLE ... SET ("group_commit_mode" = "off_mode"); ``` 3. Show create table shows this property if its value is not `off_mode` 4. For stream load, if it not set `group_commit` header, use the table property as the group commit mode; if it set `group_commit` header, use the header value. 5. doc: apache/doris-website#3465 ### Release note None ### Check List (For Author) - Test  - [ ] Regression test - [ ] Unit Test - [ ] Manual test (add detailed scripts or steps below) - [ ] No need to test or manual test. Explain why: - [ ] This is a refactor/code format and no logic has been changed. - [ ] Previous test can cover this change. - [ ] No code files have been changed. - [ ] Other reason  - Behavior changed: - [ ] No. - [ ] Yes.  - Does this need documentation? - [ ] No. - [ ] Yes.  ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label

hello-stephen · 2026-03-18T10:36:54Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

mymeiyi · 2026-03-18T10:39:45Z

run buildall

Copilot

Pull request overview

This PR introduces a new OLAP table property group_commit_mode (off/async/sync) and wires it through FE/BE so stream load can default to the table’s configured mode when the HTTP group_commit header is not provided.

Changes:

Add group_commit_mode table property with validation, persistence/show-create behavior, and (cloud) partition meta propagation.
Extend FE/BE stream load handshake to optionally fetch table group_commit_mode during loadTxnBegin when the client didn’t specify a mode header.
Add/adjust regression tests and golden outputs to cover the new property and validation rules.

Reviewed changes

Copilot reviewed 19 out of 19 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
regression-test/suites/load_p0/stream_load/test_group_commit_stream_load.groovy	Adds stream load tests for table-property-based group commit and header override behavior.
regression-test/suites/insert_p0/insert_group_commit_into.groovy	Adds DDL validation tests for group commit interval/bytes and new `group_commit_mode` property + SHOW CREATE TABLE behavior.
regression-test/data/query_p0/system/test_table_properties.out	Updates expected table properties output to include `group_commit_mode`.
gensrc/thrift/FrontendService.thrift	Adds thrift fields to request/response for “use table group_commit_mode” handshake.
gensrc/proto/cloud.proto	Adds `group_commit_mode` to tablet meta update payload for cloud mode.
fe/fe-core/src/main/java/org/apache/doris/service/FrontendServiceImpl.java	Implements early return of table `group_commit_mode` in `loadTxnBegin` when requested.
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/info/ModifyTablePropertiesOp.java	Centralizes validation via `PropertyAnalyzer` and adds validation for `group_commit_mode`.
fe/fe-core/src/main/java/org/apache/doris/datasource/InternalCatalog.java	Parses/applies `group_commit_mode` during OLAP table creation.
fe/fe-core/src/main/java/org/apache/doris/common/util/PropertyAnalyzer.java	Adds `group_commit_mode` constants + parsing/validation; tightens interval/bytes validation to `> 0`.
fe/fe-core/src/main/java/org/apache/doris/cloud/alter/CloudSchemaChangeHandler.java	Allows and propagates `group_commit_mode` updates to cloud tablet meta.
fe/fe-core/src/main/java/org/apache/doris/catalog/TableProperty.java	Stores/returns `group_commit_mode` with default `off_mode`.
fe/fe-core/src/main/java/org/apache/doris/catalog/OlapTable.java	Exposes `get/setGroupCommitMode()` through `OlapTable`.
fe/fe-core/src/main/java/org/apache/doris/catalog/Env.java	Prints `group_commit_mode` in SHOW CREATE TABLE only when not `off_mode`.
fe/fe-core/src/main/java/org/apache/doris/alter/SchemaChangeHandler.java	Permits `group_commit_mode` in allowed alter-table properties.
be/src/runtime/stream_load/stream_load_executor.cpp	Requests table mode during begin-txn when client didn’t set a mode; adopts mode without starting txn.
be/src/runtime/stream_load/stream_load_context.h	Adds `group_commit_mode` to stream load context.
be/src/runtime/stream_load/stream_load_context.cpp	Adds `GroupCommitMode` (and `LoadId`) to stream load JSON response when group commit is used.
be/src/http/action/stream_load.h	Adds helpers for group commit eligibility and WAL-space checks.
be/src/http/action/stream_load.cpp	Refactors group commit mode handling to support “table property default” and centralizes WAL-space checks.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

You can also share your feedback on Copilot code review. Take the survey.

regression-test/suites/load_p0/stream_load/test_group_commit_stream_load.groovy

+            set 'column_separator', ','
+            // NOT setting 'group_commit' header - should use table property, but set partitions
+            set 'partitions', "${tableNameAsync}"
+            set 'columns', 'id, name'
+            file "test_stream_load1.csv"


regression-test/suites/load_p0/stream_load/test_group_commit_stream_load.groovy

+        // Check data is loaded
+        def rowCount2 = sql "select count(*) from ${tableNameSync}"
+        logger.info("Row count for sync table: " + rowCount2)
+        assertTrue(rowCount2[0][0] > 0, "Data should be loaded")


regression-test/suites/insert_p0/insert_group_commit_into.groovy

+
+        def createStmt2 = sql """ SHOW CREATE TABLE ${table}_mode """
+        logger.info("SHOW CREATE TABLE result after alter: " + createStmt2)
+        assertTrue(createStmt2.toString().contains('SYNC_MODE'), "SHOW CREATE TABLE should contain sync_mode after alter")


mymeiyi · 2026-03-18T10:52:40Z

run feut

doris-robot · 2026-03-18T11:16:34Z

Cloud UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	79.06% (1786/2259)
Line Coverage	64.44% (31952/49583)
Region Coverage	65.27% (15990/24499)
Branch Coverage	55.78% (8499/15236)

mymeiyi · 2026-03-18T12:54:48Z

run buildall

hello-stephen · 2026-03-18T13:22:34Z

Cloud UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	79.06% (1786/2259)
Line Coverage	64.44% (31952/49583)
Region Coverage	65.28% (15994/24499)
Branch Coverage	55.85% (8509/15236)

doris-robot · 2026-03-18T14:20:04Z

BE UT Coverage Report

Increment line coverage 0.00% (0/71) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	52.83% (19365/36653)
Line Coverage	36.21% (181057/500074)
Region Coverage	32.67% (139874/428196)
Branch Coverage	33.68% (60919/180876)

hello-stephen · 2026-03-18T15:28:44Z

BE Regression && UT Coverage Report

Increment line coverage 83.10% (59/71) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	71.45% (25635/35876)
Line Coverage	54.28% (270922/499144)
Region Coverage	51.78% (223933/432442)
Branch Coverage	53.24% (96642/181505)

mymeiyi added 2 commits March 18, 2026 18:34

[fix](group commit) forbid set group_commit_interval_ms and group_com…

05b5b32

…mit_data_bytes to 0 (apache#61034)

mymeiyi requested a review from yiguolei as a code owner March 18, 2026 10:36

Copilot AI review requested due to automatic review settings March 18, 2026 10:36

Copilot started reviewing on behalf of mymeiyi March 18, 2026 10:37 View session

Copilot AI reviewed Mar 18, 2026

View reviewed changes

fix

a126823

yiguolei approved these changes Mar 19, 2026

View reviewed changes

yiguolei merged commit 37605bd into apache:branch-4.1 Mar 19, 2026
25 of 28 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

branch-4.1: pick some group commit prs#61487

branch-4.1: pick some group commit prs#61487
yiguolei merged 3 commits intoapache:branch-4.1from
mymeiyi:branch-4.1-pick-61034

mymeiyi commented Mar 18, 2026

Uh oh!

hello-stephen commented Mar 18, 2026

Uh oh!

mymeiyi commented Mar 18, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

mymeiyi commented Mar 18, 2026

Uh oh!

doris-robot commented Mar 18, 2026

Uh oh!

mymeiyi commented Mar 18, 2026

Uh oh!

hello-stephen commented Mar 18, 2026

Uh oh!

doris-robot commented Mar 18, 2026

Uh oh!

hello-stephen commented Mar 18, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Conversation

mymeiyi commented Mar 18, 2026

Uh oh!

hello-stephen commented Mar 18, 2026

Uh oh!

mymeiyi commented Mar 18, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

mymeiyi commented Mar 18, 2026

Uh oh!

doris-robot commented Mar 18, 2026

Cloud UT Coverage Report

Uh oh!

mymeiyi commented Mar 18, 2026

Uh oh!

hello-stephen commented Mar 18, 2026

Cloud UT Coverage Report

Uh oh!

doris-robot commented Mar 18, 2026

BE UT Coverage Report

Uh oh!

hello-stephen commented Mar 18, 2026

BE Regression && UT Coverage Report

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants