[improvement](fe) Add configurable return mode for insert publish timeout in ETL scenarios#63919
Open
wenzhenghu wants to merge 12 commits into
Open
[improvement](fe) Add configurable return mode for insert publish timeout in ETL scenarios#63919wenzhenghu wants to merge 12 commits into
wenzhenghu wants to merge 12 commits into
Conversation
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
Contributor
Author
|
run buildall |
Contributor
Author
|
run buildall |
Contributor
TPC-H: Total hot run time: 31315 ms |
Contributor
TPC-DS: Total hot run time: 172577 ms |
Contributor
Author
|
run buildall |
Contributor
FE UT Coverage ReportIncrement line coverage |
Contributor
TPC-H: Total hot run time: 31761 ms |
Contributor
TPC-DS: Total hot run time: 172440 ms |
### What problem does this PR solve?
Issue Number: N/A
Related PR: None
Problem Summary: The migrated insert publish-timeout change had targeted FE tests, but incremental coverage was still too low because several new branches were not exercised. In particular, OlapInsertExecutor.onFail() lacked direct coverage for committed and uncommitted failure cleanup paths, and SessionVariable still missed default-value, normalization, empty-input validation, and map restore branches for insert_visible_timeout_return_mode. This change expands the FE unit tests to cover those cases so the migrated logic is protected against regressions and the incremental coverage for the new code stays above the required threshold.
### Release note
None
### Check List (For Author)
- Test: FE unit test
- ./run-fe-ut.sh --run org.apache.doris.qe.SessionVariablesTest,org.apache.doris.nereids.trees.plans.commands.insert.OlapInsertExecutorTest
- ./run-fe-ut.sh --coverage --run org.apache.doris.qe.SessionVariablesTest,org.apache.doris.nereids.trees.plans.commands.insert.OlapInsertExecutorTest
- Behavior changed: No
- Does this need documentation: No
Contributor
Author
|
run buildall |
Contributor
TPC-H: Total hot run time: 31585 ms |
Contributor
TPC-DS: Total hot run time: 171226 ms |
Contributor
FE UT Coverage ReportIncrement line coverage |
Contributor
FE Regression Coverage ReportIncrement line coverage |
### What problem does this PR solve?
Issue Number: None
Related PR: None
Problem Summary: Add a regression case for the new session variable `insert_visible_timeout_return_mode`. The case injects the FE debug point `PublishVersionDaemon.stop_publish` so a normal OLAP insert can commit successfully but remain non-visible long enough to hit `insert_visible_timeout_ms`. It verifies that `committed` mode returns success, `error` mode returns the expected client error, and both rows become visible after publish resumes. The case uses the configured FE HTTP address for debug-point operations so it also works when an external regression target reports loopback host addresses in `SHOW FRONTENDS`.
### Release note
None
### Check List (For Author)
- Test: Regression test
- Run `./run-regression-test.sh --run --conf /tmp/regression-conf-remote.groovy -d insert_p0 -s test_insert_visible_timeout_return_mode -genOut`
- Behavior changed: No
- Does this need documentation: No
Contributor
Author
|
run buildall |
yiguolei
reviewed
Jun 1, 2026
yiguolei
reviewed
Jun 1, 2026
wenzhenghu
changed the title
Contributor
FE UT Coverage ReportIncrement line coverage |
### What problem does this PR solve? Issue Number: None Related PR: apache#63919 Problem Summary: Review feedback on the insert publish-timeout return-mode change requested using an enum instead of raw string constants. This commit keeps the SQL-visible values unchanged (`committed` and `error`) but stores the session variable as a dedicated enum inside FE. It also extends the generic session-variable assignment and restore paths so enum-backed variables can still be set from SQL, forwarded variables, and JSON/map persistence. The existing insert timeout behavior is unchanged, while the implementation becomes type-safe and removes the previous string normalization cleanup logic. ### Release note None ### Check List (For Author) - Test: FE Unit Test - Run `./run-fe-ut.sh --run org.apache.doris.qe.SessionVariablesTest,org.apache.doris.nereids.trees.plans.commands.insert.OlapInsertExecutorTest` - Behavior changed: No - Does this need documentation: No
Contributor
Author
|
run buildall |
Contributor
Author
|
run buildall |
Contributor
TPC-H: Total hot run time: 28969 ms |
Contributor
TPC-DS: Total hot run time: 172010 ms |
Contributor
FE Regression Coverage ReportIncrement line coverage |
Contributor
TPC-H: Total hot run time: 29293 ms |
Contributor
TPC-DS: Total hot run time: 169577 ms |
Contributor
Author
|
run buildall |
Contributor
TPC-H: Total hot run time: 29834 ms |
Contributor
TPC-DS: Total hot run time: 171658 ms |
Contributor
FE UT Coverage ReportIncrement line coverage |
Contributor
FE Regression Coverage ReportIncrement line coverage |
Contributor
Author
|
run buildall |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What problem does this PR solve?
Issue Number: close #xxx
Related PR: #xxx
Problem Summary:
Normal internal-table inserts currently treat publish timeout as a committed insert and return success with COMMITTED status. This behavior is acceptable when clients only care that the transaction has been committed and can tolerate delayed visibility, but it is unsafe for pipelines whose downstream steps depend on the inserted data already being visible.
A typical case is ETL workflows that first use CREATE TABLE AS SELECT to build a temporary table and then immediately read that table to populate a downstream result table. If the upstream transaction has been committed but is not yet VISIBLE, the downstream step may temporarily read no rows and silently write empty data into the final table, so the whole pipeline appears successful even though the result is incorrect.
Doris already returns an error in explicit transaction mode when a COMMIT statement times out before the transaction becomes visible. This change adds a compatible mode for the regular non-transactional internal-table insert path by introducing a session variable, insert_visible_timeout_return_mode, so users can choose whether publish timeout should keep returning COMMITTED or return ERR.
The implementation also keeps committed-side bookkeeping unchanged in error mode so finished load jobs, insert result metadata, and related accounting still reflect the real transaction state.
Release note
Add a session variable to control whether normal internal-table inserts return COMMITTED or ERR when publish visibility times out.
Check List (For Author)
Test
Behavior changed:
Does this need documentation?
Check List (For Reviewer who merge this PR)