[draft](ivm) Add mv ivm test for pipeline#62606
Draft
yujun777 wants to merge 112 commits into
Draft
Conversation
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
Contributor
Author
|
run buildall |
Contributor
FE Regression Coverage ReportIncrement line coverage |
|
run feut |
|
run nonConcurrent |
Contributor
FE Regression Coverage ReportIncrement line coverage |
Contributor
Author
|
run buildall |
Contributor
FE Regression Coverage ReportIncrement line coverage |
|
run buildall |
Contributor
FE Regression Coverage ReportIncrement line coverage |
|
run buildall |
Contributor
FE Regression Coverage ReportIncrement line coverage |
|
run feut |
|
run buildall |
yujun777
force-pushed
the
ivm
branch
2 times, most recently
from
13a1dd6 to
1166da3
Compare
Contributor
Author
|
run buildall |
Contributor
TPC-H: Total hot run time: 29676 ms |
Contributor
TPC-DS: Total hot run time: 169641 ms |
Contributor
FE Regression Coverage ReportIncrement line coverage |
Contributor
Author
|
run buildall |
Contributor
TPC-H: Total hot run time: 29321 ms |
Contributor
TPC-DS: Total hot run time: 170433 ms |
Contributor
FE Regression Coverage ReportIncrement line coverage |
Contributor
Author
|
run buildall |
IvmDeltaCommandBundle was a trivial wrapper around Command with no extra fields. Replace it with Command directly throughout the IVM pipeline. Key changes: - Delete IvmDeltaCommandBundle.java - Change IvmDeltaStrategy.rewrite() return type to List<Command> - Update IvmDeltaRewriter, IvmDeltaExecutor, IvmRefreshManager signatures - Update all IVM test files accordingly Unit Test: all 42 IVM FE unit tests pass
Use the aggregate row-id slot when checking grouped aggregate determinism instead of relying on insertion order from the normalize result map. Key changes: - Assert determinism through normalizeResult.isDeterministic(rowIdAlias.toSlot()). - Keep child scan row-id tracking behavior unchanged. Unit Test: - ./run-fe-ut.sh --run org.apache.doris.nereids.rules.rewrite.IvmNormalizeMtmvTest#testGroupedAggInjectsRowIdAndHiddenColumns - ./run-fe-ut.sh --run org.apache.doris.mtmv.ivm.IvmStreamRefTest,org.apache.doris.mtmv.ivm.IvmNormalizeMtmvUnionTest,org.apache.doris.mtmv.ivm.IvmAggDeltaStrategyTest,org.apache.doris.mtmv.ivm.IvmUtilTest,org.apache.doris.mtmv.ivm.IvmRefreshManagerTest,org.apache.doris.mtmv.ivm.IvmDeltaExecutorTest,org.apache.doris.mtmv.ivm.IvmSimpleScanDeltaStrategyTest,org.apache.doris.mtmv.ivm.IvmNormalizeMtmvJoinTest,org.apache.doris.mtmv.ivm.IvmDeltaRewriterTest,org.apache.doris.nereids.rules.rewrite.IvmNormalizeMtmvTest Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…ategy The class no longer covers only scan/project. It also handles inner join and union all, and is the home for any operator whose delta propagates linearly (output delta = combination of input deltas without base-table lookup or aggregate replay). Rename it to reflect the actual scope; outer join will get its own strategy in the future. Key changes: - Rename IvmSimpleScanDeltaStrategy -> IvmLinearDeltaStrategy (main + test) via git mv to preserve history - Update all references in IvmAggDeltaStrategy, IvmDeltaRewriter, tests, and mtmv/ivm/AGENTS.md - Rewrite class Javadoc to describe linear delta propagation and the currently supported operators (Scan/Project/Filter/InnerJoin/UnionAll) Unit Test: - IvmLinearDeltaStrategyTest: 32/32 pass - IvmAggDeltaStrategyTest: 25/25 pass - IvmDeltaRewriterTest: 21/21 pass Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
run() was ~147 lines mixing partition sync, refresh planning, IVM fast-path and partition-based execution. Extract three private helpers; the moved blocks are kept verbatim. Key changes: - Extract syncPartitionsIfNeeded(ctx, tableIfs): schema-change check, pct table validity, alignMvPartition + drop/add partitions - Extract tryIvmFastPath(): IVM AUTO/INCREMENTAL fast path; returns true if IVM completed the refresh, false to fall through to partition-based refresh; throws JobException on INCREMENTAL failure. Use early-return guard for the non-IVM case - Extract executePartitionBasedRefresh(context): TSO capture, batch exec loop and post-refresh IVM state reset Unit Test: - MTMVTaskTest: 7/7 pass - IvmLinearDeltaStrategyTest: 32/32 pass - IvmAggDeltaStrategyTest: 25/25 pass - IvmDeltaRewriterTest: 21/21 pass - IvmRefreshManagerTest: 17/17 pass Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
### What problem does this PR solve?
Issue Number: N/A
Related PR: N/A
Problem Summary: AUTO refresh materialized views that are eligible for incremental refresh should be persisted as IVM MVs. Before this change, CreateMTMVInfo.isEnableIvm() only checked whether the refresh method was explicitly INCREMENTAL, so AUTO MVs stayed non-IVM even when query analysis could build an incremental-refresh plan.
Root cause: In CreateMTMVInfo.isEnableIvm(), the IVM flag was derived directly from RefreshMethod.INCREMENTAL, which ignored AUTO refresh capability decided during create-time query analysis.
This patch introduces an independent create-time enableIvm flag. AUTO refresh MVs probe IVM capability through analyzeQuery(); if the probe succeeds, the MV keeps IVM metadata, and if it fails, analysis state is restored and the MV continues as a normal non-IVM MV. Explicit INCREMENTAL remains strict and does not fallback.
### Release note
Fixed an issue where AUTO materialized views that can be incrementally refreshed were not persisted as IVM materialized views.
### Check List (For Author)
- Test: Regression test and Unit Test
- ./run-fe-ut.sh --run org.apache.doris.mtmv.ivm.IvmDeltaExecutorTest,org.apache.doris.mtmv.ivm.IvmStreamRefTest,org.apache.doris.mtmv.ivm.IvmNormalizeMtmvJoinTest,org.apache.doris.mtmv.ivm.IvmAggDeltaStrategyTest,org.apache.doris.mtmv.ivm.IvmDeltaRewriterTest,org.apache.doris.mtmv.ivm.IvmRefreshManagerTest,org.apache.doris.mtmv.ivm.IvmNormalizeMtmvUnionTest,org.apache.doris.mtmv.ivm.IvmLinearDeltaStrategyTest,org.apache.doris.mtmv.ivm.IvmUtilTest,org.apache.doris.nereids.rules.rewrite.IvmNormalizeMtmvTest,org.apache.doris.nereids.trees.plans.CreateMTMVCommandTest,org.apache.doris.catalog.ShowCreateMTMVTest
- ./run-regression-test.sh --run -d mtmv_p0/ivm
- Behavior changed: Yes. AUTO MVs that pass create-time IVM analysis now persist IVM metadata.
- Does this need documentation: No
Issue Number: None
Related PR: None
Problem Summary: IVM delta refresh did not surface base table binlog-disabled cases to MV task info, so AUTO refresh could silently fall back without a task-visible reason.
Record IVM fallback reason in MV task info when base table binlog is disabled.
- Test: Unit Test
- ./run-fe-ut.sh --run org.apache.doris.nereids.rules.rewrite.IvmNormalizeMtmvTest,org.apache.doris.mtmv.ivm.IvmRefreshManagerTest,org.apache.doris.mtmv.MTMVTaskTest
- Behavior changed: Yes (IVM AUTO refresh records BINLOG_NOT_ENABLED fallback reason; explicit INCREMENTAL refresh still fails on IVM fallback)
- Does this need documentation: No
### What problem does this PR solve?
Issue Number: None
Related PR: None
Problem Summary: AUTO MTMV query analysis used a boolean return value only to make the caller retry regular analysis after IVM probe fallback. Move that retry into the helper so the caller only selects the analysis path.
### Release note
None
### Check List (For Author)
- Test: Manual test
- git diff --check -- fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/info/CreateMTMVInfo.java
- Behavior changed: No
- Does this need documentation: No
Implement O1 delta rewrite for root LEFT OUTER JOIN IVM MVs and route normalized outer-join MVs to the new strategy. Keep full refresh behavior independent from unfinished TSO binding by clearing only the stale running refresh flag for now. Key changes: - Add outer join delta strategy for preserved-side delta, nullable-side joined delta, and pad-null repair rows. - Mark and validate outer join MVs during IVM normalization, including root-only restrictions and nullable-side snapshot limitations. - Merge delta rewrite context into IvmRefreshContext and keep base table stream lookup centralized. - Add FE unit tests and regression coverage for supported and unsupported outer join IVM cases. Unit Test: - FE UT: IVM unit tests plus CreateMTMVCommandTest, 189 tests passed. - Regression: mtmv_p0/ivm, 18 suites passed. - Check: git diff --check.
Optimize nullable-side left outer join IVM rewrites by converting pure equi-join right deltas into right-side events, so the preserved-side snapshot is probed once while preserving fallback behavior for general join predicates. Key changes: - Build right-event plans for detail rows and padded-null repair rows on pure deterministic equi join keys. - Fall back to repair branches when residual non-hash predicates or unique functions appear in join keys. - Add comments explaining the event rewrite and extend outer join IVM FE/regression coverage. Unit Test: - IvmOuterJoinDeltaStrategyTest - Regression test: test_ivm_outer_join_1
### What problem does this PR solve?
Issue Number: close #xxx
Related PR: #xxx
Problem Summary: AUTO refresh MTMV uses IVM analysis as a capability probe before falling back to regular MTMV analysis. When the IVM probe fails after mutating the create state, the fallback path can reuse polluted analyze state and fail cases that should be accepted by regular MTMV analysis. This change restores the create state, reparses the original query through the same query parser path used by CREATE MTMV, resets the statement context, and then reruns regular analysis.
### Release note
None
### Check List (For Author)
- Test: Regression test / Unit Test
- `mvn test -pl fe-core -am -Dcheckstyle.skip=true -DfailIfNoTests=false -Dmaven.build.cache.enabled=false -Dtest=org.apache.doris.nereids.trees.plans.CreateMTMVCommandTest -Dfe_ut_parallel=0`
- `./run-regression-test.sh --run -d nereids_rules_p0/mv/increment_create -s self_conn_list_str_increment_create`
- `./run-regression-test.sh --run -d nereids_rules_p0/mv/increment_create -s self_conn_range_date_increment_create`
- `./run-regression-test.sh --run -d nereids_rules_p0/mv/increment_create -s self_conn_range_number_increment_create`
- Behavior changed: Yes. AUTO refresh MTMV now treats IVM probe analysis failures as capability probe failures and retries clean regular MTMV analysis.
- Does this need documentation: No
### What problem does this PR solve?
Issue Number: None
Related PR: None
Problem Summary: Add EXPLAIN REFRESH MATERIALIZED VIEW ... INCREMENTAL support for IVM dry-run debugging, including normalized plan overview and per-delta logical plan inspection.
### Release note
Add EXPLAIN REFRESH MATERIALIZED VIEW ... INCREMENTAL for IVM refresh plan debugging.
### Check List (For Author)
- Test: Unit Test and Regression test
- Unit Test: mvn test -Dcheckstyle.skip=true -DfailIfNoTests=false -Dmaven.build.cache.enabled=false -Dfe_ut_parallel=0 -Dtest=org.apache.doris.nereids.trees.plans.commands.ExplainRefreshIvmCommandTest
- Unit Test: mvn test -Dcheckstyle.skip=true -DfailIfNoTests=false -Dmaven.build.cache.enabled=false -Dfe_ut_parallel=0 -Dtest=Ivm*Test,CreateMTMVCommandTest,ShowCreateMTMVTest,ExplainRefreshIvmCommandTest
- Regression test: ./run-regression-test.sh --run -d mtmv_p0/ivm -s test_ivm_explain_refresh
- Regression test: ./run-regression-test.sh --run -d mtmv_p0/ivm
- Behavior changed: Yes. Adds a new EXPLAIN command form for IVM refresh dry-run plans.
- Does this need documentation: Yes. IVM developer guidance was updated in fe/fe-core/src/main/java/org/apache/doris/mtmv/ivm/AGENTS.md.
Add EXPLAIN REFRESH handling for IVM dry-run overview and per-delta plans, including planner explain support for a selected delta. Key changes: - Add ExplainRefreshIvmCommand for overview and FOR DELTA explain paths. - Format IVM normalized and delta plan metadata in explain output. - Add FE unit coverage and an IVM explain refresh regression case with non-empty base tables. Unit Test: - ExplainRefreshIvmCommandTest - mtmv_p0/ivm/test_ivm_explain_refresh
Keep ExplainRefreshIvmCommand aligned with FE style by removing an unused import and wrapping a long method signature. Key changes: - Remove unused java.util.List import. - Wrap explainDeltaPlan parameters to satisfy style. Unit Test: - Not run; format-only change.
Extend IVM normalization and delta rewriting so left outer join chains can propagate through non-aggregate plans and root aggregates. Key changes: - Track nullable-side context while normalizing IVM plans and reject outer joins only inside left outer join nullable sides. - Route non-aggregate rewrites through the outer join strategy and let aggregate strategy inherit outer join handling. - Remove the normalize-result outer join flag and add positive and negative coverage for the new O2 topology. Unit Test: - IvmNormalizeMtmvJoinTest, IvmOuterJoinDeltaStrategyTest, IvmAggDeltaStrategyTest, IvmDeltaRewriterTest - All IVM FE UTs - mtmv_p0/ivm regression suites
Generalize IVM outer join delta rewrite and normalize checks from left-outer-only assumptions to preserved/nullable side handling, enabling right outer join while keeping nullable-side restrictions explicit. Key changes: - Make IvmOuterJoinDeltaStrategy side-aware for preserved-side and nullable-side deltas. - Allow RIGHT_OUTER_JOIN in IVM normalize and move nullable-side UNION ALL with OlapScan rejection to UNION handling. - Add FE UT coverage for right outer join normalize and delta rewrite paths. - Split large outer join regression into smaller suites and add right outer join regression cases. Unit Test: - IvmNormalizeMtmvJoinTest - IvmOuterJoinDeltaStrategyTest - mtmv_p0/ivm test_ivm_outer_join_1,test_ivm_outer_join_2,test_ivm_outer_join_3 - mtmv_p0/ivm full regression
Unify IVM delta rewriting through the aggregate strategy entry point and remove the bare delta strategy interface. Pass refresh context explicitly through linear and outer join strategies so strategy instances no longer hold per-refresh state. Key changes: - Add singleton entry for IvmAggDeltaStrategy and route IvmDeltaRewriter through it. - Remove IvmDeltaStrategy and make linear/outer join strategy context handling visitor-based. - Update normalize result tests to check aggregate metadata directly. - Clean fully qualified class references from IVM FE unit tests. - Document that IVM FE unit test runs must include CreateMTMVCommandTest. Unit Test: - IvmAggDeltaStrategyTest, IvmDeltaExecutorTest, IvmDeltaRewriterTest, IvmLinearDeltaStrategyTest, IvmNormalizeMtmvJoinTest, IvmNormalizeMtmvUnionTest, IvmOuterJoinDeltaStrategyTest, IvmRefreshManagerTest, IvmStreamRefTest, IvmUtilTest, IvmNormalizeMtmvTest, ExplainRefreshIvmCommandTest - Regression test mtmv_p0/ivm
Split the IVM delta rewrite implementation into linear, outer join, and aggregate handlers coordinated by a visitor and command builder. Key changes: - Replace the strategy inheritance chain with independent handler classes. - Move shared rewrite utilities into IvmDeltaRewriteHelper. - Add IvmDeltaCommandBuilder for wrapping rewritten plans into insert commands. - Keep project dispatch in the visitor while routing aggregate-root projects to the aggregate handler. - Update IVM tests and CreateMTMVCommandTest coverage for filter above outer join. Unit Test: - IVM FE unit test set, including CreateMTMVCommandTest. - IvmDeltaRewriterTest and IvmLinearDeltaHandlerTest. - mtmv_p0/ivm regression suites.
…ling Move the nullable-side UNION ALL rejection down to the OlapScan check, and track union nesting explicitly in the normalize context so outer-join rules can reason about the actual scan position. Key changes: - Add an inside-union flag to NormalizeContext and reuse a child context while visiting UNION children. - Reject non-excluded OlapScan under nullable-side UNION ALL during scan normalization instead of at the union node. - Add regression coverage for UNION DISTINCT and project-wrapped scan cases under nullable-side outer joins. Unit Test: - tools/fast-compile-fe.sh --test IvmNormalizeMtmvJoinTest.java - mvn test for IVM FE UT subset including IvmNormalizeMtmvJoinTest, IvmNormalizeMtmvTest, and CreateMTMVCommandTest - IvmNormalizeMtmvTest and IvmNormalizeMtmvJoinTest passed; CreateMTMVCommandTest passed; remaining failures were environment-related in other tests
Fix FE compilation and checkstyle issues left by the IVM branch rebase onto upstream master. Key changes: - Add missing Consumer imports for MTMV command execution customizers. - Replace removed containsUniqueFunction API with containsVolatileExpression for IVM outer join key validation. - Update MTMV refresh test code for the RefreshMode-based RefreshMTMVInfo constructor and import order. Unit Test: - build.sh --fe
Support IVM rewrite for full outer joins and clean up outer join retained/null-side validation. Key changes: - Normalize full outer joins with retained/null-side row-id checks and null-side nested-outer-join rejection. - Refactor outer join delta rewrite to handle left, right, and full outer joins with shared retained/null-side repair logic. - Add FE unit and regression coverage for full outer join behavior under current mock-full-scan delta semantics. Unit Test: - IvmNormalizeMtmvJoinTest - IvmOuterJoinDeltaHandlerTest - CreateMTMVCommandTest - IVM FE unit test batch - mtmv_p0/ivm regression suite - FE build and fast FE compile
Contributor
Author
|
run buildall |
Contributor
TPC-H: Total hot run time: 31203 ms |
Update the MTMV refresh unit test to validate FE planning and translation instead of executing INSERT or refresh work that requires a real BE. Also make IVM rewrite session state cleanup local to the tests that enable it. Key changes: - Remove real data INSERT and refresh execution from the one-row complete MTMV test. - Verify the generated refresh plan translates to an OlapTableSink fragment. - Reset enableIvmNormalRewrite in finally blocks for tests that enable it. Unit Test: - UpdateMvByPartitionCommandTest - IVM FE unit test batch - mtmv_p0/ivm regression suites
Contributor
TPC-DS: Total hot run time: 169980 ms |
Contributor
Author
|
run buildall |
Contributor
TPC-H: Total hot run time: 31264 ms |
Contributor
TPC-DS: Total hot run time: 169846 ms |
Contributor
FE Regression Coverage ReportIncrement line coverage |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What problem does this PR solve?
Issue Number: close #xxx
Related PR: #xxx
Problem Summary:
Release note
None
Check List (For Author)
Test
Behavior changed:
Does this need documentation?
Check List (For Reviewer who merge this PR)