[draft](ivm) for run pipeline test - #66266
Draft
yujun777 wants to merge 226 commits into
Draft
Conversation
…ogging - Require non-null detailMessage in IVMRefreshResult.fallback() to match IVMCapabilityResult.unsupported() contract; remove single-arg overload - Add toString() to IVMRefreshResult for log readability - Add WARN logging on all fallback paths in IVMRefreshManager with MV name - Make doRefresh() the public API; remove redundant ivmRefresh() wrapper
- Remove IVMPlanPattern, IVMPlanAnalysis, IVMPlanAnalyzer, IVMDeltaPlannerDispatcher - IVMCapabilityChecker now takes List<DeltaPlanBundle> instead of IVMPlanAnalysis - IVMRefreshManager simplified to 2 deps: capabilityChecker + deltaExecutor - Delta bundles produced by Nereids rules, retrieved via MTMVAnalyzeQueryInfo - Add analyzeDeltaBundles() hook for testability - Add ivmDeltaBundles to MTMVAnalyzeQueryInfo, populated from CascadesContext - Update tests to JUnit 5 and new interface signatures - Fix checkstyle import order in CreateTableCommandTest
- Delete IvmRewriteMtmvPlan placeholder and its test - Remove rewriteRootPlan field from CascadesContext (no longer needed) - Replace IVM_REWRITE_MTMV_PLAN with IVM_NORMALIZE_MTMV_PLAN in RuleType - Add IvmNormalizeMtmvPlan skeleton (row-id injection, avg rewrite, TODO) - Add IvmDeltaScanOnly and IvmDeltaAggRoot skeletons - Merge delta rules into single topic in Rewriter
- Add IvmAnalyzeMode enum (NONE/NORMALIZE_ONLY/FULL) to replace boolean flags - Replace enableIvmRewriteInNereids with enableIvmNormalRewrite + enableIvmDeltaRewrite - MTMVPlanUtil.analyzeQuery/analyzeQueryWithSql take IvmAnalyzeMode parameter - CreateMTMVInfo: NORMALIZE_ONLY for incremental MV, NONE otherwise - ensureMTMVQueryUsable: same mode as CREATE MV - IVMRefreshManager: FULL mode (normalize + delta) - Update IvmNormalizeMtmvPlan/IvmDeltaScanOnly/IvmDeltaAggRoot to use new session vars - Fix MTMVPlanUtilTest: JUnit5, IvmAnalyzeMode.NONE, updated CountingSessionVariable - Add testAnalyzeQueryIvmAnalyzeModeSetSessionVariables covering all 3 modes
- Add IvmContext: holds Map<Slot, isDeterministic> rowIdDeterminism + List<DeltaPlanBundle>
- Replace ivmDeltaBundles in CascadesContext with Optional<IvmContext>
- IvmNormalizeMtmvPlan: whitelist-based visitor (DefaultPlanRewriter<IvmContext>)
- visitLogicalOlapScan: inject __IVM_ROW_ID__ at index 0 via LogicalProject
- MOW: Alias(buildRowIdHash(uk...), __IVM_ROW_ID__) -> deterministic
- DUP_KEYS: Alias(UuidNumeric(), __IVM_ROW_ID__) -> non-deterministic
- MOR / AGG_KEYS: throw AnalysisException
- visitLogicalProject: propagate child row-id; throw if child has none
- visit: throw for any unwhitelisted node
- buildRowIdHash: uses murmur_hash3_64 (TODO: replace with 128-bit hash)
- MTMVPlanUtil: read delta bundles from IvmContext instead of direct field
- Tests: DUP_KEYS, MOW (deterministic), MOR (throws), AGG_KEYS (throws),
project propagation, unsupported node, gate disabled
- IvmNormalizeMtmvPlan: whitelist LogicalResultSink, prepend row-id; extract hasRowIdInOutputs/prependRowId helpers - ColumnDefinition: add newIvmRowIdColumnDefinition with mv_ prefix - MTMVPlanUtil: prepend row-id ColumnDefinition at index 0; reset IVM session vars in finally block to prevent test leakage - BaseViewInfo: extract static rewriteProjectsToUserDefineAlias overload - CreateMTMVInfo: fix rewriteQuerySql to snapshot/restore rewrite map and call alias rewrite when simpleColumnDefinitions present - CreateTableCommandTest: add 4 IVM UTs covering scan, project-scan, no-alias, alias rewrite, and column count mismatch
- CreateMTMVInfo: set UNIQUE_KEYS + enable_unique_key_merge_on_write=true
for INCREMENTAL refresh MVs; reject user-specified key columns
- MTMVPlanUtil.analyzeKeys: return new List instead of mutating the
immutable input list; throw if IVM row-id column not found in columns
- MTMVPlanUtil.analyzeQuery: only reset IVM session vars in finally block
for modes that actually set them (NORMALIZE_ONLY resets NORMAL only,
FULL resets both, NONE resets neither)
- MTMVPlanUtilTest: add 4 new UTs covering UNIQUE_KEYS+MOW assertion,
DUP_KEYS for non-IVM, and rejection of user-specified UNIQUE/DUP keys
- CountingSessionVariable: count only enabling ("true") setVarOnce calls
…aRewriter Move IVM delta plan generation out of Nereids rewrite rules into an external IvmDeltaRewriter that will be called by IVMRefreshManager. IvmNormalizeMtmvPlan now stores the normalized plan in IvmContext so IVMRefreshManager can retrieve it for delta rewriting. - Add normalizedPlan field to IvmContext, store after normalization - Add ivmNormalizedPlan field to MTMVAnalyzeQueryInfo - Delete IvmDeltaScanOnly, IvmDeltaAggRoot, IvmAnalyzeMode - Remove IVM_DELTA_SCAN_ONLY/IVM_DELTA_AGG_ROOT from RuleType/Rewriter - Remove ENABLE_IVM_DELTA_REWRITE session variable - Remove deltaCommandBundles from IvmContext - Replace IvmAnalyzeMode enum with boolean enableIvmNormalize - Create skeleton IvmDeltaRewriter + IvmDeltaRewriteContext - Rewrite IVMRefreshManager.analyzeDeltaCommandBundles to use normalized plan (returns empty bundles for now, triggers fallback)
IvmDeltaRewriter no longer extends DefaultPlanRewriter. It now validates the normalized plan is a supported scan-only or project-scan pattern, extracts the base table, and produces an INSERT INTO mv command wrapped in a DeltaCommandBundle. IvmDeltaRewriteContext gains a ConnectContext field, and IVMRefreshManager.analyzeDeltaCommandBundles is wired to call the rewriter.
…es to concrete classes IVMDeltaExecutor now contains real execution logic following the MTMVTask.exec() pattern: creates ConnectContext/StatementContext/StmtExecutor, runs the command, and checks query state. IVMCapabilityChecker returns ok() by default. IVMRefreshManager uses a no-arg public constructor, instantiating both collaborators internally, with a @VisibleForTesting constructor for injection.
For INCREMENTAL MVs, attempt IVM refresh first via IVMRefreshManager. On success, return early and skip partition-based refresh. On fallback, log the reason and continue with existing refresh path.
…eltaExecutor Extract common command execution boilerplate shared by MTMVTask.exec() and IVMDeltaExecutor.executeBundle() into MTMVPlanUtil.executeCommand(). This also adds the missing audit logging to IVM delta execution.
Keep the hidden IVM row id in refresh planning and exclude it from MV nondeterministic checks. Adjust exchange fragment output expr handling for incremental refresh, rename the MV-specific collector, and add FE UT plus mtmv regression coverage. Tests: ./run-fe-ut.sh --run org.apache.doris.nereids.rules.rewrite.IvmNormalizeMtmvPlanTest,org.apache.doris.nereids.trees.plans.PlanVisitorTest,org.apache.doris.nereids.trees.plans.commands.UpdateMvByPartitionCommandTest Tests: ./run-regression-test.sh --run -d mtmv_p0 -s test_ivm_basic_mtmv
Ensure the root fragment always rewrites output exprs from the final physical plan outputs so aggregate and TopN plans do not keep stale SlotRefs. Add FE/unit and regression coverage for MTMV hidden row-id changes after complete refresh.
Disable table-sink MV rewrite in the MTMV refresh execution context so refresh planning cannot rewrite back to the target MV. Add a SessionVariable setter and extend UpdateMvByPartitionCommandTest to assert both MV rewrite switches are disabled for the refresh executor. Test: ./run-fe-ut.sh --run org.apache.doris.nereids.trees.plans.commands.UpdateMvByPartitionCommandTest
Move IVM normalization after sink binding so incremental MTMV inserts keep hidden columns aligned with bound olap sink outputs and target slots. Tests: - bash ./run-fe-ut.sh --run org.apache.doris.nereids.trees.plans.CreateTableCommandTest - bash ./run-fe-ut.sh --run org.apache.doris.mtmv.MTMVTest,org.apache.doris.nereids.rules.rewrite.IvmNormalizeMtmvPlanTest
… normalization - Remove dead condition in BindSink.getColumnToChildOutput: the second clause of the IVM hidden column skip guard was always true by definition of missingIvmHiddenColumns (columns guaranteed absent from child output) - Add integration test testSinkWithPlaceholderChildReplacesRowIdAndPreservesExprId covering the BindSink placeholder -> IvmNormalizeMtmvPlan replacement pipeline - Fix checkstyle import order violations in BindSink, IvmNormalizeMtmvPlan, MTMVTest, and IvmNormalizeMtmvPlanTest introduced in the previous commit
### What problem does this PR solve? Issue Number: None Related PR: None Problem Summary: Rename the IVM MTMV normalize rule class, its RuleType constant, and the matching FE unit test to remove the stale Plan suffix and keep the analyzer registration aligned with the new symbol names. ### Release note None ### Check List (For Author) - Test: FE unit test via bash ./run-fe-ut.sh --run org.apache.doris.nereids.rules.rewrite.IvmNormalizeMtmvTest - Behavior changed: No - Does this need documentation: No
Rename partition refresh methods to make the PARTITIONS/COMPLETE refresh path explicit. Key changes: - Rename the retry wrapper to refreshPartitionsWithRetry - Rename the refresh implementation to refreshPartitions - Update MTMVTaskTest reflection call Unit Test: - MTMVTaskTest: 31 tests passed
Replacing an incremental materialized view with `swap=false` removes the old
MV but leaves its hidden table streams registered in the database and
`TableStreamManager`. The orphaned metadata also remains after journal replay.
**Root cause:** `Alter.replaceTableInternal()` bypasses the normal MTMV drop
path, so the IVM stream cleanup in `InternalCatalog.unprotectDropTable()` is
not invoked.
**Change Summary:**
| File | Change Description |
|------|-------------------|
| `Alter.java` | Clean up the replaced IVM streams for non-swap replacement on both Leader handling and journal replay |
| `InternalCatalog.java` | Extract reusable IVM stream cleanup and hold each stream write lock during deletion |
| `AlterMTMVTest.java` | Cover non-swap cleanup, replay cleanup, and swap preservation |
| Regression test | Verify the old stream is removed and the replacement stream remains |
```mermaid
graph LR
A[REPLACE MTMV] --> B{swap}
B -->|true| C[Keep both IVM streams]
B -->|false| D[Drop old IVM streams]
E[Journal replay] --> D
D --> F[Clean database and TableStreamManager]
```
Make default incremental refresh explain match pending-stream execution while providing an explicit structural mode for exhausted streams. Key changes: - Add EXPLAIN REFRESH ... INCREMENTAL WITH ALL STREAMS syntax. - Keep ordinary REFRESH and partition refresh syntax unchanged. - Propagate exhausted-stream inclusion through IVM rewrite contexts. - Add FE parser/unit and mtmv_p0/ivm regression coverage. Unit Test: - ExplainRefreshMTMVCommandTest: 23 tests passed. - test_ivm_explain_refresh: 1 suite passed.
Incremental materialized views used the materialized view ID and the base table's short name to name internal streams. When one materialized view referenced same-named tables in different databases or catalogs, both inputs resolved to the same stream name. Creating the later stream replaced the earlier one, so incremental refresh lost one input.
Root cause: `IvmUtil.streamName()` did not include the base table catalog and database. The create, lookup, alter, and drop paths therefore could not uniquely identify or verify the stream belonging to a fully qualified base table.
Change Summary:
| Area | Change Description |
|------|--------------------|
| `IvmUtil` | Generate a fixed-length stream suffix from the length-prefixed catalog, database, and table path using SHA-256, a 128-bit prefix, and lowercase Base36. |
| Stream lifecycle | Use the fully qualified base table path for stream creation, lookup, alter cleanup, and MV cleanup; verify persisted stream ownership before using or deleting a candidate. |
| Tests | Cover deterministic naming, ownership conflicts, cross-database same-name base tables, incremental refresh, and internal stream access guards. |
The fixed-length digest keeps names within the table-name limit while separating same-named tables across database and catalog boundaries. Persisted ownership remains the source of truth: a hash collision or an unrelated object with the candidate name causes creation to fail, while cleanup leaves that object untouched.
```mermaid
flowchart LR
A[Catalog, database, table] --> B[Length-prefixed UTF-8 encoding]
B --> C[SHA-256, first 128 bits, Base36]
C --> D[IVM stream name]
D --> E[Create stream]
D --> F[Lookup for refresh]
D --> G[Alter or drop cleanup]
E --> H[Verify persisted base table ownership]
F --> H
G --> H
```
Record query IDs from IVM and ordinary refresh paths, including failure paths where a refresh statement has already created a query ID. Key changes: - Pass a query ID callback through IVM refresh execution and record it in MTMVTask. - Record query IDs for complete and partition refreshes while ignoring empty IDs. - Add unit and regression coverage for complete and incremental refreshes. Unit Test: - IvmRefreshManagerTest - test_ivm_task_query_id regression
Pass audit metadata and QueryId recording through IvmRefreshContext, and keep refresh audit statements consistent across incremental and partition-based refreshes. Key changes: - Move IVM context construction into each retry attempt. - Include refresh mode and qualified MV name in audit statements. - Reduce IvmRefreshContext constructors and update affected tests. Unit Test: - IvmRefreshManagerTest - MTMVTaskTest - fast-compile-fe main source validation
Add the DORIS-27275 INNER JOIN aggregate scenario where a dimension key is updated and then deleted before the next incremental refresh. Validate that the IVM result removes the old group. Key changes: - Add the IVM regression SQL scenario without a complete MV reference. - Add the generated expected output containing only the retained group. Unit Test: - test_ivm_inner_join_update_delete
Use one setupComputeGroup flow for IVM and partition refresh, and verify both the session context and recorded task metadata.
Rename refresh context, result, manager, and their tests to make their incremental-only scope explicit. Document the shared rewrite context/result and inline the fallback result factory. Key changes: - Rename IvmRefreshContext/Result/Manager to IvmIncrRefreshContext/Result/Manager. - Update all FE production and test references. - Add normalize, incremental, and full refresh scope comments. Unit Test: - FE main incremental compilation passed. - FE UT attempted with Maven cache disabled; blocked by Byte Buddy JVM attach failure in the environment.
Use the IVM-specific exception type for invalid hidden slots and missing full-refresh stream columns so IVM failure reasons are preserved consistently. Key changes:\n- Replace direct AnalysisException throws in IVM utility and full-refresh analysis rule.\n- Classify both failures as PLAN_REWRITE_FAILED.\n\nUnit Test:\n- IvmUtilTest (14 tests passed).\n- FE main incremental compilation passed.
Remove obsolete IVM visible TSO and refresh helper APIs, and simplify refresh exception propagation while keeping known fallback classification. Key changes: - Remove OlapTable.getVisibleTso mock and MTMVPlanUtil ctxCustomizer overloads. - Inline IVM refresh execution and propagate unknown exceptions directly. - Update affected MTMV task tests and Mockito stubs. Unit Test: - ./build.sh --fe --clean - Focused IVM/MTMV FE UT started after clean build; CreateMTMVCommandTest passed before the run was stopped for scope review.
Prevent IVM FE unit tests from inheriting stale thread-local ConnectContext and add regression coverage for aggregate joins with deltas from both inputs. Key changes: - Clear ConnectContext after each IVM delta test. - Reuse the test context when constructing and normalizing alias-based aggregate plans. - Add aggregate join incremental refresh regression coverage. Unit Test: - IvmDeltaRewriterTest - FE Maven validate - test_ivm_agg_join_2 regression test
Allow nested outer joins on the null side during IVM creation when explicitly enabled, and warn that the complex delta calculation can significantly reduce incremental refresh performance. Add context coverage and a regression test with two real incremental refreshes.
IVM refresh can race with base-table partition creation, causing the MV insert to fail before the MV partition is synchronized. Key changes: - Classify missing MV partition errors and retry IVM after synchronizing partitions. - Bound IVM retries and preserve existing fallback behavior for other failures. - Add regression coverage for ALTER ADD PARTITION with binlog updates. Unit Test: - MTMVTaskTest: 32 tests passed - mtmv_p0/ivm/test_ivm_partition_sync_retry: passed
Prevent background MTMV tasks from consuming execute-limited IVM fault injection points intended for a specific regression MV. Key changes: - Add MV-name filter points for partition-sync and IVM RPC failure injection. - Keep execute: 1 only on the action debug points and extract the FE predicates into helpers. - Update both IVM regression suites to configure and clean up filter/action points. Unit Test: - FE incremental compilation. - test_ivm_partition_sync_retry and test_ivm_execution_failure_no_fallback regression suites.
Allow nested outer joins on null-producing sides during IVM normalization and remove the obsolete session variable. Keep retained-side row-id determinism checks and add creation/refresh coverage for nested joins on both sides. Key changes: - Remove the null-side complexity guard and session variable. - Update IVM normalization and CREATE MTMV tests. - Add left-nested outer-join incremental refresh regression coverage. Unit Test: - CreateMTMVCommandTest - IvmNormalizeMTMVJoinTest, IvmFullRefreshMTMVTest, IvmIncrRefreshMTMVTest - test_ivm_nested_outer_join_create
Support stable row IDs and initial complete refreshes for OneRowRelation plans in IVM materialized views. Reject partitioned IVM definitions that contain OneRowRelation and add focused unit and regression coverage. Key changes: - Normalize OneRowRelation with a stable LARGEINT row ID and include it in plan signatures. - Detect OneRowRelation plans during refresh and force the initial IVM refresh to COMPLETE. - Add join, union, aggregate, and base-table mutation regression coverage. Unit Test: - MTMVTaskTest - IvmNormalizeMTMVTest - CreateMTMVCommandTest - test_ivm_one_row_relation_1 regression
Fix chained IVM refresh failures caused by missing commit TSO hidden columns and keep CREATE-time normalized plan signatures diagnosable. Key changes: - Treat commit TSO as a common IVM hidden column only when it is present in the plan or schema. - Align IVM hidden-column fallback, signature normalization, and MTMV unique-key merge-on-write analysis. - Add chained MTMV regression coverage for stream reads and incremental refreshes. Unit Test: - IvmRewriteContextTest, IvmUtilTest, and IvmPlanSignatureGeneratorTest - test_ivm_chained_mtmv_2 regression
Reject explicit incremental MTMV creation when its query reads a non-IVM MTMV, because the base MTMV refreshes by partition overwrite and cannot provide an incremental changelog. Add FE coverage for the creation failure. Key changes: - Reject non-IVM MTMV base tables during IVM normalization. - Add CreateMTMVCommandTest coverage for the invalid dependency. Unit Test: - CreateMTMVCommandTest: 93 tests passed.
Add typed IVM sequence calculation so native MIN_DELTA streams use BIGINT while the encoding can be extended to LARGEINT for binlog sequences. Propagate the selected sequence type through MTMV creation and delta rewrite, and add a LARGEINT outer-join regression case. Key changes: - Encode refresh version, delta index, binlog sequence, and operation according to the selected sequence type. - Propagate sequence type to MTMV hidden columns and normalize hidden-column defaults. - Add sequence boundary tests and the LARGEINT outer-join regression case. Unit Test: - IvmSequenceCalculatorTest - CreateMTMVCommandTest - MTMV IVM unit tests - test_ivm_outer_join_6
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
Contributor
Author
|
run buildall |
Contributor
TPC-H: Total hot run time: 29107 ms |
Contributor
TPC-DS: Total hot run time: 176900 ms |
Contributor
ClickBench: Total hot run time: 24.73 s |
Contributor
BE UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What problem does this PR solve?
Issue Number: close #xxx
Related PR: #xxx
Problem Summary:
Release note
None
Check List (For Author)
Test
Behavior changed:
Does this need documentation?
Check List (For Reviewer who merge this PR)