[fix](ivm) Invalidate the baseline when a column used by the MV is dropped - #67837
Merged
yujun777 merged 4 commits intoSep 11, 2026
Merged
Conversation
…opped A light schema change only touches metadata and emits no row binlog. When a column the MV uses is dropped - and typically re-added under the same name right after - the incremental refresh therefore consumes an empty delta and reports SUCCESS while the MV keeps the rows computed under the old column, so the staleness is silent. The change did reach the MTMV hook, but that only moved the MV status to SCHEMA_CHANGE, which merely re-analyses the MV query on the next refresh. By then the column is back under the same name, so the analysis succeeds again and nothing blocks the incremental path. The IVM baseline barrier was never raised here: it is only set for REPLACE TABLE, REPLACE PARTITION and partition changes. Re-analyse the MV query on the alterTable path, right after the alter was applied, and invalidate the baseline when the query no longer binds. Dropping or renaming a column the MV uses makes the query unanalysable, so this doubles as the dependency check: dropping a column the MV does not reference leaves the incremental path untouched. On an IVM base table only DROP COLUMN and RENAME TABLE can change a referenced column at all (row binlog tables reject MODIFY COLUMN, RENAME COLUMN and REORDER COLUMNS), so those two operations are the whole scope. A dropped base table needs no handling - the IVM stream records the base table id and stops being usable, which already fails the refresh. The check analyses with a context of its own rather than the one that issued the alter, because the analysis reuses and closes the statement context it is handed.
yujun777
requested review from
gavinchou,
luwei16 and
morrySnow
as code owners
September 11, 2026 03:32
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
Contributor
Author
|
run buildall |
Contributor
FE UT Coverage ReportIncrement line coverage |
Renaming an IVM base table invalidated the IVM baseline: the query-usability probe cannot succeed after a rename, because the MV query still spells the old table name. That invalidation is invisible to the user -- the refresh resolves its base tables from the query before it ever checks the baseline, so it reports the missing table either way -- but the flag it sets is durable: renaming the table back left every strict INCREMENTAL refresh rejected until a COMPLETE one had been run. Key changes: - Skip the query-usability probe when the alter is a rename, detected by the base table name changing. Only RenameTableOp and ReplaceTableOp fill newBaseTableInfo (Alter#processAlterTable), and ReplaceTableOp keeps its isReplace branch, so every other op still probes and a dropped referenced column is still caught. Unit Test - IvmBaselineRebuildTest#testRenameTableBackKeepsIncrementalRefreshStartable (new) - IvmBaselineRebuildTest#testRenameTableDoesNotMarkBaselineRebuild - IvmBaselineRebuildTest (whole class)
Contributor
Author
|
run buildall |
…on cloud The suite creates its own tables and reads back only its own task status, so it does not touch cluster-wide state: declaring nonConcurrent takes it out of the parallel pool and slows the whole run down for nothing. The cloud guard came in with the older IVM suites without a stated reason, and both cases this suite covers were reported from a cloud deployment, so skipping cloud hid exactly the regression it exists to catch. Key changes: - Drop the nonConcurrent group and the isCloudMode early return from test_ivm_drop_referenced_column_baseline_rebuild, leaving the suite body unchanged. Unit Test - Not run locally: only the suite group and the cloud guard were removed.
Contributor
Author
|
run buildall |
…golden file The suite compared SQL result sets with assertEquals on List.toString(), which hides the diff when a query breaks and ties the assertion to the list formatting. Both queries now go through order_qt_* and their expected rows live in the suite's .out file. Key changes: - Replace the two mvRows assertEquals checks with order_qt_mv_rows_baseline and order_qt_mv_rows_after_aba, and drop the now unused mvRows closure. - Add regression-test/data/mtmv_p0/ivm/test_ivm_drop_referenced_column_baseline_rebuild.out. Unit Test - test_ivm_drop_referenced_column_baseline_rebuild: run with -forceGenOut to produce the golden file, then again without it to confirm the comparisons hold.
Contributor
Author
|
run buildall |
Contributor
Author
|
run feut |
morrySnow
approved these changes
Sep 11, 2026
luwei16
approved these changes
Sep 11, 2026
Contributor
Author
|
run vault_p0 |
Contributor
Author
|
run external |
Contributor
TPC-H: Total hot run time: 16849 ms |
Contributor
TPC-DS: Total hot run time: 83554 ms |
Contributor
ClickBench: Total hot run time: 14.9 s |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What problem does this PR solve?
Trace issue: #65418
Problem Summary:
A light schema change on an IVM base table (
DROP COLUMN, usually followed by anADD COLUMNwith the same name) only changes metadata and emits no row binlog. When the dropped column is one the MV uses, the nextREFRESH ... INCREMENTALconsumes an empty delta and reports SUCCESS, while the MV silently keeps the rows that were computed under the old column.Dropping the column did reach the MTMV hook, but that only moved the MV status to
SCHEMA_CHANGE. That state merely re-analyses the MV query on the next refresh - and by then the column is usually back under the same name, so the analysis succeeds and the incremental refresh proceeds. The IVM baseline barrier (requireCompleteBaselineRebuild) was never raised on this path: it is only set forREPLACE TABLE,REPLACE PARTITIONand partition changes.What this PR does
Re-analyse the MV query on the
alterTablepath, right after the alter was applied, and invalidate the IVM baseline when the query no longer binds.This needs no lineage machinery: dropping or renaming a column the MV uses makes the query unanalysable, so the failure itself is the dependency signal - and dropping a column the MV does not reference leaves the incremental path untouched.
Scope: on an IVM base table only
DROP COLUMNandRENAME TABLEcan change a referenced column at all (row binlog tables rejectMODIFY COLUMN,RENAME COLUMNandREORDER COLUMNS), so those two operations are the whole surface. A dropped base table needs no handling - the IVM stream records the base table id and stops being usable, which already fails the refresh.The analysis runs in a context of its own rather than the session that issued the alter, because the underlying
analyzeQueryWithSqlreuses and closes the statement context it is handed.Release note
None
Check List (For Author)
Test
Behavior changed:
REFRESH ... INCREMENTALis rejected withIVM baseline rebuild is pending ... run an AUTO or COMPLETE refresh firstinstead of silently reporting SUCCESS. Dropping a column the MV does not use is unaffected.Does this need documentation?