[fix](mtmv) Check agg/window partition keys only for tracked partition outputs#65393
Open
foxtail463 wants to merge 2 commits into
Open
[fix](mtmv) Check agg/window partition keys only for tracked partition outputs#65393foxtail463 wants to merge 2 commits into
foxtail463 wants to merge 2 commits into
Conversation
foxtail463
requested review from
924060929,
englefly,
morrySnow and
starocean999
as code owners
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
Contributor
Author
DROP DATABASE IF EXISTS mv_partition_unrelated_right_branch_repro;
CREATE DATABASE mv_partition_unrelated_right_branch_repro;
USE mv_partition_unrelated_right_branch_repro;
CREATE TABLE mv_pr_left (
sale_ord_id BIGINT NOT NULL,
reg_no VARCHAR(20),
dt DATE NOT NULL
)
DUPLICATE KEY(sale_ord_id, reg_no, dt)
PARTITION BY RANGE(dt) (
PARTITION p20260701 VALUES [('2026-07-01'), ('2026-07-02'))
)
DISTRIBUTED BY HASH(sale_ord_id) BUCKETS 1
PROPERTIES ("replication_num" = "1");
CREATE TABLE mv_pr_right (
order_id BIGINT NOT NULL,
erp VARCHAR(20),
dept_2_new VARCHAR(20),
create_time DATETIME,
dt DATE NOT NULL,
is_bypass_tp_ic_cmpl TINYINT
)
DUPLICATE KEY(order_id, erp, dept_2_new, create_time, dt)
PARTITION BY RANGE(dt) (
PARTITION p20260701 VALUES [('2026-07-01'), ('2026-07-02'))
)
DISTRIBUTED BY HASH(order_id) BUCKETS 1
PROPERTIES ("replication_num" = "1");
INSERT INTO mv_pr_left VALUES
(1, 'r1', '2026-07-01'),
(2, 'r2', '2026-07-01');
INSERT INTO mv_pr_right VALUES
(1, 'e1_old', 'd1', '2026-07-01 10:00:00', '2026-07-01', 1),
(1, 'e1_new', 'd2', '2026-07-01 11:00:00', '2026-07-01', 1),
(2, 'e2', 'd3', '2026-07-01 12:00:00', '2026-07-01', 1);
DROP MATERIALIZED VIEW IF EXISTS mv_pr_right_window_rn;
CREATE MATERIALIZED VIEW mv_pr_right_window_rn
BUILD IMMEDIATE REFRESH ON MANUAL
PARTITION BY (dt)
DISTRIBUTED BY RANDOM BUCKETS 1
PROPERTIES ("replication_num" = "1")
AS
SELECT
a.reg_no,
a.sale_ord_id,
a.dt,
r.erp,
r.dept_2_new
FROM mv_pr_left AS a
LEFT JOIN (
SELECT
order_id,
erp,
dept_2_new,
ROW_NUMBER() OVER (PARTITION BY order_id ORDER BY create_time DESC NULLS LAST) AS rn,
dt AS r_dt
FROM mv_pr_right
WHERE dt = '2026-07-01' AND is_bypass_tp_ic_cmpl = 1
) AS r
ON a.sale_ord_id = r.order_id
AND r.rn = 1
WHERE a.dt = '2026-07-01';
DROP MATERIALIZED VIEW IF EXISTS mv_pr_right_agg;
CREATE MATERIALIZED VIEW mv_pr_right_agg
BUILD IMMEDIATE REFRESH ON MANUAL
PARTITION BY (dt)
DISTRIBUTED BY RANDOM BUCKETS 1
PROPERTIES ("replication_num" = "1")
AS
SELECT
a.reg_no,
a.sale_ord_id,
a.dt,
r.max_create_time
FROM mv_pr_left AS a
LEFT JOIN (
SELECT
order_id,
MAX(create_time) AS max_create_time
FROM mv_pr_right
WHERE dt = '2026-07-01'
GROUP BY order_id
) AS r
ON a.sale_ord_id = r.order_id
WHERE a.dt = '2026-07-01';
SHOW CREATE MATERIALIZED VIEW mv_pr_right_window_rn;
SHOW CREATE MATERIALIZED VIEW mv_pr_right_agg;
|
Contributor
Author
|
run buildall |
Contributor
TPC-H: Total hot run time: 29228 ms |
Contributor
TPC-DS: Total hot run time: 179480 ms |
Contributor
ClickBench: Total hot run time: 24.96 s |
Contributor
FE Regression Coverage ReportIncrement line coverage |
Contributor
Author
|
run buildall |
Contributor
FE Regression Coverage ReportIncrement line coverage |
Contributor
TPC-H: Total hot run time: 29661 ms |
morrySnow
reviewed
Jul 9, 2026
morrySnow
left a comment
morrySnow
left a comment
Contributor
There was a problem hiding this comment.
Thank you for this fix! The approach of gating partition-key validation on whether the current plan output actually contains a tracked partition column is sound and correctly handles the described scenario (aggregate/window/PartitionTopN on a branch that doesn't carry the partition column).
I have one inline comment:
PartitionIncrementMaintainer.java, new visitLogicalPartitionTopN method: The fail-reason string "window partition sets doesn't contain the target partition" appears to be copy-pasted from visitLogicalWindow (line 438). In this context, the node being checked is a LogicalPartitionTopN, not a window — the message should reference "partition topN" or similar instead of "window". This would be misleading during debugging when a LogicalPartitionTopN triggers a partition-tracking failure.
Everything else looks good — the refactoring of checkWindowPartition to delegate to checkPartitionKeysContainPartitionToCheck is clean, the new planOutputContainsPartitionColumnToCheck guard is well-commented and applied consistently across all three visitor methods, and the regression test changes correctly reflect the newly-accepted MV patterns.
Contributor
TPC-DS: Total hot run time: 181432 ms |
Contributor
ClickBench: Total hot run time: 24.93 s |
morrySnow
reviewed
Jul 9, 2026
| public Void visitLogicalPartitionTopN(LogicalPartitionTopN<? extends Plan> partitionTopN, | ||
| PartitionIncrementCheckContext context) { | ||
| if (!planOutputContainsPartitionColumnToCheck(partitionTopN, context)) { | ||
| return super.visitLogicalPartitionTopN(partitionTopN, context); |
Contributor
There was a problem hiding this comment.
The error message says "window partition sets" but this method handles LogicalPartitionTopN, not a window. Consider changing to "PartitionTopN partition keys does not contain the target partition" to avoid misleading debug output.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem Summary:
MV partition tracking could reject valid partitioned MVs when an aggregate, window, or rewritten PartitionTopN exists on a branch that does not carry the MV partition column. A typical case is a left join where the MV partitions by the left table column, while the right subquery has row_number() or group by on unrelated keys.
Solution:
Gate aggregate/window/partition-topn partition-key validation by whether the current plan output contains a tracked MV partition slot. Add LogicalPartitionTopN handling for rewritten row_number ... rn = 1