[Enhancement] Materialized views support refresh granularity splits #12926

Astralidea · 2022-11-03T08:45:55Z

What type of PR is this：

Which issues of this PR fixes ：

Fixes #12925

Problem Summary(Required) ：

Checklist:

I have added test cases for my bug fix or my new feature
I have added user document for my new feature or new function

Astralidea · 2022-11-03T13:27:54Z

run starrocks_admit_test

Astralidea · 2022-11-04T02:28:21Z

run starrocks_admit_test

ABingHuang · 2022-11-07T03:48:19Z

fe/fe-core/src/main/java/com/starrocks/common/util/PropertyAnalyzer.java

+                throw new AnalysisException("Partition Refresh Number: " + e.getMessage());
+            }
+            if (partitionRefreshNumber <= 0) {
+                throw new AnalysisException("Partition Refresh Number should larger than 0.");


how to unset refresh number?
what if the user do not specify the refresh number, what is the default logic?

This feature will do in Support alter materialized view properties or later PR

ABingHuang · 2022-11-07T03:56:20Z

...re/src/main/java/com/starrocks/scheduler/PartitionBasedMaterializedViewRefreshProcessor.java

+    }
+
+    @VisibleForTesting
+    public void filterPartitionByRefreshNumber(Set<String> partitionsToRefresh, MaterializedView materializedView) {


have you test if one source base table partition has two target mv partitions can run successfully?

In fact, this configuration has nothing to do with the source table, only the number of partitions of the materialized view table.

Astralidea · 2022-11-07T07:38:29Z

run starrocks_admit_test

Astralidea · 2022-11-07T09:25:11Z

run starrocks_admit_test

Astralidea · 2022-11-07T11:59:52Z

run starrocks_admit_test

Astralidea · 2022-11-07T12:42:00Z

run starrocks_fe_unittest

Astralidea · 2022-11-07T12:42:11Z

run starrocks_admit_test

Astralidea · 2022-11-07T13:08:06Z

run starrocks_fe_unittest

mofeiatwork · 2022-11-08T00:18:37Z

@Mergifyio rebase

mergify · 2022-11-08T00:19:24Z

rebase

❌ Base branch update has failed

Git reported the following error:

Rebasing (1/6)
error: could not apply 2fecef310... Materialized views support refresh granularity splits
Resolve all conflicts manually, mark them as resolved with
"git add/rm <conflicted_files>", then run "git rebase --continue".
You can instead skip this commit: run "git rebase --skip".
To abort and get back to the state before "git rebase", run "git rebase --abort".
Could not apply 2fecef310... Materialized views support refresh granularity splits
Auto-merging fe/fe-core/src/main/java/com/starrocks/server/LocalMetastore.java
Auto-merging fe/fe-core/src/main/java/com/starrocks/scheduler/PartitionBasedMaterializedViewRefreshProcessor.java
Auto-merging fe/fe-core/src/main/java/com/starrocks/common/util/PropertyAnalyzer.java
Auto-merging fe/fe-core/src/main/java/com/starrocks/catalog/TableProperty.java
CONFLICT (content): Merge conflict in fe/fe-core/src/main/java/com/starrocks/catalog/TableProperty.java
Auto-merging fe/fe-core/src/main/java/com/starrocks/catalog/MaterializedView.java

err-code: AFE3D

Astralidea · 2022-11-08T01:11:20Z

run starrocks_admit_test

sonarcloud · 2022-11-08T01:18:23Z

Kudos, SonarCloud Quality Gate passed!

0 Bugs
0 Vulnerabilities
0 Security Hotspots
0 Code Smells

0.0% Coverage
0.0% Duplication

wanpengfei-git · 2022-11-08T01:18:51Z

[FE PR Coverage Check]

😞 fail : 69 / 113 (61.06%)

file detail

	path	covered_line	new_line	coverage	not_covered_line_detail
🔵	com/starrocks/alter/Alter.java	0	1	00.00%	[277]
🔵	com/starrocks/common/util/PropertyAnalyzer.java	0	12	00.00%	[202, 210, 218, 219, 221, 222, 223, 224, 225, 226, 228, 230]
🔵	com/starrocks/scheduler/ExecuteOption.java	0	1	00.00%	[20]
🔵	com/starrocks/server/LocalMetastore.java	1	5	20.00%	[3320, 3321, 3322, 3323]
🔵	com/starrocks/catalog/MaterializedView.java	3	6	50.00%	[638, 639, 640]
🔵	com/starrocks/scheduler/PartitionBasedMaterializedViewRefreshProcessor.java	42	64	65.62%	[175, 187, 227, 228, 229, 231, 236, 237, 238, 239, 240, 241, 242, 243, 245, 246, 247, 248, 250, 251, 252, 689]
🔵	com/starrocks/catalog/TableProperty.java	10	11	90.91%	[145]
🔵	com/starrocks/qe/StmtExecutor.java	3	3	100.00%	[]
🔵	com/starrocks/scheduler/MvTaskRunContext.java	9	9	100.00%	[]
🔵	com/starrocks/clone/DynamicPartitionScheduler.java	1	1	100.00%	[]

Materialized views support refresh granularity splits

2fecef3

wanpengfei-git added the be-build label Nov 3, 2022

Fix UT.

e67e19a

wanpengfei-git added be-build and removed be-build labels Nov 3, 2022

Add UT.

b8539d2

wanpengfei-git added be-build and removed be-build labels Nov 3, 2022

Fix.

4d9ff47

wanpengfei-git added be-build and removed be-build labels Nov 3, 2022

Fix UT.

8442764

wanpengfei-git added be-build and removed be-build labels Nov 4, 2022

Merge branch 'StarRocks:main' into SR-13379

7f7d0b3

wanpengfei-git added be-build and removed be-build labels Nov 4, 2022

Merge branch 'main' into SR-13379

26a3af4

wanpengfei-git added be-build and removed be-build labels Nov 7, 2022

Astralidea added the sig/materialized view label Nov 7, 2022

ABingHuang reviewed Nov 7, 2022

View reviewed changes

gengjun-git previously approved these changes Nov 7, 2022

View reviewed changes

Astralidea enabled auto-merge (squash) November 7, 2022 07:28

Youngwb approved these changes Nov 7, 2022

View reviewed changes

Youngwb previously approved these changes Nov 7, 2022

View reviewed changes

Merge branch 'StarRocks:main' into SR-13379

4d337e5

wanpengfei-git added be-build and removed be-build labels Nov 7, 2022

Fix

78cea18

Astralidea dismissed stale reviews from Youngwb and gengjun-git via 78cea18 November 7, 2022 08:14

wanpengfei-git added be-build and removed be-build labels Nov 7, 2022

gengjun-git approved these changes Nov 7, 2022

View reviewed changes

Youngwb approved these changes Nov 7, 2022

View reviewed changes

Merge branch 'StarRocks:main' into SR-13379

835ac32

wanpengfei-git added be-build and removed be-build labels Nov 7, 2022

Merge branch 'StarRocks:main' into SR-13379

4f6ecd0

wanpengfei-git added be-build and removed be-build labels Nov 8, 2022

Astralidea merged commit 7e59eca into StarRocks:main Nov 8, 2022

Astralidea added the behavior_changed label Nov 22, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Enhancement] Materialized views support refresh granularity splits #12926

[Enhancement] Materialized views support refresh granularity splits #12926

Astralidea commented Nov 3, 2022

Astralidea commented Nov 3, 2022

Astralidea commented Nov 4, 2022

ABingHuang Nov 7, 2022

Astralidea Nov 7, 2022

ABingHuang Nov 7, 2022

Astralidea Nov 7, 2022

Astralidea commented Nov 7, 2022

Astralidea commented Nov 7, 2022

Astralidea commented Nov 7, 2022

Astralidea commented Nov 7, 2022

Astralidea commented Nov 7, 2022

Astralidea commented Nov 7, 2022

mofeiatwork commented Nov 8, 2022

mergify bot commented Nov 8, 2022

Astralidea commented Nov 8, 2022

sonarcloud bot commented Nov 8, 2022

wanpengfei-git commented Nov 8, 2022

[Enhancement] Materialized views support refresh granularity splits #12926

[Enhancement] Materialized views support refresh granularity splits #12926

Conversation

Astralidea commented Nov 3, 2022

What type of PR is this：

Which issues of this PR fixes ：

Problem Summary(Required) ：

Checklist:

Astralidea commented Nov 3, 2022

Astralidea commented Nov 4, 2022

ABingHuang Nov 7, 2022

Choose a reason for hiding this comment

Astralidea Nov 7, 2022

Choose a reason for hiding this comment

ABingHuang Nov 7, 2022

Choose a reason for hiding this comment

Astralidea Nov 7, 2022

Choose a reason for hiding this comment

Astralidea commented Nov 7, 2022

Astralidea commented Nov 7, 2022

Astralidea commented Nov 7, 2022

Astralidea commented Nov 7, 2022

Astralidea commented Nov 7, 2022

Astralidea commented Nov 7, 2022

mofeiatwork commented Nov 8, 2022

mergify bot commented Nov 8, 2022

❌ Base branch update has failed

Astralidea commented Nov 8, 2022

sonarcloud bot commented Nov 8, 2022

wanpengfei-git commented Nov 8, 2022

[FE PR Coverage Check]

file detail