Skip to content

[fix](mv) Distinguish partition compensation from UNION ALL rewrite - #66445

Open
foxtail463 wants to merge 1 commit into
apache:masterfrom
foxtail463:fix/mv-partition-compensation-detection
Open

[fix](mv) Distinguish partition compensation from UNION ALL rewrite#66445
foxtail463 wants to merge 1 commit into
apache:masterfrom
foxtail463:fix/mv-partition-compensation-detection

Conversation

@foxtail463

Copy link
Copy Markdown
Contributor

Problem Summary:
MV rewrite treated removing invalid MV partitions as requiring UNION ALL, causing
valid rewrites to be rejected.

Solution:
Separate partition compensation detection from base-table partition union requirements.
Only check UNION ALL capability when base-table compensation is needed.

@hello-stephen

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@foxtail463

Copy link
Copy Markdown
Contributor Author

run buildall

@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-H: Total hot run time: 28777 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 5ee9e8993cf27e16bcb5a31fe9521193c8b6616b, data reload: false

------ Round 1 ----------------------------------
============================================
q1	17657	4008	3927	3927
q2	2050	318	197	197
q3	10275	1400	790	790
q4	4682	482	340	340
q5	7485	881	544	544
q6	189	182	144	144
q7	751	812	610	610
q8	9329	1527	1547	1527
q9	5352	4106	4065	4065
q10	6730	1620	1333	1333
q11	505	358	332	332
q12	729	574	462	462
q13	18087	3303	2732	2732
q14	262	255	239	239
q15	q16	733	731	654	654
q17	926	976	1015	976
q18	6556	5623	5590	5590
q19	1305	1286	1022	1022
q20	801	699	562	562
q21	5864	2574	2432	2432
q22	430	357	299	299
Total cold run time: 100698 ms
Total hot run time: 28777 ms

----- Round 2, with runtime_filter_mode=off -----
============================================
q1	4253	4159	4163	4159
q2	297	314	206	206
q3	4507	4903	4337	4337
q4	2187	2275	1406	1406
q5	4246	4092	4066	4066
q6	222	172	129	129
q7	1663	1565	1423	1423
q8	2509	2335	2102	2102
q9	7453	7465	7554	7465
q10	4307	4228	3828	3828
q11	562	425	376	376
q12	703	744	517	517
q13	3197	3409	2879	2879
q14	303	312	287	287
q15	q16	697	710	652	652
q17	1288	1271	1262	1262
q18	12178	11047	11886	11047
q19	1156	1149	1185	1149
q20	2222	2181	1934	1934
q21	6132	4805	4777	4777
q22	571	475	405	405
Total cold run time: 60653 ms
Total hot run time: 54406 ms

@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-DS: Total hot run time: 166596 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 5ee9e8993cf27e16bcb5a31fe9521193c8b6616b, data reload: false

query5	4301	588	449	449
query6	469	220	212	212
query7	4883	589	355	355
query8	319	163	146	146
query9	8753	4067	4065	4065
query10	479	346	296	296
query11	5828	2177	2020	2020
query12	149	102	97	97
query13	1245	620	433	433
query14	6108	4288	4008	4008
query14_1	3843	3817	3823	3817
query15	206	194	180	180
query16	1013	501	478	478
query17	968	735	584	584
query18	2458	467	357	357
query19	209	190	174	174
query20	106	102	103	102
query21	235	157	138	138
query22	13012	12996	12747	12747
query23	15914	15051	14660	14660
query23_1	15515	15094	15065	15065
query24	8154	1714	1263	1263
query24_1	1267	1253	1264	1253
query25	607	432	348	348
query26	1315	349	201	201
query27	2669	617	387	387
query28	4607	2065	2038	2038
query29	1042	605	467	467
query30	335	253	219	219
query31	1178	1106	1031	1031
query32	112	55	54	54
query33	507	298	240	240
query34	1182	1128	644	644
query35	785	761	634	634
query36	766	786	703	703
query37	152	102	93	93
query38	1848	1762	1678	1678
query39	826	807	784	784
query39_1	773	772	784	772
query40	240	155	143	143
query41	65	60	61	60
query42	90	88	90	88
query43	325	320	276	276
query44	1451	789	780	780
query45	178	169	163	163
query46	1057	1192	737	737
query47	1519	1526	1434	1434
query48	411	406	306	306
query49	587	403	288	288
query50	1141	442	363	363
query51	10433	10458	10077	10077
query52	83	91	79	79
query53	269	283	204	204
query54	278	240	232	232
query55	81	70	66	66
query56	319	292	285	285
query57	1002	1000	913	913
query58	283	257	240	240
query59	1514	1628	1389	1389
query60	294	275	260	260
query61	153	149	143	143
query62	398	326	266	266
query63	228	196	198	196
query64	2911	1041	868	868
query65	3876	3782	3775	3775
query66	1856	483	354	354
query67	27517	28076	28024	28024
query68	3272	1607	982	982
query69	406	302	251	251
query70	898	802	799	799
query71	377	330	316	316
query72	3012	2741	2384	2384
query73	838	822	441	441
query74	4607	4471	4290	4290
query75	2385	2349	2000	2000
query76	2329	1149	796	796
query77	353	371	270	270
query78	11003	11068	10589	10589
query79	1443	1197	780	780
query80	1265	545	463	463
query81	532	324	283	283
query82	653	176	134	134
query83	397	342	297	297
query84	316	170	134	134
query85	991	601	520	520
query86	407	230	227	227
query87	1978	1963	1813	1813
query88	3807	2851	2812	2812
query89	396	319	291	291
query90	1905	208	206	206
query91	204	192	164	164
query92	59	63	53	53
query93	1676	1492	952	952
query94	727	340	301	301
query95	766	593	484	484
query96	1063	759	356	356
query97	2442	2454	2303	2303
query98	199	183	184	183
query99	739	732	599	599
Total cold run time: 254293 ms
Total hot run time: 166596 ms

@hello-stephen

Copy link
Copy Markdown
Contributor
ClickBench: Total hot run time: 23.74 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 5ee9e8993cf27e16bcb5a31fe9521193c8b6616b, data reload: false

query1	0.01	0.01	0.00
query2	0.09	0.05	0.05
query3	0.25	0.14	0.13
query4	1.60	0.14	0.14
query5	0.24	0.21	0.21
query6	1.16	0.81	0.85
query7	0.04	0.00	0.00
query8	0.05	0.03	0.03
query9	0.37	0.30	0.31
query10	0.55	0.53	0.54
query11	0.18	0.13	0.14
query12	0.17	0.14	0.13
query13	0.46	0.45	0.46
query14	0.99	1.01	1.01
query15	0.60	0.57	0.57
query16	0.32	0.33	0.34
query17	1.10	1.10	1.09
query18	0.21	0.19	0.20
query19	1.98	1.99	1.97
query20	0.02	0.01	0.01
query21	15.43	0.21	0.15
query22	4.85	0.05	0.06
query23	16.13	0.30	0.12
query24	3.00	0.41	0.32
query25	0.11	0.06	0.04
query26	0.72	0.21	0.16
query27	0.04	0.03	0.04
query28	3.46	0.76	0.35
query29	12.51	4.06	3.16
query30	0.27	0.16	0.15
query31	2.77	0.56	0.32
query32	3.21	0.59	0.49
query33	3.15	3.16	3.10
query34	15.56	3.92	3.29
query35	3.23	3.19	3.24
query36	0.56	0.43	0.40
query37	0.08	0.06	0.06
query38	0.05	0.04	0.03
query39	0.04	0.02	0.02
query40	0.17	0.15	0.14
query41	0.08	0.04	0.03
query42	0.04	0.03	0.03
query43	0.05	0.04	0.03
Total cold run time: 95.9 s
Total hot run time: 23.74 s

@hello-stephen

Copy link
Copy Markdown
Contributor

FE Regression Coverage Report

Increment line coverage 20.00% (6/30) 🎉
Increment coverage report
Complete coverage report

@seawinde

seawinde commented Aug 7, 2026

Copy link
Copy Markdown
Member

/review

@github-actions

github-actions Bot commented Aug 7, 2026

Copy link
Copy Markdown
Contributor

Codex automated review failed and did not complete.

Error: All Codex review accounts are usage-limited; earliest retry is 2026-08-08T03:32:00Z.
Workflow run: https://github.com/apache/doris/actions/runs/31157893366

Please trigger /review again after that time.

@seawinde

seawinde commented Aug 7, 2026

Copy link
Copy Markdown
Member

当前单元测试只验证了 hasPartitionCompensation(),没有覆盖实际修复逻辑。建议补充回归测试:查询聚合时不包含 MV 分区列,使 canUnionRewrite() 返回 false;删除基表分区后,仅需移除失效 MV 分区且无需 UNION ALL,验证 MV rewrite 成功且结果正确。现有删除分区测试包含分区列,无法覆盖该分支。

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants