Skip to content

[fix](mtmv) Fix compensate union wrongly when direct query is empty relation (#51700) #51898

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 1 commit into
base: branch-3.0
Choose a base branch
from
Open

[fix](mtmv) Fix compensate union wrongly when direct query is empty relation (#51700) #51898

wants to merge 1 commit into from
+130 −115

Conversation

seawinde
Copy link
Contributor

@seawinde seawinde commented Jun 18, 2025 •
edited
Loading

commitId: 85b7231
pr: #51700

What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@seawinde
[fix](mtmv) Fix compensate union wrongly when direct query is empty r…
e9b1f7a 
…elation (apache#51700)

if query is as following which contains external table, if some
partition in external table is empty, when mv partiton is invalid, this
would cause compensate many union all empty relation
```sql
SELECT * FROM ${catalogName}.`test_paimon_spark`.test_tb_mix_format a
left join ${tableName} b on a.id=b.user_id;
```
@seawinde seawinde requested a review from dataroaring as a code owner June 18, 2025 11:35
@seawinde
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 40096 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit e9b1f7a5350e7c1dafb97b72b79bdc1456fae306, data reload: false

------ Round 1 ----------------------------------
q1	17579	6813	6602	6602
q2	2065	166	171	166
q3	10560	1070	1163	1070
q4	10572	806	811	806
q5	7746	2908	2865	2865
q6	216	133	130	130
q7	981	621	602	602
q8	9356	1982	2011	1982
q9	6694	6420	6427	6420
q10	7021	2279	2247	2247
q11	477	254	264	254
q12	393	213	203	203
q13	17786	2987	2997	2987
q14	233	210	202	202
q15	504	461	454	454
q16	486	366	373	366
q17	986	555	592	555
q18	7313	6758	6786	6758
q19	1382	1088	1083	1083
q20	466	208	206	206
q21	3920	3159	3189	3159
q22	1068	1002	979	979
Total cold run time: 107804 ms
Total hot run time: 40096 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6563	6544	6567	6544
q2	332	236	229	229
q3	2938	2735	2860	2735
q4	1987	1766	1792	1766
q5	5810	5758	5744	5744
q6	206	126	124	124
q7	2187	1812	1776	1776
q8	3350	3571	3558	3558
q9	8962	8819	8936	8819
q10	3557	3560	3575	3560
q11	600	495	518	495
q12	804	599	616	599
q13	9276	3125	3175	3125
q14	300	265	257	257
q15	500	476	460	460
q16	488	439	418	418
q17	1858	1630	1630	1630
q18	8458	7826	7793	7793
q19	1699	1591	1588	1588
q20	2127	1813	1829	1813
q21	5058	5073	5080	5073
q22	1149	1027	1047	1027
Total cold run time: 68209 ms
Total hot run time: 59133 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 197359 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit e9b1f7a5350e7c1dafb97b72b79bdc1456fae306, data reload: false

query1	1301	909	909	909
query2	6340	1983	1931	1931
query3	10798	4341	4139	4139
query4	60922	28903	23843	23843
query5	5195	461	439	439
query6	394	171	175	171
query7	5445	315	304	304
query8	313	233	213	213
query9	8555	2606	2577	2577
query10	469	270	251	251
query11	17489	15641	15895	15641
query12	160	107	108	107
query13	1453	479	443	443
query14	9819	6850	7314	6850
query15	209	181	187	181
query16	6995	448	430	430
query17	1220	580	603	580
query18	1679	347	315	315
query19	238	161	164	161
query20	119	112	108	108
query21	211	103	106	103
query22	4751	4669	4799	4669
query23	34814	34245	33804	33804
query24	6168	2909	2940	2909
query25	546	427	426	426
query26	654	167	164	164
query27	1929	365	347	347
query28	3993	2234	2183	2183
query29	737	447	416	416
query30	237	154	162	154
query31	985	790	828	790
query32	65	52	52	52
query33	444	304	288	288
query34	922	497	513	497
query35	832	731	719	719
query36	1062	959	924	924
query37	107	71	71	71
query38	4081	3980	4066	3980
query39	1543	1619	1473	1473
query40	203	100	97	97
query41	48	49	47	47
query42	119	103	101	101
query43	505	505	483	483
query44	1178	817	817	817
query45	186	167	168	167
query46	1158	705	729	705
query47	1991	1984	1955	1955
query48	472	389	403	389
query49	734	398	385	385
query50	847	442	417	417
query51	7386	7217	7267	7217
query52	102	91	90	90
query53	262	190	182	182
query54	568	460	459	459
query55	83	80	79	79
query56	246	254	243	243
query57	1327	1222	1213	1213
query58	212	205	216	205
query59	3320	3074	3052	3052
query60	282	256	252	252
query61	116	110	109	109
query62	788	676	674	674
query63	222	187	183	183
query64	1388	660	620	620
query65	3256	3261	3194	3194
query66	680	291	291	291
query67	16054	15583	15553	15553
query68	4181	580	570	570
query69	433	256	262	256
query70	1115	1121	1103	1103
query71	325	250	247	247
query72	6364	4015	4067	4015
query73	749	339	338	338
query74	10299	9190	9290	9190
query75	3365	2632	2616	2616
query76	2129	1040	1110	1040
query77	501	274	270	270
query78	10609	9636	9667	9636
query79	1477	583	591	583
query80	877	432	417	417
query81	487	218	217	217
query82	1247	87	91	87
query83	245	148	139	139
query84	286	85	79	79
query85	925	305	285	285
query86	327	299	299	299
query87	4401	4297	4233	4233
query88	3619	2386	2363	2363
query89	410	294	287	287
query90	1916	183	181	181
query91	179	147	151	147
query92	61	50	48	48
query93	1863	558	554	554
query94	739	291	279	279
query95	349	260	257	257
query96	608	279	278	278
query97	3333	3184	3114	3114
query98	213	194	203	194
query99	1605	1302	1296	1296
Total cold run time: 312164 ms
Total hot run time: 197359 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 29.71 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit e9b1f7a5350e7c1dafb97b72b79bdc1456fae306, data reload: false

query1	0.03	0.02	0.03
query2	0.06	0.03	0.03
query3	0.23	0.06	0.06
query4	1.62	0.10	0.10
query5	0.54	0.51	0.52
query6	1.14	0.73	0.73
query7	0.02	0.02	0.02
query8	0.04	0.03	0.02
query9	0.57	0.50	0.50
query10	0.54	0.56	0.56
query11	0.16	0.10	0.11
query12	0.14	0.12	0.12
query13	0.60	0.59	0.60
query14	0.78	0.78	0.82
query15	0.85	0.83	0.82
query16	0.38	0.37	0.39
query17	1.09	1.07	1.07
query18	0.22	0.21	0.22
query19	1.99	1.90	1.85
query20	0.01	0.01	0.01
query21	15.40	0.61	0.57
query22	2.48	2.93	1.30
query23	17.08	0.91	0.82
query24	3.14	1.76	0.99
query25	0.24	0.19	0.06
query26	0.47	0.13	0.14
query27	0.07	0.04	0.06
query28	9.91	0.54	0.44
query29	12.57	3.32	3.29
query30	0.25	0.06	0.07
query31	2.85	0.39	0.40
query32	3.24	0.47	0.47
query33	3.00	3.00	3.02
query34	17.15	4.42	4.49
query35	4.54	4.48	4.52
query36	0.68	0.48	0.49
query37	0.09	0.06	0.06
query38	0.04	0.03	0.04
query39	0.04	0.02	0.02
query40	0.17	0.14	0.12
query41	0.08	0.03	0.02
query42	0.04	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 104.58 s
Total hot run time: 29.71 s

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@dataroaring dataroaring Awaiting requested review from dataroaring dataroaring is a code owner

At least 1 approving review is required to merge this pull request.

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@seawinde @doris-robot