Skip to content

[enhance](mtmv)MTMV support refresh by time range #36134

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
zddr wants to merge 29 commits into from
Closed

[enhance](mtmv)MTMV support refresh by time range #36134

zddr wants to merge 29 commits into from

Conversation

@zddr
Copy link
Contributor

@zddr zddr commented Jun 11, 2024 •
edited
Loading

support REFRESH MATERIALIZED VIEW mv1 partition start '2017-01-01' end '2017-01-02';

REFRESH MATERIALIZED VIEW mv1 partition [('2017-02-01'), ('2017-02-02'));

@doris-robot
Copy link

doris-robot commented Jun 11, 2024

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@zddr
Copy link
Contributor Author

zddr commented Jun 11, 2024

run buildall

@zddr
Copy link
Contributor Author

zddr commented Jun 11, 2024

run buildall

@doris-robot
Copy link

doris-robot commented Jun 11, 2024

TPC-H: Total hot run time: 39947 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit f7d67ae82026d4b05bda48df47c25a2669a6dc33, data reload: false

------ Round 1 ----------------------------------
q1	17701	4458	4300	4300
q2	2031	192	188	188
q3	10529	1086	1174	1086
q4	10193	810	811	810
q5	7528	2670	2662	2662
q6	224	139	134	134
q7	968	598	587	587
q8	9217	2080	2114	2080
q9	9016	6549	6518	6518
q10	9023	3710	3706	3706
q11	443	232	228	228
q12	432	234	225	225
q13	18728	2956	2935	2935
q14	273	229	226	226
q15	520	474	470	470
q16	509	379	374	374
q17	982	681	663	663
q18	8090	7579	7487	7487
q19	7137	1545	1464	1464
q20	672	319	324	319
q21	4891	3156	3251	3156
q22	385	329	330	329
Total cold run time: 119492 ms
Total hot run time: 39947 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4408	4213	4209	4209
q2	375	269	268	268
q3	3111	2910	2943	2910
q4	1940	1691	1687	1687
q5	5539	5462	5494	5462
q6	225	129	126	126
q7	2254	1876	1880	1876
q8	3314	3399	3412	3399
q9	8655	8824	8797	8797
q10	4049	3753	3850	3753
q11	599	501	485	485
q12	811	666	648	648
q13	17268	3144	3178	3144
q14	315	280	261	261
q15	526	477	479	477
q16	474	431	426	426
q17	1827	1519	1515	1515
q18	8227	7896	7944	7896
q19	1829	1679	1536	1536
q20	2973	1871	1851	1851
q21	5113	4684	4999	4684
q22	624	542	585	542
Total cold run time: 74456 ms
Total hot run time: 55952 ms

@doris-robot
Copy link

doris-robot commented Jun 11, 2024

TPC-DS: Total hot run time: 173594 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit f7d67ae82026d4b05bda48df47c25a2669a6dc33, data reload: false

query1	924	376	384	376
query2	6371	2372	2410	2372
query3	6630	208	213	208
query4	18999	17331	17323	17323
query5	3586	462	460	460
query6	243	159	151	151
query7	4587	297	301	297
query8	312	316	300	300
query9	8407	2327	2317	2317
query10	544	290	278	278
query11	10627	10026	9994	9994
query12	114	94	84	84
query13	1650	364	357	357
query14	8521	7887	6821	6821
query15	235	198	186	186
query16	7674	273	262	262
query17	1556	536	512	512
query18	1972	280	299	280
query19	192	159	153	153
query20	86	83	79	79
query21	209	126	127	126
query22	4249	4132	4008	4008
query23	33813	33724	33679	33679
query24	11094	2845	2904	2845
query25	667	375	371	371
query26	1229	157	150	150
query27	2527	330	336	330
query28	7290	2100	2080	2080
query29	929	648	605	605
query30	265	154	152	152
query31	972	764	727	727
query32	97	51	54	51
query33	766	302	286	286
query34	1041	471	478	471
query35	746	629	637	629
query36	1108	951	950	950
query37	160	74	80	74
query38	2997	2858	2797	2797
query39	925	845	834	834
query40	208	131	129	129
query41	59	54	53	53
query42	110	112	105	105
query43	632	563	540	540
query44	1179	737	742	737
query45	200	161	168	161
query46	1085	705	725	705
query47	1852	1790	1785	1785
query48	366	311	317	311
query49	859	408	409	408
query50	781	390	406	390
query51	6775	6730	6631	6631
query52	106	89	97	89
query53	368	298	287	287
query54	895	463	456	456
query55	76	74	74	74
query56	292	274	279	274
query57	1147	1043	1047	1043
query58	254	257	239	239
query59	3404	3344	3401	3344
query60	303	282	286	282
query61	107	106	101	101
query62	619	461	443	443
query63	313	286	288	286
query64	8997	2306	1843	1843
query65	3208	3086	3082	3082
query66	784	328	337	328
query67	15663	15071	14911	14911
query68	6232	551	568	551
query69	724	439	447	439
query70	1204	1098	1056	1056
query71	510	270	272	270
query72	8480	5375	5757	5375
query73	792	326	326	326
query74	5826	5488	5435	5435
query75	4779	2680	2710	2680
query76	4105	948	918	918
query77	790	294	295	294
query78	10515	9897	9628	9628
query79	4392	515	511	511
query80	1017	469	455	455
query81	554	219	218	218
query82	1304	105	100	100
query83	420	162	162	162
query84	268	88	86	86
query85	1329	270	273	270
query86	447	309	301	301
query87	3270	3065	3082	3065
query88	4144	2464	2446	2446
query89	500	369	365	365
query90	1874	189	184	184
query91	125	100	94	94
query92	62	48	53	48
query93	5053	517	511	511
query94	1152	186	190	186
query95	399	315	313	313
query96	608	272	261	261
query97	3191	3005	3003	3003
query98	209	199	193	193
query99	1142	838	864	838
Total cold run time: 280373 ms
Total hot run time: 173594 ms

@doris-robot
Copy link

doris-robot commented Jun 11, 2024

ClickBench: Total hot run time: 30.74 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit f7d67ae82026d4b05bda48df47c25a2669a6dc33, data reload: false

query1	0.04	0.03	0.03
query2	0.08	0.04	0.04
query3	0.23	0.05	0.05
query4	1.67	0.09	0.08
query5	0.52	0.49	0.48
query6	1.13	0.73	0.72
query7	0.02	0.02	0.01
query8	0.05	0.04	0.04
query9	0.53	0.50	0.47
query10	0.53	0.54	0.54
query11	0.15	0.12	0.11
query12	0.15	0.12	0.12
query13	0.60	0.58	0.58
query14	0.79	0.78	0.77
query15	0.82	0.80	0.81
query16	0.36	0.36	0.36
query17	0.95	0.97	0.96
query18	0.19	0.26	0.23
query19	1.74	1.73	1.66
query20	0.01	0.01	0.00
query21	15.44	0.65	0.65
query22	4.46	6.64	2.35
query23	18.25	1.36	1.32
query24	2.08	0.23	0.22
query25	0.15	0.10	0.09
query26	0.28	0.16	0.17
query27	0.08	0.08	0.08
query28	13.28	1.01	1.00
query29	12.61	3.28	3.22
query30	0.26	0.06	0.06
query31	2.87	0.38	0.38
query32	3.27	0.47	0.46
query33	2.87	2.85	2.91
query34	17.22	4.37	4.44
query35	4.47	4.43	4.52
query36	0.66	0.48	0.46
query37	0.17	0.16	0.15
query38	0.15	0.14	0.15
query39	0.04	0.04	0.04
query40	0.18	0.13	0.13
query41	0.09	0.06	0.06
query42	0.06	0.05	0.04
query43	0.04	0.03	0.04
Total cold run time: 109.54 s
Total hot run time: 30.74 s

morrySnow
morrySnow reviewed Jun 12, 2024
View reviewed changes
@zddr
Copy link
Contributor Author

zddr commented Jun 14, 2024

run buildall

@doris-robot
Copy link

doris-robot commented Jun 14, 2024

TPC-H: Total hot run time: 39521 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 4327e77d7af3d2035e6cd8a9ae4b8f32c96e4641, data reload: false

------ Round 1 ----------------------------------
q1	17597	4319	4269	4269
q2	2015	199	196	196
q3	10431	1203	1166	1166
q4	10186	793	872	793
q5	7468	2644	2611	2611
q6	219	137	136	136
q7	950	634	605	605
q8	9212	2060	2067	2060
q9	8909	6491	6468	6468
q10	8880	3698	3746	3698
q11	439	245	237	237
q12	432	229	232	229
q13	18733	2935	2962	2935
q14	262	212	233	212
q15	523	460	473	460
q16	514	378	389	378
q17	954	685	666	666
q18	7940	7324	7303	7303
q19	6619	1425	1387	1387
q20	639	318	318	318
q21	4907	3062	3966	3062
q22	388	332	338	332
Total cold run time: 118217 ms
Total hot run time: 39521 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4362	4229	4215	4215
q2	367	259	266	259
q3	2958	2933	2816	2816
q4	1973	1655	1686	1655
q5	5609	5470	5439	5439
q6	222	134	137	134
q7	2178	1872	1802	1802
q8	3266	3402	3430	3402
q9	8646	8658	8857	8658
q10	4048	3904	3719	3719
q11	607	500	493	493
q12	833	617	625	617
q13	17080	3131	3134	3131
q14	315	294	283	283
q15	511	493	490	490
q16	489	455	443	443
q17	1801	1507	1496	1496
q18	8106	7852	7719	7719
q19	1784	1590	1596	1590
q20	3042	1908	1900	1900
q21	5134	5076	4788	4788
q22	609	536	527	527
Total cold run time: 73940 ms
Total hot run time: 55576 ms

@doris-robot
Copy link

doris-robot commented Jun 14, 2024

TPC-DS: Total hot run time: 173290 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 4327e77d7af3d2035e6cd8a9ae4b8f32c96e4641, data reload: false

query1	906	378	369	369
query2	6465	2396	2148	2148
query3	6638	208	209	208
query4	18585	17235	17450	17235
query5	3650	496	458	458
query6	238	159	169	159
query7	4585	310	296	296
query8	309	291	290	290
query9	8426	2423	2407	2407
query10	570	307	299	299
query11	10430	9936	9906	9906
query12	117	91	87	87
query13	1660	359	357	357
query14	10184	6798	7485	6798
query15	228	197	184	184
query16	7397	259	258	258
query17	1510	557	532	532
query18	1889	274	274	274
query19	192	163	160	160
query20	91	86	83	83
query21	204	136	139	136
query22	4317	4046	4183	4046
query23	33805	33709	33588	33588
query24	11358	2871	2856	2856
query25	582	371	386	371
query26	1089	156	160	156
query27	2455	324	324	324
query28	6299	2131	2099	2099
query29	916	625	633	625
query30	240	159	160	159
query31	965	766	744	744
query32	102	56	54	54
query33	764	297	283	283
query34	1076	485	505	485
query35	753	639	626	626
query36	1129	977	1011	977
query37	143	75	76	75
query38	2874	2804	2833	2804
query39	892	844	821	821
query40	238	137	136	136
query41	57	54	51	51
query42	115	104	108	104
query43	615	556	508	508
query44	1227	740	736	736
query45	198	166	168	166
query46	1074	715	711	711
query47	1834	1778	1789	1778
query48	378	313	305	305
query49	861	420	420	420
query50	772	394	405	394
query51	6760	6669	6686	6669
query52	114	92	95	92
query53	366	296	296	296
query54	887	449	452	449
query55	77	75	75	75
query56	315	276	282	276
query57	1106	1048	1084	1048
query58	266	248	283	248
query59	3460	3045	3230	3045
query60	318	297	296	296
query61	112	123	105	105
query62	606	450	453	450
query63	326	299	291	291
query64	9056	2369	1842	1842
query65	3211	3133	3130	3130
query66	782	337	344	337
query67	15523	15011	14767	14767
query68	7393	547	532	532
query69	728	464	385	385
query70	1120	1170	1120	1120
query71	513	282	281	281
query72	8463	5703	5340	5340
query73	802	327	325	325
query74	5853	5472	5503	5472
query75	4294	2664	2669	2664
query76	4159	898	897	897
query77	644	306	309	306
query78	10440	9985	9923	9923
query79	8319	535	521	521
query80	1615	474	467	467
query81	538	221	223	221
query82	767	106	99	99
query83	203	166	164	164
query84	269	83	89	83
query85	1319	311	282	282
query86	438	292	316	292
query87	3293	3116	3088	3088
query88	5120	2354	2343	2343
query89	530	379	388	379
query90	1894	187	189	187
query91	128	98	98	98
query92	65	50	50	50
query93	6133	503	498	498
query94	1192	192	184	184
query95	406	310	307	307
query96	584	277	264	264
query97	3187	3023	3077	3023
query98	228	204	191	191
query99	1317	855	842	842
Total cold run time: 286641 ms
Total hot run time: 173290 ms

@doris-robot
Copy link

doris-robot commented Jun 14, 2024

ClickBench: Total hot run time: 30.18 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 4327e77d7af3d2035e6cd8a9ae4b8f32c96e4641, data reload: false

query1	0.04	0.03	0.03
query2	0.08	0.04	0.04
query3	0.23	0.05	0.05
query4	1.68	0.06	0.08
query5	0.49	0.49	0.48
query6	1.13	0.72	0.71
query7	0.02	0.01	0.02
query8	0.06	0.04	0.04
query9	0.55	0.48	0.48
query10	0.52	0.54	0.55
query11	0.15	0.11	0.12
query12	0.14	0.12	0.13
query13	0.60	0.59	0.60
query14	0.77	0.76	0.79
query15	0.84	0.82	0.81
query16	0.37	0.37	0.34
query17	1.02	0.95	1.03
query18	0.23	0.23	0.24
query19	1.78	1.78	1.69
query20	0.01	0.00	0.01
query21	15.40	0.65	0.66
query22	3.95	8.62	1.49
query23	18.27	1.36	1.42
query24	2.10	0.22	0.23
query25	0.17	0.08	0.08
query26	0.28	0.17	0.18
query27	0.09	0.09	0.08
query28	13.20	1.00	1.00
query29	12.67	3.29	3.29
query30	0.29	0.08	0.07
query31	2.84	0.39	0.39
query32	3.27	0.48	0.47
query33	2.88	2.92	2.90
query34	17.09	4.45	4.41
query35	4.55	4.56	4.48
query36	0.64	0.47	0.46
query37	0.18	0.16	0.16
query38	0.15	0.14	0.15
query39	0.04	0.03	0.04
query40	0.17	0.15	0.14
query41	0.10	0.05	0.05
query42	0.06	0.06	0.06
query43	0.05	0.04	0.04
Total cold run time: 109.15 s
Total hot run time: 30.18 s

@zddr
Copy link
Contributor Author

zddr commented Jun 14, 2024

run buildall

@doris-robot
Copy link

doris-robot commented Jun 14, 2024

TPC-H: Total hot run time: 40332 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 2e762201ad4c1529f0179e44ab69af3216cf6ad5, data reload: false

------ Round 1 ----------------------------------
q1	17696	4460	4229	4229
q2	2019	186	201	186
q3	10518	1086	1180	1086
q4	10195	812	847	812
q5	7451	2672	2622	2622
q6	218	135	135	135
q7	939	629	601	601
q8	9217	2024	2094	2024
q9	9037	6475	6477	6475
q10	8921	3763	3680	3680
q11	447	239	238	238
q12	446	231	223	223
q13	17832	2968	3012	2968
q14	256	215	206	206
q15	509	471	478	471
q16	517	389	372	372
q17	961	684	637	637
q18	8003	7463	7388	7388
q19	6780	1477	1480	1477
q20	645	311	338	311
q21	4941	3863	4017	3863
q22	397	329	328	328
Total cold run time: 117945 ms
Total hot run time: 40332 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4366	4276	4228	4228
q2	368	262	270	262
q3	3024	2969	2949	2949
q4	1975	1770	1828	1770
q5	5469	5486	5393	5393
q6	229	135	134	134
q7	2270	1834	1845	1834
q8	3314	3403	3371	3371
q9	8738	8731	8751	8731
q10	4183	3663	3799	3663
q11	615	501	499	499
q12	821	652	626	626
q13	17020	3144	3163	3144
q14	290	288	274	274
q15	536	495	473	473
q16	500	426	428	426
q17	1806	1500	1498	1498
q18	8143	8021	7823	7823
q19	1803	1547	1475	1475
q20	2799	1886	1855	1855
q21	5166	4894	4865	4865
q22	657	529	564	529
Total cold run time: 74092 ms
Total hot run time: 55822 ms

@doris-robot
Copy link

doris-robot commented Jun 14, 2024

TPC-DS: Total hot run time: 173564 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 2e762201ad4c1529f0179e44ab69af3216cf6ad5, data reload: false

query1	937	381	373	373
query2	6446	2323	2347	2323
query3	6628	202	203	202
query4	18892	17344	17169	17169
query5	3641	478	473	473
query6	230	156	157	156
query7	4581	302	307	302
query8	333	284	286	284
query9	8499	2390	2380	2380
query10	568	322	280	280
query11	10611	10048	9768	9768
query12	125	89	87	87
query13	1659	366	374	366
query14	10163	6987	6953	6953
query15	231	190	189	189
query16	7616	278	277	277
query17	1393	561	523	523
query18	1972	282	286	282
query19	201	166	157	157
query20	92	84	87	84
query21	216	135	130	130
query22	4342	4117	4044	4044
query23	33496	33685	33801	33685
query24	10993	2945	2988	2945
query25	590	387	372	372
query26	721	152	151	151
query27	2311	318	325	318
query28	5994	2112	2120	2112
query29	892	608	616	608
query30	254	157	160	157
query31	960	786	762	762
query32	90	52	60	52
query33	650	285	274	274
query34	871	492	472	472
query35	752	637	637	637
query36	1116	965	969	965
query37	142	69	74	69
query38	2934	2817	2847	2817
query39	903	846	839	839
query40	216	128	129	128
query41	57	51	49	49
query42	109	102	104	102
query43	605	584	536	536
query44	1091	710	704	704
query45	192	166	161	161
query46	1066	707	722	707
query47	1876	1756	1770	1756
query48	353	281	285	281
query49	826	393	403	393
query50	750	380	391	380
query51	6827	6683	6721	6683
query52	106	88	89	88
query53	363	298	291	291
query54	893	451	435	435
query55	77	72	78	72
query56	297	257	266	257
query57	1120	1006	1059	1006
query58	244	256	257	256
query59	3315	3290	3121	3121
query60	288	270	276	270
query61	101	90	88	88
query62	581	424	439	424
query63	311	288	291	288
query64	8489	2240	1767	1767
query65	3187	3117	3142	3117
query66	752	322	330	322
query67	15453	14919	14892	14892
query68	6104	531	541	531
query69	653	463	402	402
query70	1224	1169	1106	1106
query71	479	263	268	263
query72	7868	5526	5780	5526
query73	781	316	322	316
query74	5878	5493	5504	5493
query75	3901	2697	2650	2650
query76	3692	1008	967	967
query77	688	293	291	291
query78	10277	10031	9827	9827
query79	2501	509	513	509
query80	2512	455	503	455
query81	575	222	226	222
query82	745	117	105	105
query83	303	162	163	162
query84	265	89	83	83
query85	1197	282	265	265
query86	415	322	324	322
query87	3276	3072	3089	3072
query88	3853	2328	2333	2328
query89	480	395	388	388
query90	1760	190	186	186
query91	125	97	97	97
query92	60	52	50	50
query93	2475	510	507	507
query94	979	192	187	187
query95	405	321	318	318
query96	589	266	267	266
query97	3244	3027	3074	3027
query98	226	215	205	205
query99	1173	833	827	827
Total cold run time: 271293 ms
Total hot run time: 173564 ms

@doris-robot
Copy link

doris-robot commented Jun 14, 2024

ClickBench: Total hot run time: 30.26 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 2e762201ad4c1529f0179e44ab69af3216cf6ad5, data reload: false

query1	0.04	0.03	0.04
query2	0.08	0.04	0.04
query3	0.23	0.05	0.04
query4	1.68	0.08	0.09
query5	0.49	0.48	0.49
query6	1.13	0.72	0.72
query7	0.02	0.02	0.01
query8	0.05	0.05	0.05
query9	0.54	0.48	0.47
query10	0.55	0.55	0.52
query11	0.16	0.11	0.12
query12	0.15	0.12	0.13
query13	0.60	0.60	0.60
query14	0.78	0.79	0.79
query15	0.82	0.82	0.81
query16	0.36	0.36	0.35
query17	1.02	1.01	1.01
query18	0.23	0.23	0.27
query19	1.76	1.72	1.71
query20	0.01	0.01	0.01
query21	15.43	0.66	0.65
query22	4.33	7.25	1.69
query23	18.23	1.54	1.23
query24	2.08	0.22	0.22
query25	0.17	0.09	0.07
query26	0.26	0.18	0.17
query27	0.06	0.07	0.08
query28	13.30	1.03	1.00
query29	12.66	3.29	3.23
query30	0.26	0.06	0.06
query31	2.86	0.39	0.39
query32	3.30	0.48	0.47
query33	2.93	2.91	2.94
query34	17.33	4.45	4.40
query35	4.51	4.47	4.54
query36	0.66	0.49	0.46
query37	0.18	0.15	0.15
query38	0.15	0.14	0.14
query39	0.04	0.04	0.04
query40	0.17	0.13	0.15
query41	0.09	0.05	0.05
query42	0.05	0.04	0.05
query43	0.05	0.04	0.04
Total cold run time: 109.8 s
Total hot run time: 30.26 s

@zddr
Copy link
Contributor Author

zddr commented Jun 17, 2024

run p0

@zddr
Copy link
Contributor Author

zddr commented Jul 10, 2024

run buildall

@doris-robot
Copy link

doris-robot commented Jul 10, 2024

TPC-H: Total hot run time: 39651 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 02acdd3fad65881540ffed80e888a6da238e1b19, data reload: false

------ Round 1 ----------------------------------
q1	17602	4372	4257	4257
q2	2005	189	181	181
q3	10448	1224	1090	1090
q4	10190	786	869	786
q5	7484	2632	2627	2627
q6	217	137	133	133
q7	960	595	614	595
q8	9224	2085	2050	2050
q9	9009	6511	6507	6507
q10	8948	3739	3735	3735
q11	457	234	235	234
q12	435	222	224	222
q13	17773	2978	2964	2964
q14	266	228	230	228
q15	517	482	484	482
q16	504	372	371	371
q17	954	696	727	696
q18	7913	7406	7319	7319
q19	6122	1430	1402	1402
q20	668	321	338	321
q21	4942	3125	3119	3119
q22	389	336	332	332
Total cold run time: 117027 ms
Total hot run time: 39651 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4415	4267	4274	4267
q2	369	266	270	266
q3	2930	2776	2857	2776
q4	1993	1735	1699	1699
q5	5645	5470	5470	5470
q6	229	139	136	136
q7	2227	1881	1879	1879
q8	3294	3373	3434	3373
q9	8636	8681	8782	8681
q10	4092	3823	3750	3750
q11	606	507	514	507
q12	825	641	671	641
q13	16246	3111	3158	3111
q14	304	268	268	268
q15	534	493	491	491
q16	489	437	433	433
q17	1815	1539	1504	1504
q18	8070	8015	7868	7868
q19	1825	1511	1553	1511
q20	2592	1895	1856	1856
q21	5141	4812	4677	4677
q22	649	554	602	554
Total cold run time: 72926 ms
Total hot run time: 55718 ms

@doris-robot
Copy link

doris-robot commented Jul 10, 2024

TPC-DS: Total hot run time: 174288 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 02acdd3fad65881540ffed80e888a6da238e1b19, data reload: false

query1	908	366	366	366
query2	6438	2397	2404	2397
query3	6638	204	210	204
query4	23718	17565	17525	17525
query5	3746	480	485	480
query6	251	192	172	172
query7	4593	291	289	289
query8	322	301	314	301
query9	8499	2390	2358	2358
query10	432	295	271	271
query11	10741	10080	10272	10080
query12	113	89	83	83
query13	1661	385	379	379
query14	10167	7769	7718	7718
query15	237	188	187	187
query16	7806	301	296	296
query17	1838	546	513	513
query18	1921	273	287	273
query19	197	145	153	145
query20	93	84	88	84
query21	208	129	125	125
query22	4723	4235	4180	4180
query23	34018	33432	33473	33432
query24	10619	2914	2869	2869
query25	583	393	399	393
query26	709	149	145	145
query27	2199	277	284	277
query28	6231	2135	2121	2121
query29	922	639	619	619
query30	258	154	150	150
query31	989	758	741	741
query32	98	50	54	50
query33	659	316	280	280
query34	904	511	495	495
query35	678	586	572	572
query36	1136	971	951	951
query37	145	83	86	83
query38	2960	2815	2789	2789
query39	897	877	810	810
query40	205	121	116	116
query41	56	50	49	49
query42	114	95	101	95
query43	580	571	558	558
query44	1072	733	744	733
query45	195	157	162	157
query46	1068	747	710	710
query47	1878	1765	1809	1765
query48	385	307	297	297
query49	851	404	420	404
query50	781	386	389	386
query51	6843	6792	6684	6684
query52	112	89	91	89
query53	360	290	282	282
query54	867	450	446	446
query55	74	72	73	72
query56	302	253	271	253
query57	1160	1037	1049	1037
query58	248	250	244	244
query59	3459	3142	3251	3142
query60	306	294	273	273
query61	94	121	98	98
query62	799	639	655	639
query63	325	286	287	286
query64	9200	2173	1615	1615
query65	3152	3093	3080	3080
query66	744	324	329	324
query67	15304	15122	14991	14991
query68	4499	535	550	535
query69	526	411	323	323
query70	1186	1156	1059	1059
query71	373	291	282	282
query72	7058	5461	5141	5141
query73	743	321	318	318
query74	5997	5502	5510	5502
query75	3337	2676	2643	2643
query76	2299	924	956	924
query77	451	295	313	295
query78	9547	9059	8993	8993
query79	1787	516	517	516
query80	1916	467	471	467
query81	601	218	216	216
query82	1052	129	134	129
query83	293	160	172	160
query84	262	92	86	86
query85	1117	324	311	311
query86	387	325	332	325
query87	3266	3143	3055	3055
query88	3263	2461	2449	2449
query89	473	382	388	382
query90	1650	186	183	183
query91	132	102	105	102
query92	59	47	51	47
query93	1463	512	505	505
query94	1046	214	206	206
query95	400	310	314	310
query96	595	274	276	274
query97	3184	3064	3071	3064
query98	225	209	200	200
query99	1628	1258	1298	1258
Total cold run time: 270604 ms
Total hot run time: 174288 ms

@doris-robot
Copy link

doris-robot commented Jul 10, 2024

ClickBench: Total hot run time: 30.6 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 02acdd3fad65881540ffed80e888a6da238e1b19, data reload: false

query1	0.04	0.03	0.03
query2	0.08	0.04	0.04
query3	0.23	0.06	0.06
query4	1.66	0.08	0.08
query5	0.49	0.49	0.48
query6	1.15	0.73	0.73
query7	0.02	0.01	0.01
query8	0.05	0.04	0.04
query9	0.55	0.49	0.49
query10	0.56	0.54	0.55
query11	0.15	0.12	0.11
query12	0.14	0.12	0.12
query13	0.60	0.58	0.57
query14	0.77	0.77	0.76
query15	0.86	0.80	0.83
query16	0.38	0.38	0.35
query17	0.97	1.01	1.05
query18	0.23	0.23	0.22
query19	1.80	1.70	1.76
query20	0.01	0.01	0.01
query21	15.38	0.77	0.66
query22	4.24	6.90	1.88
query23	18.27	1.35	1.37
query24	2.08	0.24	0.23
query25	0.15	0.09	0.08
query26	0.29	0.21	0.20
query27	0.45	0.23	0.23
query28	13.28	1.01	1.00
query29	12.56	3.34	3.32
query30	0.25	0.06	0.05
query31	2.89	0.39	0.39
query32	3.27	0.48	0.47
query33	2.89	2.89	2.96
query34	17.08	4.29	4.36
query35	4.40	4.43	4.37
query36	0.65	0.47	0.48
query37	0.19	0.16	0.15
query38	0.15	0.14	0.15
query39	0.04	0.04	0.03
query40	0.16	0.12	0.13
query41	0.10	0.05	0.05
query42	0.05	0.05	0.05
query43	0.04	0.04	0.03
Total cold run time: 109.6 s
Total hot run time: 30.6 s

@zddr
Copy link
Contributor Author

zddr commented Jul 11, 2024

run cloud_p1

seawinde
seawinde reviewed Jul 12, 2024
View reviewed changes
fe/fe-core/src/main/java/org/apache/doris/mtmv/MTMVPartitionUtil.java
return false;
}

private static DateV2Literal transferPartitionKeyToDate(PartitionKey partitionKey) throws AnalysisException {
Copy link
Contributor

@seawinde seawinde Jul 12, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if mv is partitoned by hour, maybe we should support datatime? it may be general

Copy link
Contributor Author

@zddr zddr Jul 12, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If there is a need, I will implement it in another PR

@github-actions
Copy link
Contributor

github-actions bot commented Jul 12, 2024

PR approved by anyone and no changes requested.

@zddr
Copy link
Contributor Author

zddr commented Jul 22, 2024

run buildall

@doris-robot
Copy link

doris-robot commented Jul 22, 2024

TPC-H: Total hot run time: 40287 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit b2a00895bd1dbee449468bbb8df0825dd9798149, data reload: false

------ Round 1 ----------------------------------
q1	17704	5061	4470	4470
q2	2025	188	192	188
q3	10504	1210	1159	1159
q4	10187	823	775	775
q5	7531	2702	2703	2702
q6	219	135	138	135
q7	958	599	613	599
q8	9210	2074	2065	2065
q9	8674	6544	6560	6544
q10	8819	3806	3827	3806
q11	446	231	234	231
q12	402	230	226	226
q13	18722	2986	3035	2986
q14	276	233	244	233
q15	527	494	484	484
q16	499	397	384	384
q17	986	623	712	623
q18	8156	7633	7911	7633
q19	8517	1327	1368	1327
q20	662	324	322	322
q21	4877	3107	3226	3107
q22	354	289	288	288
Total cold run time: 120255 ms
Total hot run time: 40287 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4485	4279	4269	4269
q2	375	272	281	272
q3	3165	2997	2919	2919
q4	2005	1746	1707	1707
q5	5564	5465	5497	5465
q6	221	140	139	139
q7	2252	1880	1869	1869
q8	3294	3442	3414	3414
q9	8767	8921	8847	8847
q10	4059	3853	3879	3853
q11	617	521	522	521
q12	795	647	635	635
q13	16170	3122	3193	3122
q14	327	303	302	302
q15	523	478	505	478
q16	510	446	455	446
q17	1807	1531	1515	1515
q18	8177	7960	7765	7765
q19	1813	1521	1458	1458
q20	2983	1892	1866	1866
q21	5199	4870	4933	4870
q22	571	536	498	498
Total cold run time: 73679 ms
Total hot run time: 56230 ms

@doris-robot
Copy link

doris-robot commented Jul 22, 2024

TPC-DS: Total hot run time: 175132 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit b2a00895bd1dbee449468bbb8df0825dd9798149, data reload: false

query1	910	373	372	372
query2	6450	2022	1915	1915
query3	6635	214	208	208
query4	25391	17559	17378	17378
query5	3646	471	489	471
query6	260	186	158	158
query7	4598	281	280	280
query8	245	195	191	191
query9	8441	2482	2453	2453
query10	422	296	290	290
query11	12061	10220	10025	10025
query12	117	86	84	84
query13	1636	393	369	369
query14	10024	7828	8106	7828
query15	230	167	170	167
query16	7491	479	477	477
query17	1113	538	534	534
query18	1905	276	279	276
query19	199	146	153	146
query20	91	84	82	82
query21	218	129	133	129
query22	4375	4139	4191	4139
query23	34131	33892	33694	33694
query24	11206	2942	2963	2942
query25	618	413	418	413
query26	775	154	157	154
query27	2309	277	292	277
query28	6284	2064	2065	2064
query29	886	656	621	621
query30	256	158	155	155
query31	980	799	774	774
query32	98	57	59	57
query33	758	365	318	318
query34	889	504	507	504
query35	896	782	754	754
query36	1116	996	983	983
query37	144	88	85	85
query38	2967	2909	2846	2846
query39	933	828	830	828
query40	198	125	123	123
query41	50	45	47	45
query42	116	99	102	99
query43	522	490	485	485
query44	1183	736	716	716
query45	201	163	162	162
query46	1097	721	714	714
query47	1863	1826	1812	1812
query48	367	297	293	293
query49	859	421	428	421
query50	777	404	399	399
query51	6804	6585	6623	6585
query52	106	102	99	99
query53	366	287	298	287
query54	925	470	451	451
query55	77	74	73	73
query56	344	271	275	271
query57	1141	1085	1059	1059
query58	250	244	258	244
query59	2982	2816	2712	2712
query60	295	281	300	281
query61	94	97	97	97
query62	784	656	657	656
query63	319	287	282	282
query64	9271	2242	1661	1661
query65	3189	3084	3084	3084
query66	718	334	340	334
query67	15592	15066	15245	15066
query68	4484	527	535	527
query69	489	332	358	332
query70	1179	1150	1077	1077
query71	372	287	271	271
query72	7215	5784	5884	5784
query73	750	326	329	326
query74	6108	5692	5638	5638
query75	3380	2767	2669	2669
query76	2319	1008	993	993
query77	433	303	291	291
query78	9595	9102	8846	8846
query79	2463	517	519	517
query80	1513	481	467	467
query81	592	234	221	221
query82	897	139	137	137
query83	254	168	169	168
query84	250	87	84	84
query85	1836	320	321	320
query86	493	321	306	306
query87	3348	3200	3196	3196
query88	4040	2439	2448	2439
query89	473	384	383	383
query90	1770	199	194	194
query91	129	99	101	99
query92	59	49	49	49
query93	3134	500	494	494
query94	1166	310	285	285
query95	414	316	320	316
query96	622	281	282	281
query97	3191	3009	3029	3009
query98	222	197	193	193
query99	1471	1277	1268	1268
Total cold run time: 276855 ms
Total hot run time: 175132 ms

@doris-robot
Copy link

doris-robot commented Jul 22, 2024

ClickBench: Total hot run time: 31.82 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit b2a00895bd1dbee449468bbb8df0825dd9798149, data reload: false

query1	0.04	0.04	0.03
query2	0.07	0.04	0.04
query3	0.23	0.04	0.05
query4	1.68	0.07	0.08
query5	0.51	0.49	0.48
query6	1.13	0.72	0.72
query7	0.02	0.01	0.02
query8	0.05	0.05	0.05
query9	0.54	0.50	0.50
query10	0.54	0.55	0.53
query11	0.16	0.11	0.11
query12	0.15	0.11	0.11
query13	0.59	0.59	0.58
query14	0.77	0.80	0.80
query15	0.84	0.81	0.81
query16	0.36	0.35	0.34
query17	1.02	0.97	0.97
query18	0.22	0.22	0.22
query19	1.79	1.67	1.67
query20	0.01	0.01	0.01
query21	15.39	0.76	0.66
query22	3.58	6.73	3.04
query23	18.40	1.37	1.33
query24	2.20	0.22	0.21
query25	0.17	0.08	0.09
query26	0.30	0.21	0.22
query27	0.45	0.23	0.24
query28	13.23	1.02	1.02
query29	12.60	3.33	3.34
query30	0.26	0.06	0.05
query31	2.86	0.40	0.38
query32	3.27	0.47	0.48
query33	2.88	3.02	2.92
query34	17.02	4.37	4.33
query35	4.43	4.43	4.40
query36	0.65	0.48	0.50
query37	0.18	0.16	0.16
query38	0.15	0.16	0.15
query39	0.05	0.03	0.03
query40	0.15	0.12	0.12
query41	0.10	0.05	0.05
query42	0.05	0.04	0.04
query43	0.05	0.04	0.04
Total cold run time: 109.14 s
Total hot run time: 31.82 s

@zddr zddr marked this pull request as draft September 10, 2024 02:12
@zddr zddr closed this Feb 25, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@morrySnow morrySnow Awaiting requested review from morrySnow

1 more reviewer

@seawinde seawinde seawinde approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

kind/need-document reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@zddr @doris-robot @morrySnow @seawinde