[enhance](mtmv)Limit the number of partitions for table creation #30867

zddr · 2024-02-05T10:20:06Z

Proposed changes

Issue Number: close #xxx

Creating too many partitions is time-consuming, so limiting the number of partitions
add more case,such as mor,mow

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

doris-robot · 2024-02-05T10:20:10Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

zddr · 2024-02-05T10:20:17Z

run buildall

zddr · 2024-02-05T12:31:10Z

run buildall

doris-robot · 2024-02-05T13:46:19Z

TPC-H: Total hot run time: 36560 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 1bc48c6995533aa0975d07819c74f3eba4917eb9, data reload: false

------ Round 1 ----------------------------------
q1	17662	4493	4368	4368
q2	2044	142	135	135
q3	10598	920	893	893
q4	4643	770	690	690
q5	7689	2875	2801	2801
q6	183	120	121	120
q7	1121	726	713	713
q8	9296	1980	2021	1980
q9	7270	6354	6283	6283
q10	8090	2447	2390	2390
q11	427	212	209	209
q12	790	268	270	268
q13	18015	3311	3291	3291
q14	278	249	241	241
q15	536	489	487	487
q16	459	389	397	389
q17	921	513	466	466
q18	6781	5975	5842	5842
q19	1563	1387	1373	1373
q20	651	342	343	342
q21	7019	3067	2992	2992
q22	808	303	287	287
Total cold run time: 106844 ms
Total hot run time: 36560 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4408	4292	4387	4292
q2	318	239	238	238
q3	2963	2829	2789	2789
q4	1854	1687	1628	1628
q5	5106	5155	5165	5155
q6	190	113	115	113
q7	2098	1784	1773	1773
q8	3106	3188	3193	3188
q9	8338	8266	8297	8266
q10	5907	3505	3506	3505
q11	541	474	444	444
q12	737	595	562	562
q13	13434	3084	3063	3063
q14	276	252	264	252
q15	523	487	489	487
q16	506	474	474	474
q17	1848	1703	1679	1679
q18	8040	7595	7512	7512
q19	10145	1549	1485	1485
q20	2126	1927	1919	1919
q21	4743	4490	4703	4490
q22	581	466	464	464
Total cold run time: 77788 ms
Total hot run time: 53778 ms

doris-robot · 2024-02-05T13:57:45Z

TPC-DS: Total hot run time: 173922 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 1bc48c6995533aa0975d07819c74f3eba4917eb9, data reload: false

query1	952	343	340	340
query2	6538	1930	1875	1875
query3	6692	208	207	207
query4	31808	21967	21933	21933
query5	4302	422	361	361
query6	262	171	163	163
query7	4610	279	279	279
query8	246	181	189	181
query9	9116	2256	2264	2256
query10	426	236	221	221
query11	19218	15605	15360	15360
query12	125	74	76	74
query13	1619	416	413	413
query14	9579	6780	7166	6780
query15	246	176	178	176
query16	8143	267	241	241
query17	2112	531	500	500
query18	2109	261	263	261
query19	375	141	144	141
query20	86	76	77	76
query21	200	126	130	126
query22	4751	4642	4648	4642
query23	31276	30018	30014	30014
query24	10528	2796	2733	2733
query25	556	347	341	341
query26	804	137	149	137
query27	2303	297	292	292
query28	6820	1833	1813	1813
query29	857	619	603	603
query30	280	133	144	133
query31	895	680	694	680
query32	90	62	49	49
query33	721	216	223	216
query34	1100	454	460	454
query35	911	741	735	735
query36	1049	920	932	920
query37	100	57	59	57
query38	3287	3103	3056	3056
query39	1289	1265	1248	1248
query40	180	96	98	96
query41	40	35	34	34
query42	94	90	93	90
query43	494	426	486	426
query44	1118	675	685	675
query45	193	186	177	177
query46	1039	668	665	665
query47	1601	1528	1499	1499
query48	426	352	345	345
query49	1068	289	296	289
query50	760	369	383	369
query51	5302	5151	5139	5139
query52	99	87	83	83
query53	331	270	272	270
query54	290	215	209	209
query55	80	72	79	72
query56	226	196	202	196
query57	962	885	857	857
query58	205	182	179	179
query59	2215	2365	2289	2289
query60	242	222	209	209
query61	85	83	86	83
query62	639	337	348	337
query63	296	268	278	268
query64	5015	3628	3768	3628
query65	3241	3195	3225	3195
query66	831	323	308	308
query67	14329	13962	13998	13962
query68	5467	520	517	517
query69	480	333	321	321
query70	1235	1165	1263	1165
query71	324	256	251	251
query72	6298	2840	2694	2694
query73	699	315	326	315
query74	6566	6264	6132	6132
query75	3126	2368	2302	2302
query76	3264	1012	1049	1012
query77	363	233	226	226
query78	9194	8696	8573	8573
query79	2236	484	498	484
query80	1304	348	340	340
query81	540	198	191	191
query82	801	79	82	79
query83	237	127	127	127
query84	236	81	77	77
query85	2054	340	330	330
query86	504	283	289	283
query87	3368	3204	3211	3204
query88	3896	2306	2318	2306
query89	433	359	359	359
query90	1952	163	164	163
query91	156	121	126	121
query92	55	43	43	43
query93	2331	484	483	483
query94	1268	175	171	171
query95	8082	359	7943	359
query96	591	272	278	272
query97	4217	4101	4116	4101
query98	221	193	194	193
query99	1182	692	705	692
Total cold run time: 290462 ms
Total hot run time: 173922 ms

doris-robot · 2024-02-05T14:03:13Z

ClickBench: Total hot run time: 31.17 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 1bc48c6995533aa0975d07819c74f3eba4917eb9, data reload: false

query1	0.04	0.03	0.03
query2	0.06	0.02	0.03
query3	0.23	0.06	0.07
query4	1.64	0.10	0.09
query5	0.53	0.52	0.50
query6	1.21	0.65	0.63
query7	0.02	0.01	0.01
query8	0.04	0.02	0.03
query9	0.54	0.50	0.50
query10	0.55	0.53	0.57
query11	0.12	0.08	0.09
query12	0.12	0.09	0.09
query13	0.61	0.61	0.61
query14	0.78	0.81	0.79
query15	0.79	0.78	0.78
query16	0.39	0.40	0.39
query17	1.02	1.00	1.02
query18	0.21	0.26	0.23
query19	1.88	1.77	1.77
query20	0.01	0.01	0.01
query21	15.41	0.56	0.55
query22	2.75	2.23	2.35
query23	17.75	0.82	0.84
query24	2.45	0.81	0.71
query25	0.35	0.16	0.09
query26	0.43	0.14	0.15
query27	0.06	0.05	0.05
query28	12.32	0.85	0.84
query29	12.56	3.16	3.28
query30	0.67	0.55	0.55
query31	2.78	0.34	0.35
query32	3.34	0.48	0.49
query33	3.26	3.22	3.26
query34	16.16	4.37	4.27
query35	4.31	4.29	4.30
query36	1.09	1.04	1.06
query37	0.07	0.05	0.05
query38	0.03	0.02	0.03
query39	0.02	0.02	0.02
query40	0.17	0.13	0.13
query41	0.07	0.02	0.01
query42	0.03	0.02	0.01
query43	0.02	0.01	0.02
Total cold run time: 106.89 s
Total hot run time: 31.17 s

doris-robot · 2024-02-05T14:06:21Z

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 1bc48c6995533aa0975d07819c74f3eba4917eb9 with default session variables
Stream load json:         18 seconds loaded 2358488459 Bytes, about 124 MB/s
Stream load orc:          59 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       14.6 seconds inserted 10000000 Rows, about 684K ops/s

…limit_partition_num

zddr · 2024-02-06T08:56:31Z

run buildall

fe/fe-core/src/main/java/org/apache/doris/analysis/PartitionDesc.java

zddr · 2024-02-06T09:16:12Z

run buildall

github-actions · 2024-02-06T09:16:36Z

PR approved by at least one committer and no changes requested.

github-actions · 2024-02-06T09:16:38Z

PR approved by anyone and no changes requested.

zddr · 2024-02-06T10:50:04Z

run buildall

doris-robot · 2024-02-06T11:21:18Z

TPC-H: Total hot run time: 37023 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit b5b376f018ce93e803f47e49c509985ad7a907d9, data reload: false

------ Round 1 ----------------------------------
q1	17651	4518	4442	4442
q2	2073	151	137	137
q3	10580	907	952	907
q4	4657	779	723	723
q5	7696	2818	2841	2818
q6	183	121	120	120
q7	1149	755	721	721
q8	9284	2022	2020	2020
q9	7249	6408	6346	6346
q10	8098	2412	2454	2412
q11	422	209	218	209
q12	767	274	267	267
q13	18010	3327	3286	3286
q14	268	240	254	240
q15	534	501	487	487
q16	469	401	419	401
q17	927	534	505	505
q18	6934	6060	5877	5877
q19	1576	1477	1406	1406
q20	616	352	327	327
q21	6757	3076	3115	3076
q22	809	306	296	296
Total cold run time: 106709 ms
Total hot run time: 37023 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4496	4408	4374	4374
q2	333	242	246	242
q3	2982	2855	2798	2798
q4	1897	1636	1710	1636
q5	5184	5309	5237	5237
q6	189	115	117	115
q7	2139	1767	1796	1767
q8	3112	3220	3234	3220
q9	8433	8298	8350	8298
q10	5806	3570	3599	3570
q11	541	452	464	452
q12	776	593	603	593
q13	7542	3137	3073	3073
q14	270	250	256	250
q15	528	489	486	486
q16	513	453	474	453
q17	1803	1689	1676	1676
q18	8063	7749	7626	7626
q19	10464	1547	1552	1547
q20	2133	1924	1916	1916
q21	4746	4627	4518	4518
q22	540	456	488	456
Total cold run time: 72490 ms
Total hot run time: 54303 ms

doris-robot · 2024-02-06T11:32:39Z

TPC-DS: Total hot run time: 173774 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit b5b376f018ce93e803f47e49c509985ad7a907d9, data reload: false

query1	920	345	337	337
query2	6559	1937	1923	1923
query3	6693	209	205	205
query4	31535	22033	22077	22033
query5	4301	354	357	354
query6	256	166	166	166
query7	4603	287	279	279
query8	233	176	191	176
query9	9093	2281	2272	2272
query10	412	215	211	211
query11	18300	15403	15316	15316
query12	126	77	75	75
query13	1615	418	421	418
query14	9628	7304	6818	6818
query15	243	193	184	184
query16	8148	267	251	251
query17	2011	546	497	497
query18	2121	264	256	256
query19	345	134	140	134
query20	80	76	73	73
query21	202	140	121	121
query22	4845	4445	4708	4445
query23	31190	30071	30110	30071
query24	10533	2748	2754	2748
query25	583	364	350	350
query26	1423	143	147	143
query27	2909	304	310	304
query28	7491	1839	1831	1831
query29	877	608	605	605
query30	280	132	140	132
query31	922	693	704	693
query32	92	55	54	54
query33	723	222	213	213
query34	1096	454	454	454
query35	883	759	744	744
query36	1036	936	926	926
query37	115	57	59	57
query38	3275	3111	3103	3103
query39	1293	1260	1250	1250
query40	260	95	93	93
query41	36	34	35	34
query42	93	98	94	94
query43	529	509	490	490
query44	1105	694	697	694
query45	192	184	172	172
query46	1047	667	640	640
query47	1540	1541	1437	1437
query48	443	355	365	355
query49	1129	288	289	288
query50	764	380	374	374
query51	5339	5113	5124	5113
query52	96	87	95	87
query53	335	265	267	265
query54	280	226	218	218
query55	87	79	80	79
query56	225	198	196	196
query57	997	913	907	907
query58	209	182	179	179
query59	2589	2406	2352	2352
query60	240	215	218	215
query61	84	86	82	82
query62	665	350	334	334
query63	288	264	256	256
query64	5986	3686	3353	3353
query65	3233	3229	3211	3211
query66	1057	314	316	314
query67	14434	13754	14079	13754
query68	4114	507	525	507
query69	450	316	340	316
query70	1237	1217	1216	1216
query71	306	243	252	243
query72	5972	2853	2671	2671
query73	707	327	325	325
query74	6580	6268	6196	6196
query75	3030	2371	2335	2335
query76	2550	914	962	914
query77	444	233	229	229
query78	9161	8801	8613	8613
query79	2967	485	495	485
query80	2075	347	351	347
query81	534	197	200	197
query82	827	86	78	78
query83	254	130	134	130
query84	291	82	84	82
query85	2416	351	337	337
query86	488	294	324	294
query87	3434	3173	3220	3173
query88	3808	2330	2341	2330
query89	443	359	381	359
query90	2149	164	164	164
query91	150	123	124	123
query92	58	45	44	44
query93	2996	456	476	456
query94	1455	181	177	177
query95	8105	365	348	348
query96	584	280	271	271
query97	4258	4165	4133	4133
query98	220	199	188	188
query99	1117	685	686	685
Total cold run time: 293452 ms
Total hot run time: 173774 ms

doris-robot · 2024-02-06T11:38:03Z

ClickBench: Total hot run time: 29.92 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit b5b376f018ce93e803f47e49c509985ad7a907d9, data reload: false

query1	0.04	0.03	0.04
query2	0.06	0.03	0.02
query3	0.22	0.06	0.06
query4	1.67	0.09	0.10
query5	0.53	0.52	0.52
query6	1.22	0.65	0.64
query7	0.01	0.02	0.01
query8	0.04	0.02	0.03
query9	0.55	0.51	0.52
query10	0.56	0.55	0.56
query11	0.12	0.09	0.08
query12	0.11	0.09	0.09
query13	0.60	0.61	0.60
query14	0.80	0.82	0.82
query15	0.79	0.78	0.77
query16	0.37	0.40	0.37
query17	1.02	1.03	1.03
query18	0.22	0.27	0.24
query19	1.88	1.78	1.79
query20	0.02	0.01	0.01
query21	15.40	0.58	0.60
query22	2.63	2.71	1.38
query23	17.14	0.93	0.70
query24	2.37	1.18	0.21
query25	0.41	0.16	0.15
query26	0.28	0.13	0.14
query27	0.05	0.06	0.04
query28	11.99	0.86	0.83
query29	12.49	3.24	3.23
query30	0.62	0.55	0.55
query31	2.78	0.33	0.35
query32	3.36	0.49	0.48
query33	3.22	3.23	3.22
query34	15.74	4.31	4.28
query35	4.32	4.36	4.26
query36	1.12	1.06	1.08
query37	0.06	0.06	0.05
query38	0.04	0.02	0.02
query39	0.02	0.02	0.02
query40	0.16	0.12	0.13
query41	0.07	0.02	0.01
query42	0.03	0.01	0.01
query43	0.03	0.03	0.02
Total cold run time: 105.16 s
Total hot run time: 29.92 s

doris-robot · 2024-02-06T11:41:08Z

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit b5b376f018ce93e803f47e49c509985ad7a907d9 with default session variables
Stream load json:         18 seconds loaded 2358488459 Bytes, about 124 MB/s
Stream load orc:          59 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       14.5 seconds inserted 10000000 Rows, about 689K ops/s

github-actions · 2024-02-07T03:05:57Z

PR approved by at least one committer and no changes requested.

…0867) - Creating too many partitions is time-consuming, so limiting the number of partitions - add more case,such as `mor`,`mow`

…ache#30867) - Creating too many partitions is time-consuming, so limiting the number of partitions - add more case,such as `mor`,`mow`

…0867) - Creating too many partitions is time-consuming, so limiting the number of partitions - add more case,such as `mor`,`mow`

…ache#30867) - Creating too many partitions is time-consuming, so limiting the number of partitions - add more case,such as `mor`,`mow`

Limit the number of partitions for table creation

f3da7bf

zddr added 2 commits February 5, 2024 20:22

Limit the number of partitions for table creation

453e7c3

Limit the number of partitions for table creation

1bc48c6

zddr added 12 commits February 6, 2024 15:26

change fe config to session variable

2495c36

case

6fb8ab1

case

b7ae428

case

aeeb854

case

709431d

case

6f58b14

case

f64c643

case

4f28451

case

83739e1

1

6e986c8

change fe config to session variable

0cd1b34

Merge remote-tracking branch 'origin/mv_limit_partition_num' into mv_…

6448185

…limit_partition_num

morrySnow reviewed Feb 6, 2024

View reviewed changes

fe/fe-core/src/main/java/org/apache/doris/analysis/PartitionDesc.java Outdated Show resolved Hide resolved

change fe config to session variable

bbc78b8

morrySnow previously approved these changes Feb 6, 2024

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label Feb 6, 2024

github-actions bot added the reviewed label Feb 6, 2024

change fe config to session variable

b5b376f

zddr dismissed morrySnow’s stale review via b5b376f February 6, 2024 10:44

github-actions bot removed the approved Indicates a PR has been approved by one committer. label Feb 6, 2024

morrySnow approved these changes Feb 7, 2024

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label Feb 7, 2024

seawinde approved these changes Feb 7, 2024

View reviewed changes

morrySnow merged commit 003cc28 into apache:master Feb 7, 2024

yiguolei pushed a commit that referenced this pull request Feb 7, 2024

[enhance](mtmv) Limit the number of partitions for table creation (#3…

5ec288c

…0867) - Creating too many partitions is time-consuming, so limiting the number of partitions - add more case,such as `mor`,`mow`

yiguolei pushed a commit that referenced this pull request Feb 12, 2024

[enhance](mtmv) Limit the number of partitions for table creation (#3…

ffa8167

…0867) - Creating too many partitions is time-consuming, so limiting the number of partitions - add more case,such as `mor`,`mow`

yiguolei pushed a commit that referenced this pull request Feb 16, 2024

[enhance](mtmv) Limit the number of partitions for table creation (#3…

3017c0a

…0867) - Creating too many partitions is time-consuming, so limiting the number of partitions - add more case,such as `mor`,`mow`

zddr deleted the mv_limit_partition_num branch March 28, 2024 02:29

[enhance](mtmv)Limit the number of partitions for table creation #30867

[enhance](mtmv)Limit the number of partitions for table creation #30867

Uh oh!

Conversation

zddr commented Feb 5, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Proposed changes

Further comments

Uh oh!

doris-robot commented Feb 5, 2024

Uh oh!

zddr commented Feb 5, 2024

Uh oh!

zddr commented Feb 5, 2024

Uh oh!

doris-robot commented Feb 5, 2024

Uh oh!

doris-robot commented Feb 5, 2024

Uh oh!

doris-robot commented Feb 5, 2024

Uh oh!

doris-robot commented Feb 5, 2024

Uh oh!

zddr commented Feb 6, 2024

Uh oh!

Uh oh!

zddr commented Feb 6, 2024

Uh oh!

github-actions bot commented Feb 6, 2024

Uh oh!

github-actions bot commented Feb 6, 2024

Uh oh!

zddr commented Feb 6, 2024

Uh oh!

doris-robot commented Feb 6, 2024

Uh oh!

doris-robot commented Feb 6, 2024

Uh oh!

doris-robot commented Feb 6, 2024

Uh oh!

doris-robot commented Feb 6, 2024

Uh oh!

github-actions bot commented Feb 7, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

zddr commented Feb 5, 2024 •

edited

Loading