Skip to content

[fix](create table) Optimize create table function `createPartitionWi…#31234

Merged
yiguolei merged 1 commit intoapache:masterfrom
deardeng:fix-create-partition
Feb 22, 2024
Merged

[fix](create table) Optimize create table function `createPartitionWi…#31234
yiguolei merged 1 commit intoapache:masterfrom
deardeng:fix-create-partition

Conversation

@deardeng
Copy link
Contributor

@deardeng deardeng commented Feb 21, 2024

…thIndices` parameters

Proposed changes

Issue Number: close #xxx

createPartitionWithIndices 函数30多个参数，多半都是olapTable的成员变量，优化下

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

@doris-robot
Copy link

doris-robot commented Feb 21, 2024

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

@deardeng
Copy link
Contributor Author

deardeng commented Feb 21, 2024

run buildall

@doris-robot
Copy link

doris-robot commented Feb 21, 2024

TPC-H: Total hot run time: 41255 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 1704365f0fc35618b1d76e59bb6a3dee157b15a3, data reload: false

------ Round 1 ----------------------------------
q1	17682	5004	4904	4904
q2	2036	138	139	138
q3	10576	1029	987	987
q4	4648	955	970	955
q5	7674	3197	3267	3197
q6	200	128	130	128
q7	1260	774	762	762
q8	9237	2021	2090	2021
q9	7675	6707	6666	6666
q10	8314	2642	2647	2642
q11	421	224	237	224
q12	763	326	323	323
q13	18028	3703	3648	3648
q14	302	267	258	258
q15	589	515	502	502
q16	474	426	408	408
q17	925	863	848	848
q18	7457	6633	6571	6571
q19	1539	1487	1499	1487
q20	633	361	330	330
q21	6893	3911	3976	3911
q22	879	354	345	345
Total cold run time: 108205 ms
Total hot run time: 41255 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4875	4922	4871	4871
q2	298	192	185	185
q3	3618	3598	3566	3566
q4	2548	2513	2532	2513
q5	5748	5734	5766	5734
q6	207	127	126	126
q7	2266	1661	1668	1661
q8	3047	3118	3107	3107
q9	8721	8742	8706	8706
q10	6769	4241	4237	4237
q11	517	371	376	371
q12	767	544	545	544
q13	4347	3392	3372	3372
q14	279	250	232	232
q15	607	511	491	491
q16	480	435	433	433
q17	1686	1624	1637	1624
q18	8301	7705	7660	7660
q19	1620	1633	1623	1623
q20	2123	1836	1854	1836
q21	6563	6205	6198	6198
q22	575	532	515	515
Total cold run time: 65962 ms
Total hot run time: 59605 ms

@doris-robot
Copy link

doris-robot commented Feb 21, 2024

TPC-DS: Total hot run time: 176906 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 1704365f0fc35618b1d76e59bb6a3dee157b15a3, data reload: false

query1	934	348	363	348
query2	6504	1798	1696	1696
query3	6733	208	203	203
query4	23117	21104	21228	21104
query5	4212	373	378	373
query6	269	187	169	169
query7	4607	308	292	292
query8	251	199	195	195
query9	8427	2799	2778	2778
query10	417	235	227	227
query11	15026	14508	14486	14486
query12	145	84	81	81
query13	1696	421	423	421
query14	9298	7662	7781	7662
query15	206	193	195	193
query16	7545	259	250	250
query17	1406	553	535	535
query18	1951	275	262	262
query19	197	143	151	143
query20	87	82	82	82
query21	193	126	117	117
query22	4998	4828	4756	4756
query23	32468	31592	31457	31457
query24	12774	3458	3468	3458
query25	641	374	353	353
query26	1872	172	169	169
query27	3023	315	316	315
query28	6584	1829	1818	1818
query29	1124	640	624	624
query30	284	144	145	144
query31	950	754	759	754
query32	93	59	59	59
query33	723	245	246	245
query34	1037	493	496	493
query35	935	825	839	825
query36	1032	899	870	870
query37	169	59	61	59
query38	3291	3153	3208	3153
query39	1387	1331	1313	1313
query40	282	104	106	104
query41	44	35	36	35
query42	105	99	104	99
query43	477	462	447	447
query44	1066	689	709	689
query45	203	180	178	178
query46	1055	784	777	777
query47	1640	1571	1523	1523
query48	428	339	349	339
query49	1218	313	306	306
query50	775	374	372	372
query51	4442	4267	4274	4267
query52	111	90	95	90
query53	399	309	300	300
query54	296	218	217	217
query55	85	81	77	77
query56	218	207	208	207
query57	1050	942	974	942
query58	228	206	207	206
query59	2303	2197	2140	2140
query60	253	213	220	213
query61	84	88	82	82
query62	596	368	367	367
query63	329	286	287	286
query64	6310	3061	3137	3061
query65	3287	3267	3222	3222
query66	1349	326	316	316
query67	14449	14598	14285	14285
query68	5124	552	535	535
query69	533	356	354	354
query70	1251	1206	1195	1195
query71	439	258	256	256
query72	6312	2785	2641	2641
query73	712	310	316	310
query74	6967	6494	6441	6441
query75	3213	2589	2546	2546
query76	3256	1167	1211	1167
query77	350	235	227	227
query78	9442	8841	8823	8823
query79	954	510	494	494
query80	506	359	350	350
query81	439	207	207	207
query82	167	85	86	85
query83	143	118	121	118
query84	226	79	81	79
query85	1029	343	344	343
query86	293	298	311	298
query87	3476	3273	3295	3273
query88	2745	2295	2303	2295
query89	447	360	364	360
query90	1937	171	165	165
query91	153	130	129	129
query92	53	52	51	51
query93	1028	513	495	495
query94	1179	183	177	177
query95	498	373	381	373
query96	591	261	261	261
query97	4416	4308	4284	4284
query98	222	208	198	198
query99	1061	747	754	747
Total cold run time: 268850 ms
Total hot run time: 176906 ms

@doris-robot
Copy link

doris-robot commented Feb 21, 2024

ClickBench: Total hot run time: 30.74 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 1704365f0fc35618b1d76e59bb6a3dee157b15a3, data reload: false

query1	0.02	0.03	0.02
query2	0.07	0.02	0.02
query3	0.23	0.08	0.08
query4	1.64	0.11	0.10
query5	0.49	0.48	0.48
query6	1.32	0.61	0.63
query7	0.02	0.01	0.01
query8	0.04	0.03	0.03
query9	0.53	0.45	0.44
query10	0.50	0.49	0.48
query11	0.12	0.09	0.10
query12	0.11	0.10	0.10
query13	0.59	0.59	0.59
query14	0.76	0.79	0.79
query15	0.83	0.79	0.80
query16	0.32	0.32	0.33
query17	0.91	0.88	0.92
query18	0.15	0.19	0.16
query19	1.74	1.69	1.63
query20	0.01	0.01	0.01
query21	15.43	0.63	0.59
query22	3.35	3.69	2.28
query23	17.54	1.10	0.94
query24	2.17	0.29	0.48
query25	0.61	0.08	0.07
query26	0.17	0.14	0.13
query27	0.06	0.05	0.05
query28	11.89	0.85	0.84
query29	12.65	3.48	3.31
query30	0.54	0.48	0.52
query31	2.78	0.35	0.35
query32	3.34	0.47	0.48
query33	3.12	3.17	3.14
query34	15.36	4.52	4.46
query35	4.51	4.50	4.49
query36	1.07	0.94	0.97
query37	0.08	0.05	0.05
query38	0.04	0.02	0.03
query39	0.02	0.02	0.02
query40	0.18	0.14	0.15
query41	0.07	0.02	0.02
query42	0.03	0.01	0.01
query43	0.02	0.03	0.02
Total cold run time: 105.43 s
Total hot run time: 30.74 s

@doris-robot
Copy link

doris-robot commented Feb 21, 2024

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 1704365f0fc35618b1d76e59bb6a3dee157b15a3 with default session variables
Stream load json:         18 seconds loaded 2358488459 Bytes, about 124 MB/s
Stream load orc:          59 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet:      32 seconds loaded 861443392 Bytes, about 25 MB/s
Insert into select:       13.5 seconds inserted 10000000 Rows, about 740K ops/s

dataroaring
dataroaring approved these changes Feb 21, 2024
View reviewed changes
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Feb 21, 2024
@github-actions
Copy link
Contributor

github-actions bot commented Feb 21, 2024

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Contributor

github-actions bot commented Feb 21, 2024

PR approved by anyone and no changes requested.

@deardeng
Copy link
Contributor Author

deardeng commented Feb 22, 2024

run buildall

@deardeng
Copy link
Contributor Author

deardeng commented Feb 22, 2024

run fe_ut

@doris-robot
Copy link

doris-robot commented Feb 22, 2024

TPC-H: Total hot run time: 40793 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 1704365f0fc35618b1d76e59bb6a3dee157b15a3, data reload: false

------ Round 1 ----------------------------------
q1	17675	4969	4868	4868
q2	2047	135	129	129
q3	10599	983	977	977
q4	4658	943	950	943
q5	7636	3113	3146	3113
q6	192	136	136	136
q7	1244	770	745	745
q8	9342	2053	2044	2044
q9	7425	6600	6564	6564
q10	8300	2633	2629	2629
q11	414	212	214	212
q12	744	324	321	321
q13	17930	3615	3576	3576
q14	280	256	249	249
q15	600	534	529	529
q16	470	397	400	397
q17	908	871	867	867
q18	7412	6604	6580	6580
q19	1551	1487	1483	1483
q20	546	280	256	256
q21	6412	3843	3950	3843
q22	867	336	332	332
Total cold run time: 107252 ms
Total hot run time: 40793 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5312	4810	4828	4810
q2	293	187	184	184
q3	3607	3561	3562	3561
q4	2508	2523	2518	2518
q5	5718	5703	5710	5703
q6	211	128	133	128
q7	2223	1666	1652	1652
q8	2986	3045	3069	3045
q9	8665	8641	8642	8641
q10	6752	4207	4212	4207
q11	505	381	368	368
q12	759	534	532	532
q13	4398	3415	3400	3400
q14	268	244	229	229
q15	584	530	494	494
q16	474	432	428	428
q17	1642	1579	1574	1574
q18	8175	7547	7595	7547
q19	1636	1631	1627	1627
q20	2096	1865	1830	1830
q21	6495	6093	6072	6072
q22	544	500	534	500
Total cold run time: 65851 ms
Total hot run time: 59050 ms

@doris-robot
Copy link

doris-robot commented Feb 22, 2024

TPC-DS: Total hot run time: 176221 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 1704365f0fc35618b1d76e59bb6a3dee157b15a3, data reload: false

query1	916	352	345	345
query2	6522	1837	1680	1680
query3	6701	203	210	203
query4	23112	21059	21056	21056
query5	4295	382	377	377
query6	253	173	170	170
query7	4603	306	288	288
query8	256	192	198	192
query9	8419	2857	2847	2847
query10	421	220	223	220
query11	15158	14495	14480	14480
query12	145	83	80	80
query13	1708	424	417	417
query14	9220	7775	7744	7744
query15	216	191	195	191
query16	7512	250	247	247
query17	1401	559	523	523
query18	1949	265	257	257
query19	191	146	152	146
query20	89	85	82	82
query21	191	121	129	121
query22	4842	4673	4611	4611
query23	32372	31417	31377	31377
query24	12791	3357	3315	3315
query25	653	369	362	362
query26	1900	158	160	158
query27	3045	315	329	315
query28	6588	1857	1843	1843
query29	1143	633	626	626
query30	282	139	148	139
query31	925	743	752	743
query32	91	57	56	56
query33	730	240	227	227
query34	1061	491	508	491
query35	964	825	821	821
query36	1009	939	896	896
query37	188	62	65	62
query38	3291	3179	3198	3179
query39	1390	1328	1312	1312
query40	282	112	106	106
query41	35	34	33	33
query42	112	99	104	99
query43	470	447	435	435
query44	1077	686	711	686
query45	200	190	179	179
query46	1058	790	762	762
query47	1656	1542	1525	1525
query48	411	357	350	350
query49	1240	313	311	311
query50	770	371	383	371
query51	4412	4336	4337	4336
query52	105	100	95	95
query53	404	299	302	299
query54	296	225	231	225
query55	85	81	78	78
query56	235	198	205	198
query57	1050	986	943	943
query58	218	203	198	198
query59	2234	2096	2137	2096
query60	242	214	206	206
query61	83	82	83	82
query62	599	376	378	376
query63	362	311	299	299
query64	6260	3045	3128	3045
query65	3282	3252	3245	3245
query66	1340	331	329	329
query67	14469	14400	14089	14089
query68	5082	571	553	553
query69	522	357	354	354
query70	1329	1215	1270	1215
query71	452	258	257	257
query72	6306	2794	2613	2613
query73	703	306	312	306
query74	6827	6393	6341	6341
query75	3206	2597	2563	2563
query76	3270	1107	1205	1107
query77	365	240	237	237
query78	9482	8783	8763	8763
query79	975	528	528	528
query80	511	356	345	345
query81	440	207	200	200
query82	165	84	93	84
query83	143	125	130	125
query84	219	81	76	76
query85	1026	362	340	340
query86	303	308	305	305
query87	3440	3301	3263	3263
query88	2777	2287	2285	2285
query89	449	372	376	372
query90	1920	165	167	165
query91	154	125	130	125
query92	58	48	51	48
query93	994	518	504	504
query94	1094	178	179	178
query95	434	333	342	333
query96	582	265	275	265
query97	4459	4298	4289	4289
query98	225	206	206	206
query99	1046	681	740	681
Total cold run time: 268488 ms
Total hot run time: 176221 ms

@doris-robot
Copy link

doris-robot commented Feb 22, 2024

ClickBench: Total hot run time: 30.9 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 1704365f0fc35618b1d76e59bb6a3dee157b15a3, data reload: false

query1	0.03	0.02	0.02
query2	0.07	0.03	0.03
query3	0.23	0.06	0.07
query4	1.64	0.09	0.09
query5	0.48	0.48	0.48
query6	1.38	0.62	0.64
query7	0.02	0.01	0.01
query8	0.04	0.02	0.02
query9	0.52	0.45	0.45
query10	0.49	0.49	0.49
query11	0.13	0.09	0.09
query12	0.12	0.10	0.10
query13	0.60	0.59	0.59
query14	0.77	0.78	0.79
query15	0.83	0.79	0.79
query16	0.32	0.33	0.33
query17	0.91	0.92	0.91
query18	0.19	0.16	0.17
query19	1.74	1.71	1.61
query20	0.01	0.01	0.02
query21	15.43	0.63	0.61
query22	2.95	4.61	2.26
query23	17.53	1.04	1.03
query24	2.22	0.66	0.31
query25	0.61	0.07	0.06
query26	0.19	0.15	0.14
query27	0.06	0.05	0.05
query28	11.73	0.86	0.83
query29	12.53	3.30	3.33
query30	0.52	0.55	0.48
query31	2.77	0.35	0.37
query32	3.32	0.48	0.48
query33	3.09	3.13	3.13
query34	15.36	4.53	4.54
query35	4.49	4.46	4.49
query36	1.06	0.96	0.95
query37	0.07	0.06	0.05
query38	0.04	0.03	0.03
query39	0.03	0.01	0.01
query40	0.17	0.15	0.15
query41	0.08	0.01	0.02
query42	0.02	0.02	0.02
query43	0.03	0.02	0.02
Total cold run time: 104.82 s
Total hot run time: 30.9 s

@doris-robot
Copy link

doris-robot commented Feb 22, 2024

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 1704365f0fc35618b1d76e59bb6a3dee157b15a3 with default session variables
Stream load json:         20 seconds loaded 2358488459 Bytes, about 112 MB/s
Stream load orc:          59 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       14.3 seconds inserted 10000000 Rows, about 699K ops/s

@yiguolei yiguolei merged commit f46f7e2 into apache:master Feb 22, 2024
feiniaofeiafei pushed a commit to feiniaofeiafei/doris that referenced this pull request Feb 23, 2024
@deardeng @feiniaofeiafei
[fix](create table) Optimize create table function `createPartitionWi…
1f4a40f 
…thIndices` parameters (apache#31234)
deardeng added a commit to deardeng/incubator-doris that referenced this pull request Mar 18, 2024
@deardeng
[fix](create table) Optimize create table function `createPartitionWi…
d70fc8d 
…thIndices` parameters (apache#31234)
@deardeng deardeng mentioned this pull request Mar 18, 2024
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@yiguolei yiguolei yiguolei approved these changes

@dataroaring dataroaring dataroaring approved these changes

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@deardeng @doris-robot @yiguolei @dataroaring