Skip to content

[opt](Nereids) use enable_nereids_dml to control ctas and create table#29284

Merged
yiguolei merged 1 commit intoapache:masterfrom
morrySnow:add_switch_to_create_table
Dec 29, 2023
Merged

[opt](Nereids) use enable_nereids_dml to control ctas and create table#29284
yiguolei merged 1 commit intoapache:masterfrom
morrySnow:add_switch_to_create_table

Conversation

@morrySnow
Copy link
Contributor

@morrySnow morrySnow commented Dec 29, 2023

Proposed changes

Issue Number: close #xxx

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

@morrySnow
Copy link
Contributor Author

morrySnow commented Dec 29, 2023

run buildall

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Dec 29, 2023
@github-actions
Copy link
Contributor

github-actions bot commented Dec 29, 2023

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Contributor

github-actions bot commented Dec 29, 2023

PR approved by anyone and no changes requested.

@doris-robot
Copy link

doris-robot commented Dec 29, 2023

TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G', run with scripts in https://github.com/apache/doris/tree/master/tools/tpch-tools

Tpch sf100 test result on commit 652fbad0473c151b3e5698c88f07ae7a2f0fd999, data reload: false

------ Round 1 ----------------------------------
q1	17623	5346	5126	5126
q2	2036	160	145	145
q3	10583	1132	1091	1091
q4	10189	765	806	765
q5	7870	2934	2972	2934
q6	215	136	135	135
q7	937	522	543	522
q8	9306	2061	2044	2044
q9	6984	6405	6410	6405
q10	8260	3064	3031	3031
q11	438	213	223	213
q12	399	240	247	240
q13	18018	3670	3638	3638
q14	249	214	226	214
q15	595	551	547	547
q16	466	403	395	395
q17	975	505	441	441
q18	7343	6673	6696	6673
q19	1621	1243	1360	1243
q20	737	330	330	330
q21	2832	2384	2434	2384
q22	397	339	324	324
Total cold run time: 108073 ms
Total hot run time: 38840 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5474	5133	5093	5093
q2	337	270	280	270
q3	3325	3277	3252	3252
q4	2127	2020	1998	1998
q5	5815	5778	5792	5778
q6	220	131	133	131
q7	2394	1898	1916	1898
q8	3399	3469	3525	3469
q9	8818	8829	8821	8821
q10	3813	3869	3879	3869
q11	588	498	494	494
q12	794	652	621	621
q13	7179	3206	3206	3206
q14	297	276	277	276
q15	623	565	553	553
q16	561	519	503	503
q17	1941	1758	1725	1725
q18	8690	8342	8272	8272
q19	1638	1638	1626	1626
q20	2201	1964	1969	1964
q21	5598	5291	5300	5291
q22	546	468	466	466
Total cold run time: 66378 ms
Total hot run time: 59576 ms

@doris-robot
Copy link

doris-robot commented Dec 29, 2023

TPC-DS test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G', run with scripts in https://github.com/apache/doris/tree/master/tools/tpcds-tools

TPC-DS sf100 test result on commit 652fbad0473c151b3e5698c88f07ae7a2f0fd999, data reload: false

run tpcds-sf100 query with default conf and session variables
query1	940	358	335	335
query2	6413	1901	1849	1849
query3	6655	222	217	217
query4	27119	22619	22381	22381
query5	5994	586	613	586
query6	287	184	195	184
query7	4585	286	265	265
query8	229	210	199	199
query9	8169	2770	2782	2770
query10	458	269	269	269
query11	16241	15556	15630	15556
query12	137	82	79	79
query13	1681	348	328	328
query14	12346	7447	7375	7375
query15	250	192	195	192
query16	6325	304	290	290
query17	1754	523	497	497
query18	1929	267	262	262
query19	269	146	139	139
query20	84	82	83	82
query21	189	99	100	99
query22	5075	4570	4496	4496
query23	32099	31601	31428	31428
query24	10815	2838	2814	2814
query25	559	378	360	360
query26	779	150	150	150
query27	2515	292	290	290
query28	6125	2013	1994	1994
query29	918	419	417	417
query30	286	141	154	141
query31	976	798	790	790
query32	89	65	67	65
query33	732	283	281	281
query34	836	452	455	452
query35	916	792	762	762
query36	1361	1179	1286	1179
query37	113	83	85	83
query38	3411	3285	3232	3232
query39	1337	1291	1276	1276
query40	185	98	92	92
query41	38	35	40	35
query42	101	93	94	93
query43	555	492	483	483
query44	1096	762	770	762
query45	203	190	187	187
query46	1085	663	657	657
query47	1750	1623	1508	1508
query48	355	265	262	262
query49	1064	337	340	337
query50	757	348	350	348
query51	5412	5326	5320	5320
query52	97	86	92	86
query53	218	160	155	155
query54	1380	605	625	605
query55	103	92	85	85
query56	221	209	209	209
query57	1031	992	967	967
query58	234	210	211	210
query59	2731	2697	2575	2575
query60	272	241	245	241
query61	90	95	85	85
query62	626	482	477	477
query63	170	157	154	154
query64	4648	1751	1772	1751
query65	3363	3253	3254	3253
query66	971	343	332	332
query67	15594	15332	15270	15270
query68	11732	546	527	527
query69	600	276	263	263
query70	1688	1636	1595	1595
query71	572	240	235	235
query72	5409	3553	3624	3553
query73	1432	328	318	318
query74	6921	6505	6513	6505
query75	5542	2350	2277	2277
query76	5242	1147	1105	1105
query77	871	284	273	273
query78	9112	8632	8546	8546
query79	6075	517	530	517
query80	1831	388	380	380
query81	511	214	211	211
query82	339	133	120	120
query83	388	142	145	142
query84	259	56	56	56
query85	1887	290	285	285
query86	408	382	426	382
query87	3557	3356	3344	3344
query88	3297	2443	2437	2437
query89	367	269	279	269
query90	2006	254	247	247
query91	122	95	94	94
query92	71	54	56	54
query93	2983	462	524	462
query94	938	222	219	219
query95	527	484	466	466
query96	674	342	338	338
query97	4281	4150	4173	4150
query98	225	199	192	192
query99	1080	848	812	812
Total cold run time: 296293 ms
Total hot run time: 181077 ms

@morrySnow
Copy link
Contributor Author

morrySnow commented Dec 29, 2023

run p1

@morrySnow
Copy link
Contributor Author

morrySnow commented Dec 29, 2023

run clickbench

@doris-robot
Copy link

doris-robot commented Dec 29, 2023

TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G', run with scripts in https://github.com/apache/doris/tree/master/tools/tpch-tools

Tpch sf100 test result on commit 652fbad0473c151b3e5698c88f07ae7a2f0fd999, data reload: false

run tpch-sf100 query with default conf and session variables
q1	5345	5110	5159	5110
q2	391	164	158	158
q3	1484	1144	1162	1144
q4	1070	839	836	836
q5	3099	3116	3077	3077
q6	228	138	136	136
q7	994	539	564	539
q8	2157	2313	2221	2221
q9	6724	6640	6712	6640
q10	3218	3153	3108	3108
q11	344	217	228	217
q12	389	250	245	245
q13	4412	3658	3626	3626
q14	250	220	225	220
q15	631	586	555	555
q16	486	410	404	404
q17	1045	483	518	483
q18	7113	6757	6952	6757
q19	1655	2006	1483	1483
q20	567	377	362	362
q21	2865	2436	2491	2436
q22	407	327	332	327
Total cold run time: 44874 ms
Total hot run time: 40084 ms

run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1	5050	5028	5061	5028
q2	337	250	255	250
q3	3351	3332	3312	3312
q4	2136	2043	2038	2038
q5	5917	5929	5928	5928
q6	235	131	133	131
q7	2385	1939	1912	1912
q8	3539	3658	3664	3658
q9	9079	8974	9012	8974
q10	3875	3920	3946	3920
q11	592	492	480	480
q12	808	633	634	633
q13	3889	3195	3252	3195
q14	313	268	277	268
q15	631	580	553	553
q16	555	511	501	501
q17	1998	1805	1770	1770
q18	8771	8297	8450	8297
q19	1738	1737	1732	1732
q20	2286	2008	2001	2001
q21	5768	5293	5309	5293
q22	545	489	450	450
Total cold run time: 63798 ms
Total hot run time: 60324 ms

@doris-robot
Copy link

doris-robot commented Dec 29, 2023

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 47.44 seconds
stream load tsv: 577 seconds loaded 74807831229 Bytes, about 123 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 28.4 seconds inserted 10000000 Rows, about 352K ops/s
storage size: 17183980584 Bytes

yiguolei
yiguolei approved these changes Dec 29, 2023
View reviewed changes
Copy link
Contributor

@yiguolei yiguolei left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@yiguolei yiguolei merged commit fcb90dc into apache:master Dec 29, 2023
HappenLee pushed a commit to HappenLee/incubator-doris that referenced this pull request Jan 12, 2024
@morrySnow @HappenLee
[opt](Nereids) use enable_nereids_dml to control ctas and create table (
a7d1094 
apache#29284)
@morrySnow morrySnow deleted the add_switch_to_create_table branch February 27, 2024 10:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@yiguolei yiguolei yiguolei approved these changes

@starocean999 starocean999 starocean999 approved these changes

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@morrySnow @doris-robot @yiguolei @starocean999