Skip to content

[feature](workloadgroup) support nereids internal query and all dml query (#28054)#38139

Closed
cambyzju wants to merge 2 commits intoapache:branch-2.0from
cambyzju:fix-2.0-queue-0719
Closed

[feature](workloadgroup) support nereids internal query and all dml query (#28054)#38139
cambyzju wants to merge 2 commits intoapache:branch-2.0from
cambyzju:fix-2.0-queue-0719

Conversation

@cambyzju
Copy link
Copy Markdown
Contributor

cherry-pick #28054 to branch-2.0

The reason is:
INSERT INTO SELECT is a very common type of ETL in Doris, and they always take lots of resources.

@doris-robot
Copy link
Copy Markdown

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@cambyzju
Copy link
Copy Markdown
Contributor Author

run buildall

@doris-robot
Copy link
Copy Markdown

TPC-H: Total hot run time: 50113 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 9e0f441ce7796522f9acffdebef29361dbde624a, data reload: false

------ Round 1 ----------------------------------
q1	17674	4458	4354	4354
q2	2090	160	152	152
q3	10439	1902	1946	1902
q4	10313	1263	1304	1263
q5	8426	3923	3848	3848
q6	240	126	122	122
q7	2052	1628	1608	1608
q8	9480	2738	2718	2718
q9	11090	10541	10448	10448
q10	8597	3512	3473	3473
q11	424	253	257	253
q12	474	308	316	308
q13	18381	3988	4035	3988
q14	362	327	328	327
q15	511	454	466	454
q16	678	558	566	558
q17	1137	969	972	969
q18	7307	6917	6951	6917
q19	1796	1687	1625	1625
q20	524	313	315	313
q21	4466	4134	4080	4080
q22	528	433	439	433
Total cold run time: 116989 ms
Total hot run time: 50113 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4285	4313	4329	4313
q2	320	237	222	222
q3	4174	4212	4172	4172
q4	2753	2752	2743	2743
q5	7200	7160	7156	7156
q6	237	122	119	119
q7	3241	2860	2838	2838
q8	4342	4506	4510	4506
q9	17359	17286	17034	17034
q10	4295	4326	4293	4293
q11	758	700	699	699
q12	1074	845	878	845
q13	6740	3786	3775	3775
q14	450	434	420	420
q15	513	461	469	461
q16	723	680	676	676
q17	3794	3971	3809	3809
q18	8899	8825	8728	8728
q19	1730	1739	1674	1674
q20	2390	2149	2120	2120
q21	8525	8448	8376	8376
q22	1039	967	981	967
Total cold run time: 84841 ms
Total hot run time: 79946 ms

@doris-robot
Copy link
Copy Markdown

TPC-DS: Total hot run time: 203606 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 9e0f441ce7796522f9acffdebef29361dbde624a, data reload: false

query1	945	435	390	390
query2	6556	2599	2574	2574
query3	6919	214	202	202
query4	20048	17943	17948	17943
query5	19737	6491	6564	6491
query6	297	220	234	220
query7	4160	303	317	303
query8	433	412	375	375
query9	3126	2700	2612	2612
query10	428	310	302	302
query11	11244	10691	10702	10691
query12	126	75	73	73
query13	5606	726	704	704
query14	17326	13049	13261	13049
query15	413	246	246	246
query16	6442	287	278	278
query17	1691	1476	884	884
query18	2286	411	423	411
query19	214	147	150	147
query20	81	78	85	78
query21	187	99	98	98
query22	5342	5111	4995	4995
query23	32579	31966	31870	31870
query24	7005	6493	6571	6493
query25	520	446	436	436
query26	524	166	170	166
query27	1798	305	306	305
query28	6180	2376	2315	2315
query29	2971	2743	2649	2649
query30	250	166	180	166
query31	901	725	736	725
query32	70	64	58	58
query33	396	281	258	258
query34	839	486	519	486
query35	1117	975	897	897
query36	1538	1261	1005	1005
query37	94	63	61	61
query38	3104	2940	2915	2915
query39	1377	1341	1342	1341
query40	210	99	104	99
query41	56	43	44	43
query42	86	82	82	82
query43	677	623	711	623
query44	1143	708	727	708
query45	247	236	236	236
query46	1220	969	1004	969
query47	1846	1693	1782	1693
query48	1045	727	729	727
query49	614	379	388	379
query50	868	615	663	615
query51	4784	4712	4680	4680
query52	94	88	89	88
query53	472	327	329	327
query54	2684	2453	2454	2453
query55	81	92	87	87
query56	240	202	219	202
query57	1173	1152	1081	1081
query58	218	228	218	218
query59	4050	3784	3846	3784
query60	219	212	217	212
query61	96	96	105	96
query62	806	463	446	446
query63	492	352	348	348
query64	2569	1462	1514	1462
query65	3617	3535	3546	3535
query66	810	375	392	375
query67	16060	17036	15796	15796
query68	8377	660	655	655
query69	571	354	357	354
query70	1656	1324	1317	1317
query71	410	306	310	306
query72	6542	3506	3523	3506
query73	726	318	312	312
query74	6247	5832	5867	5832
query75	4631	3670	3693	3670
query76	4670	1152	1186	1152
query77	647	252	250	250
query78	12993	12523	11939	11939
query79	12967	643	662	643
query80	951	396	408	396
query81	500	236	232	232
query82	1353	105	104	104
query83	181	132	131	131
query84	266	72	71	71
query85	968	339	340	339
query86	334	303	293	293
query87	3250	3052	3022	3022
query88	5560	2314	2326	2314
query89	450	315	275	275
query90	1929	215	206	206
query91	174	136	134	134
query92	57	52	56	52
query93	6253	560	583	560
query94	849	212	207	207
query95	1114	1049	1045	1045
query96	657	327	322	322
query97	6573	6478	6395	6395
query98	190	178	165	165
query99	2944	918	831	831
Total cold run time: 316337 ms
Total hot run time: 203606 ms

@doris-robot
Copy link
Copy Markdown

ClickBench: Total hot run time: 31.07 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 9e0f441ce7796522f9acffdebef29361dbde624a, data reload: false

query1	0.02	0.02	0.02
query2	0.06	0.02	0.03
query3	0.24	0.05	0.05
query4	1.78	0.08	0.07
query5	0.54	0.52	0.52
query6	1.22	0.61	0.61
query7	0.01	0.01	0.01
query8	0.04	0.03	0.03
query9	0.54	0.53	0.51
query10	0.56	0.56	0.56
query11	0.12	0.09	0.09
query12	0.12	0.10	0.09
query13	0.62	0.61	0.61
query14	0.79	0.78	0.79
query15	0.79	0.76	0.76
query16	0.37	0.38	0.38
query17	1.00	1.00	1.01
query18	0.24	0.24	0.25
query19	1.95	1.86	1.83
query20	0.01	0.01	0.01
query21	15.49	0.56	0.55
query22	1.98	2.70	1.80
query23	17.79	1.05	0.99
query24	4.91	1.32	0.99
query25	0.33	0.12	0.03
query26	0.56	0.17	0.15
query27	0.05	0.05	0.03
query28	7.99	0.79	0.74
query29	12.73	2.31	2.33
query30	0.61	0.52	0.51
query31	2.82	0.39	0.37
query32	3.37	0.50	0.50
query33	3.07	3.06	3.20
query34	15.28	4.80	4.78
query35	4.88	4.83	4.90
query36	1.10	1.00	1.02
query37	0.06	0.04	0.05
query38	0.03	0.02	0.02
query39	0.02	0.02	0.01
query40	0.16	0.14	0.14
query41	0.07	0.01	0.02
query42	0.02	0.01	0.02
query43	0.02	0.01	0.01
Total cold run time: 104.36 s
Total hot run time: 31.07 s

@doris-robot
Copy link
Copy Markdown

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 9e0f441ce7796522f9acffdebef29361dbde624a with default session variables
Stream load json:         20 seconds loaded 2358488459 Bytes, about 112 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      32 seconds loaded 861443392 Bytes, about 25 MB/s
Insert into select:       21.1 seconds inserted 10000000 Rows, about 473K ops/s

Copy link
Copy Markdown
Contributor

@wangbo wangbo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions
Copy link
Copy Markdown
Contributor

PR approved by at least one committer and no changes requested.

@github-actions github-actions Bot added approved Indicates a PR has been approved by one committer. reviewed labels Jul 22, 2024
@github-actions
Copy link
Copy Markdown
Contributor

PR approved by anyone and no changes requested.

@yiguolei
Copy link
Copy Markdown
Contributor

Not pick this to 2.0 because this PR has many following prs and we fix some meta recovery bug in 2.1.

@yiguolei yiguolei closed this Jul 22, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by one committer. kind/test reviewed workload-group

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants