Skip to content

[pipelineX](scanner) Use the actual instances num when ignore data distribution#32081

Merged
Gabriel39 merged 1 commit intoapache:masterfrom
Gabriel39:opt_0312
Mar 12, 2024
Merged

[pipelineX](scanner) Use the actual instances num when ignore data distribution#32081
Gabriel39 merged 1 commit intoapache:masterfrom
Gabriel39:opt_0312

Conversation

@Gabriel39
Copy link
Contributor

@Gabriel39 Gabriel39 commented Mar 12, 2024

Proposed changes

Create clickbench table with 48buckets and use 92 cores machine, Q29 is accelerated from 11s+ to 1.8s+

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

@Gabriel39
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 34.97% (8554/24459)
Line Coverage: 26.81% (69496/259260)
Region Coverage: 26.06% (36066/138415)
Branch Coverage: 23.02% (18432/80070)
Coverage Report: http://coverage.selectdb-in.cc/coverage/4a1b373a30441a6a5bfcda0697f00663115e6b19_4a1b373a30441a6a5bfcda0697f00663115e6b19/report/index.html

@doris-robot
Copy link

TPC-H: Total hot run time: 36353 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 4a1b373a30441a6a5bfcda0697f00663115e6b19, data reload: false

------ Round 1 ----------------------------------
q1	17630	4223	4086	4086
q2	2025	157	148	148
q3	10566	935	883	883
q4	4653	721	681	681
q5	7593	2695	2768	2695
q6	180	125	125	125
q7	1198	815	793	793
q8	9477	2005	2036	2005
q9	7359	6443	6450	6443
q10	8192	2472	2460	2460
q11	428	227	231	227
q12	806	288	287	287
q13	17946	3037	2976	2976
q14	275	245	246	245
q15	485	453	442	442
q16	485	400	383	383
q17	935	554	557	554
q18	6492	5773	5751	5751
q19	1575	1406	1418	1406
q20	532	279	276	276
q21	7336	3199	3334	3199
q22	801	289	288	288
Total cold run time: 106969 ms
Total hot run time: 36353 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4069	4056	4075	4056
q2	319	226	234	226
q3	2952	2817	2857	2817
q4	1828	1532	1554	1532
q5	5176	5235	5211	5211
q6	194	124	118	118
q7	2239	1800	1809	1800
q8	3126	3258	3259	3258
q9	8538	8487	8518	8487
q10	6090	3687	3670	3670
q11	534	434	435	434
q12	720	566	560	560
q13	13093	2854	2858	2854
q14	271	251	255	251
q15	485	448	428	428
q16	440	423	405	405
q17	1721	1452	1469	1452
q18	7533	7254	6939	6939
q19	4736	1518	1485	1485
q20	1903	1708	1675	1675
q21	4685	4582	4555	4555
q22	551	461	458	458
Total cold run time: 71203 ms
Total hot run time: 52671 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 186669 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 4a1b373a30441a6a5bfcda0697f00663115e6b19, data reload: false

query1	923	353	336	336
query2	7599	2032	1929	1929
query3	6708	219	210	210
query4	31221	20645	20658	20645
query5	4433	415	401	401
query6	284	178	175	175
query7	4620	297	294	294
query8	224	170	167	167
query9	9899	2299	2282	2282
query10	456	238	241	238
query11	17085	13999	14086	13999
query12	131	87	83	83
query13	1615	414	418	414
query14	12532	10729	10663	10663
query15	342	194	193	193
query16	8167	275	268	268
query17	2065	574	546	546
query18	2108	295	285	285
query19	347	161	161	161
query20	91	110	88	88
query21	200	127	124	124
query22	4627	4439	4484	4439
query23	31843	31096	31265	31096
query24	10658	2860	2843	2843
query25	626	381	377	377
query26	1172	160	163	160
query27	2400	354	356	354
query28	7444	1892	1854	1854
query29	887	640	620	620
query30	302	148	152	148
query31	963	696	712	696
query32	104	63	55	55
query33	779	257	254	254
query34	1041	483	491	483
query35	819	614	617	614
query36	1006	851	841	841
query37	121	80	78	78
query38	3275	3152	3152	3152
query39	1433	1513	1364	1364
query40	204	113	110	110
query41	49	49	47	47
query42	98	96	102	96
query43	480	445	452	445
query44	1236	712	720	712
query45	282	256	262	256
query46	1094	698	692	692
query47	1674	1614	1615	1614
query48	450	359	378	359
query49	1099	342	360	342
query50	749	387	393	387
query51	6659	6630	6575	6575
query52	99	96	91	91
query53	349	290	283	283
query54	311	237	244	237
query55	87	79	83	79
query56	245	228	234	228
query57	1067	996	1021	996
query58	230	200	224	200
query59	2735	2689	2634	2634
query60	266	243	251	243
query61	111	111	112	111
query62	630	420	407	407
query63	299	279	278	278
query64	5238	3922	3479	3479
query65	3079	3039	2990	2990
query66	825	355	321	321
query67	14860	14162	14154	14154
query68	8416	546	543	543
query69	633	384	373	373
query70	1227	1150	1132	1132
query71	528	268	268	268
query72	6333	2654	2522	2522
query73	722	323	324	323
query74	7094	6561	6517	6517
query75	9911	8084	7995	7995
query76	5105	917	935	917
query77	681	270	255	255
query78	10412	9593	9573	9573
query79	10215	512	524	512
query80	1671	434	446	434
query81	530	208	215	208
query82	854	203	198	198
query83	205	147	145	145
query84	282	79	77	77
query85	1467	340	313	313
query86	422	306	301	301
query87	3440	3252	3228	3228
query88	4821	2348	2331	2331
query89	500	358	401	358
query90	2043	179	178	178
query91	164	135	127	127
query92	63	49	49	49
query93	6791	509	495	495
query94	1188	180	181	180
query95	439	330	326	326
query96	606	278	274	274
query97	4377	4270	4308	4270
query98	225	213	205	205
query99	1135	747	761	747
Total cold run time: 317555 ms
Total hot run time: 186669 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 29.95 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 4a1b373a30441a6a5bfcda0697f00663115e6b19, data reload: false

query1	0.05	0.03	0.03
query2	0.06	0.03	0.02
query3	0.23	0.06	0.06
query4	1.67	0.10	0.09
query5	0.50	0.47	0.48
query6	1.24	0.63	0.62
query7	0.02	0.01	0.02
query8	0.03	0.03	0.03
query9	0.57	0.50	0.49
query10	0.54	0.55	0.55
query11	0.12	0.09	0.10
query12	0.11	0.09	0.10
query13	0.58	0.57	0.57
query14	0.76	0.76	0.73
query15	0.80	0.78	0.78
query16	0.38	0.39	0.40
query17	1.01	1.04	0.99
query18	0.21	0.23	0.23
query19	1.78	1.67	1.64
query20	0.02	0.01	0.01
query21	15.43	0.57	0.56
query22	2.58	2.40	1.53
query23	16.92	0.82	0.75
query24	3.59	2.46	0.85
query25	0.14	0.29	0.13
query26	0.55	0.14	0.13
query27	0.03	0.03	0.04
query28	10.80	0.85	0.88
query29	12.56	3.32	3.21
query30	0.69	0.58	0.61
query31	2.80	0.33	0.34
query32	3.39	0.44	0.44
query33	2.75	2.76	2.86
query34	16.12	4.33	4.32
query35	4.32	4.31	4.32
query36	1.09	1.00	1.02
query37	0.07	0.05	0.05
query38	0.05	0.03	0.03
query39	0.02	0.01	0.02
query40	0.17	0.13	0.13
query41	0.07	0.02	0.02
query42	0.02	0.02	0.02
query43	0.02	0.02	0.03
Total cold run time: 104.86 s
Total hot run time: 29.95 s

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 4a1b373a30441a6a5bfcda0697f00663115e6b19 with default session variables
Stream load json:         18 seconds loaded 2358488459 Bytes, about 124 MB/s
Stream load orc:          59 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       16.2 seconds inserted 10000000 Rows, about 617K ops/s

Copy link
Contributor

@HappenLee HappenLee left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Mar 12, 2024
@github-actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Contributor

PR approved by anyone and no changes requested.

@Gabriel39 Gabriel39 merged commit db2f260 into apache:master Mar 12, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by one committer. reviewed

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants