Skip to content

[feat](nereids) cost of column-prunning project is 0#35147

Closed
englefly wants to merge 1 commit intoapache:masterfrom
englefly:proj-cost
Closed

[feat](nereids) cost of column-prunning project is 0#35147
englefly wants to merge 1 commit intoapache:masterfrom
englefly:proj-cost

Conversation

@englefly
Copy link
Contributor

Proposed changes

it helps to choose mv if mv saves some expression effort.

Issue Number: close #xxx

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@englefly
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 42149 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 2ab1a102456f41d0d076e47dc99a800707b3f017, data reload: false

------ Round 1 ----------------------------------
q1	17604	4305	4246	4246
q2	2012	192	202	192
q3	10455	1254	1181	1181
q4	10194	736	763	736
q5	7451	2785	2810	2785
q6	227	131	133	131
q7	1034	615	606	606
q8	9241	2150	2113	2113
q9	10353	7062	7012	7012
q10	9503	3932	3880	3880
q11	452	242	237	237
q12	537	220	221	220
q13	17435	3218	3305	3218
q14	248	222	224	222
q15	502	465	467	465
q16	509	398	399	398
q17	1003	664	743	664
q18	8383	7726	7794	7726
q19	5730	1494	1471	1471
q20	650	319	314	314
q21	5271	4247	4045	4045
q22	366	287	290	287
Total cold run time: 119160 ms
Total hot run time: 42149 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4633	4406	4460	4406
q2	379	282	270	270
q3	3145	2948	2863	2863
q4	1954	1642	1702	1642
q5	5583	5604	5558	5558
q6	212	125	131	125
q7	2368	2042	2044	2042
q8	3238	3395	3446	3395
q9	8858	8923	8856	8856
q10	3900	3855	3786	3786
q11	614	489	483	483
q12	800	592	605	592
q13	16007	3129	3186	3129
q14	298	274	264	264
q15	514	485	481	481
q16	491	421	412	412
q17	1753	1499	1451	1451
q18	7771	7739	7629	7629
q19	1659	1596	1535	1535
q20	2036	1819	1791	1791
q21	12518	4793	4772	4772
q22	601	501	486	486
Total cold run time: 79332 ms
Total hot run time: 55968 ms

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label May 21, 2024
@github-actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Contributor

PR approved by anyone and no changes requested.

@Override
public Cost visitPhysicalProject(PhysicalProject<? extends Plan> physicalProject, PlanContext context) {
return CostV1.ofCpu(context.getSessionVariable(), 1);
long exprCount = physicalProject.getProjects().stream().filter(proj -> !(proj instanceof Slot)).count();
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should check output.size() is same with child's output.size().
otherwise, we may lose some column prune project when select best cost plan

@doris-robot
Copy link

TPC-DS: Total hot run time: 181653 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 2ab1a102456f41d0d076e47dc99a800707b3f017, data reload: false

query1	901	385	374	374
query2	7014	2333	2287	2287
query3	6647	213	214	213
query4	25000	21341	21170	21170
query5	4119	421	430	421
query6	262	170	175	170
query7	4581	299	289	289
query8	240	198	194	194
query9	8634	2411	2388	2388
query10	441	252	250	250
query11	15142	14207	14165	14165
query12	143	89	89	89
query13	1642	378	364	364
query14	9828	8553	8556	8553
query15	238	181	186	181
query16	7762	283	264	264
query17	1566	598	541	541
query18	1956	273	271	271
query19	209	169	148	148
query20	96	85	91	85
query21	196	130	135	130
query22	4209	3944	3916	3916
query23	34197	33464	33866	33464
query24	6834	2869	2898	2869
query25	486	366	380	366
query26	687	160	158	158
query27	1891	324	324	324
query28	3706	2041	2031	2031
query29	846	624	613	613
query30	251	179	195	179
query31	926	760	758	758
query32	59	52	54	52
query33	487	242	246	242
query34	878	489	486	486
query35	772	709	679	679
query36	1045	913	903	903
query37	112	69	73	69
query38	2889	2841	2771	2771
query39	1621	1539	1547	1539
query40	196	129	129	129
query41	49	45	45	45
query42	105	100	100	100
query43	607	537	530	530
query44	1120	733	742	733
query45	277	260	258	258
query46	1077	736	704	704
query47	1955	1859	1888	1859
query48	385	299	303	299
query49	775	438	413	413
query50	771	406	389	389
query51	6866	6784	6752	6752
query52	108	90	94	90
query53	353	279	277	277
query54	534	419	419	419
query55	78	74	76	74
query56	253	221	223	221
query57	1204	1147	1138	1138
query58	212	207	204	204
query59	3559	3169	3149	3149
query60	259	242	248	242
query61	95	117	89	89
query62	591	481	482	481
query63	319	288	284	284
query64	2589	1762	1711	1711
query65	3168	3111	3116	3111
query66	826	338	342	338
query67	15433	14980	14758	14758
query68	4579	533	536	533
query69	490	303	302	302
query70	1187	1079	1137	1079
query71	362	266	272	266
query72	7764	2775	2591	2591
query73	727	326	327	326
query74	6601	6198	6235	6198
query75	3454	2670	2632	2632
query76	2304	995	965	965
query77	381	275	275	275
query78	10947	10101	10046	10046
query79	1616	517	522	517
query80	995	454	438	438
query81	523	241	241	241
query82	798	97	94	94
query83	247	217	165	165
query84	247	84	83	83
query85	955	277	267	267
query86	467	323	309	309
query87	3286	3120	3144	3120
query88	3543	2343	2337	2337
query89	475	390	383	383
query90	1995	198	193	193
query91	129	98	102	98
query92	64	51	49	49
query93	1660	507	491	491
query94	1150	185	184	184
query95	398	298	299	298
query96	594	271	261	261
query97	3188	2989	3011	2989
query98	244	219	215	215
query99	1240	945	955	945
Total cold run time: 264887 ms
Total hot run time: 181653 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.67 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 2ab1a102456f41d0d076e47dc99a800707b3f017, data reload: false

query1	0.04	0.03	0.04
query2	0.09	0.04	0.04
query3	0.23	0.05	0.05
query4	1.67	0.07	0.07
query5	0.48	0.50	0.50
query6	1.13	0.72	0.72
query7	0.02	0.02	0.01
query8	0.05	0.04	0.04
query9	0.54	0.49	0.49
query10	0.54	0.54	0.54
query11	0.15	0.11	0.11
query12	0.15	0.12	0.12
query13	0.59	0.59	0.60
query14	0.78	0.78	0.79
query15	0.82	0.81	0.83
query16	0.36	0.37	0.38
query17	1.01	0.98	0.95
query18	0.22	0.25	0.22
query19	1.88	1.71	1.80
query20	0.01	0.01	0.01
query21	15.62	0.68	0.68
query22	4.42	6.46	2.02
query23	18.28	1.36	1.38
query24	1.88	0.27	0.20
query25	0.15	0.08	0.08
query26	0.26	0.17	0.17
query27	0.08	0.07	0.08
query28	13.30	1.02	1.01
query29	13.24	3.26	3.23
query30	0.25	0.05	0.05
query31	2.86	0.38	0.39
query32	3.28	0.47	0.46
query33	2.92	2.87	3.00
query34	17.22	4.37	4.40
query35	4.44	4.49	4.58
query36	0.67	0.46	0.46
query37	0.19	0.16	0.16
query38	0.16	0.15	0.15
query39	0.05	0.04	0.04
query40	0.17	0.14	0.14
query41	0.10	0.05	0.05
query42	0.06	0.05	0.04
query43	0.04	0.03	0.03
Total cold run time: 110.4 s
Total hot run time: 30.67 s

@morrySnow morrySnow closed this May 22, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by one committer. reviewed

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants