[feat](Nereids): Add support for slot pruning in functional dependencies by keanji-x · Pull Request #37045 · apache/doris

keanji-x · 2024-07-01T02:22:30Z

Proposed changes

Implement slot pruning functionality for functional dependencies to optimize performance and resource utilization. This enhancement allows for more efficient handling of dependencies by removing unnecessary slots.

doris-robot · 2024-07-01T02:22:35Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

morrySnow · 2024-07-01T06:23:21Z

run buildall

github-actions · 2024-07-01T06:24:02Z

PR approved by at least one committer and no changes requested.

github-actions · 2024-07-01T06:24:04Z

PR approved by anyone and no changes requested.

keanji-x · 2024-07-01T07:43:25Z

run buildall

doris-robot · 2024-07-01T08:42:01Z

TPC-H: Total hot run time: 40033 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 427bfb1db16b3c7d81a454ed143c9960d527acc1, data reload: false

------ Round 1 ----------------------------------
q1	17798	4476	4285	4285
q2	2279	191	201	191
q3	10522	1195	1071	1071
q4	10242	811	787	787
q5	7502	3007	2680	2680
q6	223	138	139	138
q7	963	602	601	601
q8	9244	2111	2082	2082
q9	8905	6505	6517	6505
q10	9101	3729	3741	3729
q11	452	246	238	238
q12	401	233	231	231
q13	17864	3013	3001	3001
q14	276	224	235	224
q15	513	475	480	475
q16	511	391	382	382
q17	992	675	620	620
q18	8003	7578	7409	7409
q19	5596	1511	1435	1435
q20	680	321	325	321
q21	4991	3294	3892	3294
q22	397	334	337	334
Total cold run time: 117455 ms
Total hot run time: 40033 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4363	4234	4214	4214
q2	373	278	275	275
q3	2998	2766	2780	2766
q4	2073	1713	1670	1670
q5	5491	5634	5564	5564
q6	228	136	134	134
q7	2213	1857	1840	1840
q8	3242	3449	3421	3421
q9	8712	8864	8720	8720
q10	4150	3727	3708	3708
q11	604	480	485	480
q12	821	647	626	626
q13	17825	3205	3161	3161
q14	333	290	281	281
q15	532	494	482	482
q16	489	433	440	433
q17	1847	1549	1503	1503
q18	7817	7524	7296	7296
q19	1675	1488	1476	1476
q20	2094	1794	1799	1794
q21	6433	4669	4668	4668
q22	631	559	578	559
Total cold run time: 74944 ms
Total hot run time: 55071 ms

doris-robot · 2024-07-01T08:53:27Z

TPC-DS: Total hot run time: 173212 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 427bfb1db16b3c7d81a454ed143c9960d527acc1, data reload: false

query1	1191	397	373	373
query2	6462	2218	2353	2218
query3	6642	209	216	209
query4	18830	17580	17383	17383
query5	4208	515	508	508
query6	265	185	166	166
query7	4597	304	294	294
query8	314	318	289	289
query9	8509	2407	2375	2375
query10	602	327	295	295
query11	10618	10251	10084	10084
query12	138	86	85	85
query13	1666	376	383	376
query14	9573	7224	6959	6959
query15	244	179	183	179
query16	7834	277	271	271
query17	1437	569	570	569
query18	1972	292	285	285
query19	206	156	159	156
query20	96	86	89	86
query21	220	136	135	135
query22	4337	3974	4108	3974
query23	34059	33177	33175	33175
query24	12019	2817	2869	2817
query25	690	421	388	388
query26	1868	157	163	157
query27	2910	311	317	311
query28	7559	2058	2055	2055
query29	1192	664	629	629
query30	292	151	151	151
query31	967	745	756	745
query32	101	58	57	57
query33	793	311	299	299
query34	929	482	490	482
query35	765	662	629	629
query36	1075	935	903	903
query37	289	77	80	77
query38	2874	2747	2752	2747
query39	919	819	798	798
query40	286	133	132	132
query41	55	55	56	55
query42	121	103	104	103
query43	591	541	564	541
query44	1198	731	731	731
query45	207	163	162	162
query46	1079	758	721	721
query47	1848	1761	1765	1761
query48	374	296	310	296
query49	1143	436	433	433
query50	781	394	405	394
query51	7140	6859	6737	6737
query52	109	96	97	96
query53	362	304	306	304
query54	938	454	457	454
query55	79	77	75	75
query56	314	386	277	277
query57	1143	1070	1055	1055
query58	244	245	269	245
query59	3523	3225	3074	3074
query60	317	296	289	289
query61	97	97	120	97
query62	635	440	472	440
query63	327	305	296	296
query64	9835	2312	1773	1773
query65	3159	3139	3105	3105
query66	1376	331	321	321
query67	15480	14797	14833	14797
query68	9055	555	579	555
query69	673	358	335	335
query70	1319	1138	1137	1137
query71	523	291	282	282
query72	8408	5540	5337	5337
query73	2156	335	325	325
query74	6040	5561	5441	5441
query75	4768	2676	2614	2614
query76	5073	974	904	904
query77	701	315	313	313
query78	10550	9860	9872	9860
query79	8173	527	534	527
query80	1257	493	473	473
query81	534	216	219	216
query82	761	111	105	105
query83	216	170	171	170
query84	273	82	89	82
query85	1264	282	270	270
query86	396	308	338	308
query87	3274	3136	3102	3102
query88	4859	2390	2352	2352
query89	554	385	390	385
query90	2028	187	184	184
query91	130	99	104	99
query92	63	50	48	48
query93	6685	506	507	506
query94	1281	186	194	186
query95	410	316	323	316
query96	613	273	279	273
query97	3198	3047	3006	3006
query98	206	205	197	197
query99	1187	851	884	851
Total cold run time: 298651 ms
Total hot run time: 173212 ms

doris-robot · 2024-07-01T08:58:58Z

ClickBench: Total hot run time: 30.99 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 427bfb1db16b3c7d81a454ed143c9960d527acc1, data reload: false

query1	0.04	0.03	0.03
query2	0.07	0.04	0.04
query3	0.22	0.05	0.05
query4	1.67	0.08	0.09
query5	0.50	0.47	0.49
query6	1.16	0.72	0.74
query7	0.02	0.01	0.02
query8	0.05	0.04	0.04
query9	0.56	0.50	0.48
query10	0.53	0.52	0.54
query11	0.15	0.11	0.12
query12	0.15	0.12	0.12
query13	0.59	0.59	0.62
query14	0.78	0.77	0.78
query15	0.86	0.80	0.82
query16	0.37	0.37	0.36
query17	1.05	1.03	1.04
query18	0.23	0.23	0.25
query19	1.84	1.72	1.70
query20	0.01	0.01	0.01
query21	15.43	0.74	0.67
query22	4.36	6.58	2.26
query23	18.32	1.35	1.30
query24	2.13	0.23	0.23
query25	0.15	0.09	0.08
query26	0.27	0.18	0.19
query27	0.08	0.08	0.09
query28	13.22	1.02	0.99
query29	12.57	3.23	3.29
query30	0.25	0.06	0.05
query31	2.87	0.39	0.40
query32	3.26	0.48	0.49
query33	2.89	2.88	2.95
query34	17.17	4.48	4.43
query35	4.47	4.50	4.49
query36	0.65	0.49	0.48
query37	0.18	0.16	0.16
query38	0.15	0.15	0.16
query39	0.05	0.03	0.03
query40	0.17	0.15	0.15
query41	0.10	0.04	0.05
query42	0.05	0.04	0.04
query43	0.05	0.04	0.04
Total cold run time: 109.69 s
Total hot run time: 30.99 s

keanji-x · 2024-07-01T15:52:16Z

run buildall

keanji-x · 2024-07-01T15:52:43Z

run buildall

doris-robot · 2024-07-01T17:11:32Z

TPC-H: Total hot run time: 39855 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit d8c503d95e02577a8e419ec784245dde3d4b32d0, data reload: false

------ Round 1 ----------------------------------
q1	17637	4481	4401	4401
q2	2023	189	186	186
q3	10483	1192	1014	1014
q4	10185	838	827	827
q5	7520	2679	2648	2648
q6	222	145	138	138
q7	972	602	605	602
q8	9237	2109	2079	2079
q9	8917	6555	6516	6516
q10	8991	3709	3744	3709
q11	476	233	234	233
q12	489	233	242	233
q13	17751	2971	3005	2971
q14	271	231	230	230
q15	534	478	486	478
q16	518	375	383	375
q17	976	676	754	676
q18	8023	7431	7353	7353
q19	7114	1569	1400	1400
q20	676	329	327	327
q21	5081	3134	3844	3134
q22	390	325	335	325
Total cold run time: 118486 ms
Total hot run time: 39855 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4368	4252	4193	4193
q2	367	271	264	264
q3	3007	2891	2859	2859
q4	1978	1760	1775	1760
q5	5536	5539	5461	5461
q6	222	132	140	132
q7	2157	1921	1862	1862
q8	3258	3398	3433	3398
q9	8773	8824	8844	8824
q10	4143	3779	3777	3777
q11	578	487	498	487
q12	841	653	641	641
q13	16022	3187	3187	3187
q14	302	281	279	279
q15	525	470	520	470
q16	479	432	441	432
q17	1820	1537	1503	1503
q18	8194	8885	7809	7809
q19	1798	1670	1680	1670
q20	2123	1886	1859	1859
q21	5310	4945	4892	4892
q22	631	549	553	549
Total cold run time: 72432 ms
Total hot run time: 56308 ms

doris-robot · 2024-07-01T17:22:40Z

TPC-DS: Total hot run time: 173416 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit d8c503d95e02577a8e419ec784245dde3d4b32d0, data reload: false

query1	913	377	380	377
query2	6448	2392	2385	2385
query3	6629	209	215	209
query4	20593	17526	17279	17279
query5	3675	482	471	471
query6	267	157	171	157
query7	4615	293	291	291
query8	299	265	276	265
query9	8453	2378	2357	2357
query10	563	313	269	269
query11	10451	10035	10088	10035
query12	121	86	86	86
query13	1640	363	374	363
query14	9916	7790	7188	7188
query15	229	189	185	185
query16	7918	280	271	271
query17	1879	552	540	540
query18	2041	285	276	276
query19	204	156	156	156
query20	90	82	84	82
query21	209	136	125	125
query22	4223	3974	3978	3974
query23	33807	33482	33612	33482
query24	10558	2875	2912	2875
query25	612	426	389	389
query26	718	157	157	157
query27	2272	333	320	320
query28	5949	2147	2133	2133
query29	915	670	662	662
query30	249	157	161	157
query31	1010	767	773	767
query32	97	56	57	56
query33	708	373	280	280
query34	864	475	489	475
query35	756	630	611	611
query36	1131	971	978	971
query37	140	80	78	78
query38	2958	2839	2875	2839
query39	874	819	830	819
query40	210	124	127	124
query41	53	56	53	53
query42	107	102	99	99
query43	610	526	538	526
query44	1076	737	716	716
query45	194	164	171	164
query46	1067	706	719	706
query47	1850	1808	1771	1771
query48	368	301	293	293
query49	847	397	417	397
query50	760	381	380	380
query51	6922	6767	6754	6754
query52	112	90	93	90
query53	352	288	288	288
query54	869	440	432	432
query55	78	75	72	72
query56	286	275	266	266
query57	1104	1044	1061	1044
query58	236	251	289	251
query59	3453	3069	3103	3069
query60	293	280	275	275
query61	94	87	94	87
query62	637	448	439	439
query63	309	288	290	288
query64	8503	2227	1774	1774
query65	3188	3110	3111	3110
query66	748	327	324	324
query67	15710	15030	15199	15030
query68	7120	539	531	531
query69	725	418	326	326
query70	1194	1105	1113	1105
query71	475	316	272	272
query72	8481	4972	5300	4972
query73	824	322	323	322
query74	5936	5440	5453	5440
query75	4695	2645	2651	2645
query76	4274	948	947	947
query77	753	295	297	295
query78	10460	9918	9775	9775
query79	5112	510	511	510
query80	998	479	460	460
query81	544	225	220	220
query82	459	104	110	104
query83	313	171	169	169
query84	270	85	82	82
query85	956	274	273	273
query86	353	308	328	308
query87	3311	3060	3053	3053
query88	3712	2367	2352	2352
query89	497	374	373	373
query90	1968	188	186	186
query91	127	98	94	94
query92	60	50	47	47
query93	5524	513	517	513
query94	1324	194	183	183
query95	398	315	310	310
query96	613	261	265	261
query97	3241	3020	3007	3007
query98	204	203	199	199
query99	1274	851	848	848
Total cold run time: 281225 ms
Total hot run time: 173416 ms

doris-robot · 2024-07-01T17:28:11Z

ClickBench: Total hot run time: 30.31 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit d8c503d95e02577a8e419ec784245dde3d4b32d0, data reload: false

query1	0.04	0.03	0.03
query2	0.08	0.03	0.04
query3	0.22	0.04	0.05
query4	1.67	0.07	0.07
query5	0.49	0.51	0.49
query6	1.15	0.72	0.72
query7	0.02	0.01	0.01
query8	0.05	0.04	0.04
query9	0.55	0.50	0.48
query10	0.54	0.54	0.54
query11	0.15	0.12	0.11
query12	0.15	0.12	0.12
query13	0.60	0.59	0.59
query14	0.78	0.76	0.78
query15	0.85	0.81	0.81
query16	0.37	0.37	0.35
query17	0.98	0.98	0.94
query18	0.23	0.25	0.24
query19	1.91	1.67	1.70
query20	0.02	0.01	0.01
query21	15.44	0.79	0.66
query22	4.46	6.78	1.79
query23	18.29	1.31	1.27
query24	2.18	0.24	0.22
query25	0.17	0.08	0.08
query26	0.26	0.17	0.17
query27	0.08	0.08	0.08
query28	13.23	1.01	0.99
query29	12.61	3.29	3.28
query30	0.25	0.06	0.05
query31	2.85	0.39	0.40
query32	3.26	0.47	0.46
query33	2.87	2.94	2.88
query34	17.20	4.42	4.42
query35	4.49	4.44	4.48
query36	0.66	0.48	0.48
query37	0.19	0.15	0.17
query38	0.15	0.16	0.15
query39	0.04	0.04	0.03
query40	0.17	0.15	0.15
query41	0.10	0.04	0.04
query42	0.05	0.05	0.05
query43	0.05	0.03	0.03
Total cold run time: 109.9 s
Total hot run time: 30.31 s

…es (#37045) Implement slot pruning functionality for functional dependencies to optimize performance and resource utilization. This enhancement allows for more efficient handling of dependencies by removing unnecessary slots.

keanji-x changed the title ~~[feat](Nereids): support prune slot in func deps~~ [feat](Nereids): Add support for slot pruning in functional dependencies Jul 1, 2024

morrySnow previously approved these changes Jul 1, 2024

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label Jul 1, 2024

github-actions bot added the reviewed label Jul 1, 2024

keanji-x dismissed morrySnow’s stale review via eea9d73 July 1, 2024 07:39

keanji-x force-pushed the support_prune_func_deps branch from eea9d73 to 427bfb1 Compare July 1, 2024 07:42

keanji-x added 2 commits July 1, 2024 20:51

support prune output in equal set and func deps

aeeb652

add test

9ff0311

keanji-x force-pushed the support_prune_func_deps branch from 427bfb1 to cb69f2b Compare July 1, 2024 15:48

prune slot according output

d8c503d

keanji-x force-pushed the support_prune_func_deps branch from cb69f2b to d8c503d Compare July 1, 2024 15:51

keanji-x requested a review from morrySnow July 2, 2024 03:34

morrySnow approved these changes Jul 2, 2024

View reviewed changes

morrySnow added the not-merge/2.1 label Jul 2, 2024

seawinde approved these changes Jul 2, 2024

View reviewed changes

morrySnow merged commit 40dbb59 into apache:master Jul 2, 2024

dataroaring added the dev/3.0.1-merged label Aug 5, 2024

gavinchou mentioned this pull request Aug 19, 2024

Release Note 3.0.1 #39570

Closed

Conversation

keanji-x commented Jul 1, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Proposed changes

Uh oh!

doris-robot commented Jul 1, 2024

Uh oh!

morrySnow commented Jul 1, 2024

Uh oh!

github-actions bot commented Jul 1, 2024

Uh oh!

github-actions bot commented Jul 1, 2024

Uh oh!

keanji-x commented Jul 1, 2024

Uh oh!

doris-robot commented Jul 1, 2024

Uh oh!

doris-robot commented Jul 1, 2024

Uh oh!

doris-robot commented Jul 1, 2024

Uh oh!

keanji-x commented Jul 1, 2024

Uh oh!

keanji-x commented Jul 1, 2024

Uh oh!

doris-robot commented Jul 1, 2024

Uh oh!

doris-robot commented Jul 1, 2024

Uh oh!

doris-robot commented Jul 1, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

keanji-x commented Jul 1, 2024 •

edited

Loading