Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[improvement](nereids) Simplify ScanNode projection handling by removing redundant conditions #40801

Merged
merged 1 commit into from
Sep 24, 2024

Conversation

zy-kkk
Copy link
Member

@zy-kkk zy-kkk commented Sep 13, 2024

This PR simplifies the handling of ScanNode projection logic. Previously, the code included multiple conditional checks to determine whether a projectionTuple should be generated. These conditions have been removed, and now projectionTuple is always generated for ScanNode, ensuring a consistent projection setup. Additionally, redundant handling of SlotId and SlotRef has been eliminated, making the code cleaner and easier to maintain. The behavior for OlapScanNode remains unchanged.

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@zy-kkk
Copy link
Member Author

zy-kkk commented Sep 13, 2024

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 42806 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit e0afba42539846b3b4663b6e937023612936442a, data reload: false

------ Round 1 ----------------------------------
q1	17612	7381	7264	7264
q2	2043	196	177	177
q3	10555	1265	1434	1265
q4	10384	983	1078	983
q5	7717	3146	3140	3140
q6	238	150	148	148
q7	1048	640	622	622
q8	9456	2002	2034	2002
q9	6635	6308	6322	6308
q10	7011	2564	2561	2561
q11	434	253	257	253
q12	408	229	227	227
q13	17765	3010	3033	3010
q14	287	247	250	247
q15	583	526	540	526
q16	527	441	426	426
q17	977	916	902	902
q18	7379	6812	6685	6685
q19	1378	1240	1231	1231
q20	607	339	332	332
q21	3901	3554	3483	3483
q22	1074	1014	1016	1014
Total cold run time: 108019 ms
Total hot run time: 42806 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7519	7174	7169	7169
q2	371	251	239	239
q3	3095	3124	3008	3008
q4	2085	2119	2070	2070
q5	5756	5674	5753	5674
q6	241	149	154	149
q7	2163	1768	1787	1768
q8	3450	3482	3537	3482
q9	8980	9082	8985	8985
q10	3566	3717	3702	3702
q11	599	502	501	501
q12	807	657	603	603
q13	9288	3228	3213	3213
q14	351	306	293	293
q15	600	528	531	528
q16	513	474	492	474
q17	1840	1801	1784	1784
q18	8722	8086	8099	8086
q19	1792	1775	1751	1751
q20	2136	1907	1890	1890
q21	5925	5695	5749	5695
q22	1190	1080	1061	1061
Total cold run time: 70989 ms
Total hot run time: 62125 ms

@@ -2008,20 +2008,10 @@ public PlanFragment visitPhysicalProject(PhysicalProject<? extends Plan> project
// if we change the output order in translate project, the upper node will receive wrong order
// tuple, since they get the order from project.getOutput() not scan.getOutput()./
List<SlotId> slotIdsByOrder = Lists.newArrayList();
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

remove this variable

@zy-kkk zy-kkk force-pushed the slots_order branch 2 times, most recently from 4803763 to 667d867 Compare September 18, 2024 04:19
@morrySnow
Copy link
Contributor

run buildall

Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Sep 19, 2024
Copy link
Contributor

PR approved by anyone and no changes requested.

@doris-robot
Copy link

TPC-H: Total hot run time: 41251 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 667d8678a5e637be12495a86aac1355ce78212f2, data reload: false

------ Round 1 ----------------------------------
q1	17992	7726	7313	7313
q2	2934	168	180	168
q3	10554	1161	1172	1161
q4	10219	790	795	790
q5	7770	2907	2816	2816
q6	234	151	151	151
q7	973	625	626	625
q8	9356	1956	2004	1956
q9	6611	6443	6418	6418
q10	6958	2294	2308	2294
q11	450	248	243	243
q12	403	220	222	220
q13	17782	3011	2993	2993
q14	245	212	223	212
q15	572	528	530	528
q16	661	626	612	612
q17	999	610	561	561
q18	7197	6633	6667	6633
q19	1385	1039	1138	1039
q20	598	292	286	286
q21	4034	3372	3250	3250
q22	1116	982	1040	982
Total cold run time: 109043 ms
Total hot run time: 41251 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7260	7282	7224	7224
q2	329	229	225	225
q3	2915	2816	2817	2816
q4	1965	1715	1725	1715
q5	5457	5477	5483	5477
q6	224	138	142	138
q7	2092	1749	1722	1722
q8	3228	3422	3404	3404
q9	8625	8573	8581	8573
q10	3498	3452	3431	3431
q11	585	485	469	469
q12	802	596	588	588
q13	7732	2999	3015	2999
q14	287	269	262	262
q15	554	496	523	496
q16	689	699	678	678
q17	1805	1583	1547	1547
q18	7752	7494	7403	7403
q19	1672	1539	1407	1407
q20	2062	1812	1825	1812
q21	5421	5206	5059	5059
q22	1112	1025	1013	1013
Total cold run time: 66066 ms
Total hot run time: 58458 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 190544 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 667d8678a5e637be12495a86aac1355ce78212f2, data reload: false

query1	991	384	364	364
query2	6539	2167	2045	2045
query3	6703	211	219	211
query4	34817	23517	23724	23517
query5	4353	471	491	471
query6	261	165	157	157
query7	4636	298	312	298
query8	288	218	226	218
query9	9665	2635	2632	2632
query10	480	274	289	274
query11	18198	15214	15376	15214
query12	156	98	100	98
query13	1631	422	411	411
query14	10624	7045	7435	7045
query15	278	176	184	176
query16	8168	486	480	480
query17	1679	582	569	569
query18	2144	306	309	306
query19	360	155	150	150
query20	118	105	110	105
query21	211	104	103	103
query22	4531	4147	4289	4147
query23	34967	34378	34061	34061
query24	11042	2880	2812	2812
query25	672	417	412	412
query26	1668	160	162	160
query27	2845	294	304	294
query28	8205	2492	2425	2425
query29	989	429	431	429
query30	323	166	156	156
query31	1017	810	822	810
query32	104	56	60	56
query33	790	296	306	296
query34	961	507	492	492
query35	856	723	713	713
query36	1074	934	956	934
query37	158	95	88	88
query38	3985	3955	3874	3874
query39	1471	1416	1396	1396
query40	287	101	99	99
query41	52	47	49	47
query42	122	103	97	97
query43	519	475	501	475
query44	1209	824	800	800
query45	197	174	171	171
query46	1135	705	720	705
query47	1930	1813	1829	1813
query48	482	384	374	374
query49	1191	413	408	408
query50	824	409	413	409
query51	6961	6921	6755	6755
query52	106	87	87	87
query53	267	192	184	184
query54	1240	464	474	464
query55	80	77	77	77
query56	282	274	266	266
query57	1257	1075	1087	1075
query58	250	270	245	245
query59	3188	2953	2908	2908
query60	304	280	301	280
query61	105	100	106	100
query62	821	666	676	666
query63	218	187	192	187
query64	5250	639	619	619
query65	3273	3203	3248	3203
query66	1264	313	308	308
query67	15736	15567	15463	15463
query68	4958	574	555	555
query69	438	302	296	296
query70	1148	1147	1119	1119
query71	349	302	277	277
query72	6392	3952	4008	3952
query73	780	342	360	342
query74	9839	8955	9036	8955
query75	3380	2635	2672	2635
query76	3298	945	877	877
query77	416	287	295	287
query78	9838	9142	9199	9142
query79	1406	617	601	601
query80	1141	475	442	442
query81	547	238	245	238
query82	929	141	144	141
query83	215	135	136	135
query84	230	79	89	79
query85	1178	286	280	280
query86	355	297	264	264
query87	4446	4264	4330	4264
query88	3584	2413	2375	2375
query89	399	287	287	287
query90	1882	188	189	188
query91	204	148	145	145
query92	59	48	53	48
query93	1068	546	538	538
query94	792	294	295	294
query95	356	265	255	255
query96	618	281	286	281
query97	3221	3130	3084	3084
query98	213	210	191	191
query99	1536	1293	1303	1293
Total cold run time: 301723 ms
Total hot run time: 190544 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 33.43 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 667d8678a5e637be12495a86aac1355ce78212f2, data reload: false

query1	0.05	0.05	0.04
query2	0.06	0.03	0.03
query3	0.23	0.06	0.06
query4	1.64	0.10	0.10
query5	0.50	0.51	0.50
query6	1.13	0.73	0.74
query7	0.02	0.02	0.01
query8	0.03	0.03	0.02
query9	0.56	0.50	0.49
query10	0.54	0.55	0.53
query11	0.14	0.12	0.10
query12	0.14	0.11	0.10
query13	0.62	0.60	0.60
query14	3.01	3.00	2.97
query15	0.89	0.83	0.83
query16	0.38	0.37	0.38
query17	0.98	1.03	1.03
query18	0.23	0.22	0.21
query19	1.90	1.78	1.96
query20	0.01	0.01	0.00
query21	15.34	0.60	0.57
query22	2.52	2.72	2.47
query23	17.39	0.92	0.76
query24	2.41	1.71	0.96
query25	0.28	0.09	0.06
query26	0.49	0.14	0.14
query27	0.03	0.04	0.04
query28	10.50	1.09	1.08
query29	12.58	3.22	3.18
query30	0.24	0.06	0.06
query31	2.89	0.39	0.39
query32	3.26	0.46	0.47
query33	3.00	3.05	3.05
query34	16.86	4.50	4.47
query35	4.54	4.52	4.46
query36	0.67	0.50	0.48
query37	0.08	0.06	0.06
query38	0.04	0.03	0.04
query39	0.03	0.02	0.02
query40	0.15	0.12	0.13
query41	0.08	0.02	0.02
query42	0.03	0.03	0.02
query43	0.03	0.03	0.03
Total cold run time: 106.5 s
Total hot run time: 33.43 s

@zy-kkk
Copy link
Member Author

zy-kkk commented Sep 23, 2024

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 40520 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 4228ed33fb15ddb7f378a8616205a57791d34022, data reload: false

------ Round 1 ----------------------------------
q1	17602	7352	7203	7203
q2	2025	292	294	292
q3	12208	1025	1120	1025
q4	10561	692	697	692
q5	7751	2873	2831	2831
q6	244	148	143	143
q7	1002	619	602	602
q8	9379	1904	2015	1904
q9	7567	6397	6378	6378
q10	6991	2297	2324	2297
q11	448	240	240	240
q12	405	221	209	209
q13	17770	2994	2972	2972
q14	247	211	222	211
q15	578	543	524	524
q16	683	622	606	606
q17	969	581	540	540
q18	7125	6596	6698	6596
q19	1397	987	954	954
q20	571	288	276	276
q21	4184	3055	3293	3055
q22	1105	975	970	970
Total cold run time: 110812 ms
Total hot run time: 40520 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7193	7187	7228	7187
q2	332	232	229	229
q3	3115	2971	2924	2924
q4	2035	1836	1838	1836
q5	5715	5624	5739	5624
q6	233	145	142	142
q7	2226	1845	1823	1823
q8	3386	3543	3563	3543
q9	8893	9031	8937	8937
q10	3568	3576	3569	3569
q11	580	512	495	495
q12	831	631	638	631
q13	10348	3179	3216	3179
q14	325	281	263	263
q15	592	530	511	511
q16	731	687	691	687
q17	1839	1655	1606	1606
q18	8219	7932	7774	7774
q19	1698	1546	1548	1546
q20	2140	1895	1905	1895
q21	5373	5470	5445	5445
q22	1140	1031	1034	1031
Total cold run time: 70512 ms
Total hot run time: 60877 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 191477 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 4228ed33fb15ddb7f378a8616205a57791d34022, data reload: false

query1	952	383	405	383
query2	6360	2061	1966	1966
query3	8694	198	212	198
query4	33925	23533	23574	23533
query5	3423	470	463	463
query6	272	174	172	172
query7	4198	312	301	301
query8	297	227	235	227
query9	9085	2652	2634	2634
query10	477	287	277	277
query11	17867	15147	15220	15147
query12	155	100	96	96
query13	1544	434	414	414
query14	9774	7245	7532	7245
query15	260	173	183	173
query16	7872	463	476	463
query17	1734	650	598	598
query18	2142	323	323	323
query19	381	165	157	157
query20	122	113	113	113
query21	208	111	116	111
query22	4554	4672	4359	4359
query23	35015	34607	34825	34607
query24	11270	2872	2819	2819
query25	616	420	405	405
query26	1128	163	163	163
query27	2305	302	292	292
query28	7699	2441	2415	2415
query29	817	437	437	437
query30	265	163	161	161
query31	1042	787	805	787
query32	100	56	54	54
query33	749	309	290	290
query34	904	510	497	497
query35	865	747	752	747
query36	1068	948	939	939
query37	147	81	86	81
query38	4081	3916	3943	3916
query39	1505	1430	1431	1430
query40	204	96	98	96
query41	50	47	51	47
query42	117	97	97	97
query43	526	494	473	473
query44	1215	823	813	813
query45	201	169	173	169
query46	1139	765	724	724
query47	1893	1788	1878	1788
query48	473	369	366	366
query49	873	390	383	383
query50	834	388	411	388
query51	7037	7002	6842	6842
query52	101	84	85	84
query53	252	179	175	175
query54	1223	454	462	454
query55	78	80	81	80
query56	280	263	241	241
query57	1235	1095	1093	1093
query58	232	236	224	224
query59	3102	2918	2849	2849
query60	297	267	278	267
query61	106	103	102	102
query62	820	666	679	666
query63	226	184	184	184
query64	3889	667	614	614
query65	3252	3195	3178	3178
query66	766	288	304	288
query67	16032	15478	15388	15388
query68	4811	572	566	566
query69	571	298	293	293
query70	1208	1140	1024	1024
query71	419	274	285	274
query72	7738	3831	4025	3831
query73	769	340	355	340
query74	10598	9094	9038	9038
query75	3750	2659	2685	2659
query76	3626	939	972	939
query77	666	299	297	297
query78	9992	9235	11230	9235
query79	1631	596	620	596
query80	1239	437	446	437
query81	584	242	238	238
query82	314	140	140	140
query83	325	139	144	139
query84	284	81	82	81
query85	1554	304	284	284
query86	451	295	307	295
query87	4570	4334	4360	4334
query88	3164	2391	2357	2357
query89	399	289	282	282
query90	2036	191	191	191
query91	199	163	144	144
query92	66	47	51	47
query93	1044	530	544	530
query94	1010	303	292	292
query95	345	254	255	254
query96	610	280	280	280
query97	3268	3130	3110	3110
query98	222	200	198	198
query99	1547	1286	1298	1286
Total cold run time: 299335 ms
Total hot run time: 191477 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 33.02 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 4228ed33fb15ddb7f378a8616205a57791d34022, data reload: false

query1	0.05	0.05	0.04
query2	0.07	0.03	0.03
query3	0.24	0.07	0.06
query4	1.63	0.10	0.10
query5	0.51	0.51	0.50
query6	1.13	0.73	0.72
query7	0.02	0.01	0.02
query8	0.04	0.03	0.04
query9	0.55	0.51	0.49
query10	0.54	0.57	0.54
query11	0.14	0.10	0.10
query12	0.14	0.12	0.10
query13	0.61	0.61	0.59
query14	2.96	2.91	3.32
query15	0.91	0.83	0.84
query16	0.38	0.37	0.38
query17	1.06	1.02	1.05
query18	0.20	0.20	0.20
query19	1.86	1.78	1.97
query20	0.01	0.02	0.01
query21	15.38	0.56	0.58
query22	2.96	2.39	1.92
query23	17.55	0.96	0.81
query24	2.74	1.24	1.21
query25	0.24	0.21	0.04
query26	0.47	0.13	0.13
query27	0.04	0.04	0.04
query28	10.45	1.08	1.06
query29	12.56	3.27	3.24
query30	0.24	0.05	0.06
query31	2.90	0.37	0.38
query32	3.29	0.46	0.45
query33	3.00	3.03	3.00
query34	16.88	4.44	4.46
query35	4.49	4.50	4.50
query36	0.67	0.48	0.48
query37	0.08	0.06	0.05
query38	0.04	0.03	0.04
query39	0.04	0.02	0.02
query40	0.16	0.13	0.13
query41	0.08	0.02	0.02
query42	0.04	0.02	0.02
query43	0.03	0.02	0.03
Total cold run time: 107.38 s
Total hot run time: 33.02 s

@zy-kkk
Copy link
Member Author

zy-kkk commented Sep 24, 2024

run cloud_p0

@zy-kkk zy-kkk merged commit a722a10 into apache:master Sep 24, 2024
24 of 26 checks passed
@zy-kkk zy-kkk deleted the slots_order branch September 24, 2024 06:43
zy-kkk added a commit to zy-kkk/doris that referenced this pull request Sep 25, 2024
…ing redundant conditions (apache#40801)

This PR simplifies the handling of `ScanNode` projection logic.
Previously, the code included multiple conditional checks to determine
whether a `projectionTuple` should be generated. These conditions have
been removed, and now `projectionTuple `is always generated for
`ScanNode`, ensuring a consistent projection setup. Additionally,
redundant handling of `SlotId` and `SlotRef` has been eliminated, making
the code cleaner and easier to maintain. The behavior for `OlapScanNode`
remains unchanged.
morrySnow pushed a commit that referenced this pull request Sep 26, 2024
…ing redundant conditions (#40801) (#41315)

pick from master #40801

This PR simplifies the handling of `ScanNode` projection logic.
Previously, the code included multiple conditional checks to determine
whether a `projectionTuple` should be generated. These conditions have
been removed, and now `projectionTuple `is always generated for
`ScanNode`, ensuring a consistent projection setup. Additionally,
redundant handling of `SlotId` and `SlotRef` has been eliminated, making
the code cleaner and easier to maintain. The behavior for `OlapScanNode`
remains unchanged.
dataroaring pushed a commit that referenced this pull request Oct 9, 2024
…ing redundant conditions (#40801)

This PR simplifies the handling of `ScanNode` projection logic.
Previously, the code included multiple conditional checks to determine
whether a `projectionTuple` should be generated. These conditions have
been removed, and now `projectionTuple `is always generated for
`ScanNode`, ensuring a consistent projection setup. Additionally,
redundant handling of `SlotId` and `SlotRef` has been eliminated, making
the code cleaner and easier to maintain. The behavior for `OlapScanNode`
remains unchanged.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by one committer. dev/2.1.7-merged dev/3.0.3-merged reviewed
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants