Skip to content

[fix](nereids) setOperator column prunning bug #58237

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
englefly merged 2 commits into from
Nov 24, 2025
Merged

[fix](nereids) setOperator column prunning bug #58237

englefly merged 2 commits into from
Nov 24, 2025

Conversation

@englefly
Copy link
Contributor

@englefly englefly commented Nov 21, 2025

What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@Thearas
Copy link
Contributor

Thearas commented Nov 21, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@englefly
Copy link
Contributor Author

englefly commented Nov 21, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Nov 21, 2025

TPC-H: Total hot run time: 34245 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 5e0e61daf8360b3253e1a93915f05a3defbf7848, data reload: false

------ Round 1 ----------------------------------
q1	17653	5130	4996	4996
q2	2098	314	211	211
q3	10198	1299	738	738
q4	10270	981	394	394
q5	7561	2476	2314	2314
q6	193	171	140	140
q7	905	769	628	628
q8	9330	1322	1057	1057
q9	7049	5404	5395	5395
q10	6880	2218	1805	1805
q11	514	307	282	282
q12	340	368	229	229
q13	17802	3690	3029	3029
q14	230	234	219	219
q15	589	498	493	493
q16	1015	985	936	936
q17	599	873	355	355
q18	7317	7199	6976	6976
q19	1129	944	569	569
q20	384	357	230	230
q21	3933	2595	2304	2304
q22	1052	1029	945	945
Total cold run time: 107041 ms
Total hot run time: 34245 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5102	5071	5000	5000
q2	350	387	321	321
q3	2175	2644	2306	2306
q4	1316	1786	1386	1386
q5	4218	4371	4518	4371
q6	204	166	135	135
q7	2057	1943	1807	1807
q8	2619	2645	2530	2530
q9	7614	7585	7511	7511
q10	3090	3265	2829	2829
q11	578	537	510	510
q12	682	805	631	631
q13	3525	4091	3381	3381
q14	281	309	290	290
q15	539	511	504	504
q16	1066	1120	1053	1053
q17	1214	1529	1419	1419
q18	7870	7966	7504	7504
q19	812	786	846	786
q20	2095	2049	1824	1824
q21	4673	4533	4377	4377
q22	1072	1023	1013	1013
Total cold run time: 53152 ms
Total hot run time: 51488 ms

@doris-robot
Copy link

doris-robot commented Nov 21, 2025

TPC-DS: Total hot run time: 187714 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 5e0e61daf8360b3253e1a93915f05a3defbf7848, data reload: false

query1	1043	413	401	401
query2	6569	1642	1649	1642
query3	6755	223	222	222
query4	26205	23402	23051	23051
query5	4389	625	471	471
query6	332	239	224	224
query7	4656	504	304	304
query8	324	281	253	253
query9	8726	2597	2596	2596
query10	506	352	293	293
query11	15423	15098	14760	14760
query12	177	127	112	112
query13	1697	579	480	480
query14	10487	9498	9384	9384
query15	205	190	185	185
query16	7629	684	539	539
query17	1271	787	667	667
query18	2026	458	339	339
query19	220	211	190	190
query20	136	125	121	121
query21	220	138	118	118
query22	4015	4047	3937	3937
query23	34523	33171	32946	32946
query24	8447	2452	2502	2452
query25	626	547	480	480
query26	1242	280	170	170
query27	2739	504	367	367
query28	4364	2193	2176	2176
query29	828	647	523	523
query30	300	224	203	203
query31	928	829	756	756
query32	90	79	74	74
query33	601	419	350	350
query34	809	926	542	542
query35	836	836	756	756
query36	946	1006	910	910
query37	125	111	88	88
query38	3600	3581	3459	3459
query39	1497	1434	1403	1403
query40	231	132	122	122
query41	65	63	61	61
query42	128	116	108	108
query43	494	499	471	471
query44	1296	767	773	767
query45	188	181	173	173
query46	901	1003	653	653
query47	1739	1762	1739	1739
query48	394	432	323	323
query49	794	494	408	408
query50	702	702	422	422
query51	3842	3864	3881	3864
query52	109	113	107	107
query53	243	272	200	200
query54	312	293	276	276
query55	87	92	89	89
query56	377	356	302	302
query57	1184	1186	1127	1127
query58	290	275	272	272
query59	2579	2620	2512	2512
query60	347	369	355	355
query61	171	162	158	158
query62	791	744	641	641
query63	233	194	201	194
query64	4521	1186	889	889
query65	4032	3962	3934	3934
query66	1157	432	344	344
query67	15178	15063	14831	14831
query68	8409	990	639	639
query69	493	317	281	281
query70	1402	1238	1300	1238
query71	485	336	329	329
query72	5724	4877	4858	4858
query73	672	610	369	369
query74	9118	8851	8667	8667
query75	3901	3314	2829	2829
query76	3716	1183	754	754
query77	831	417	327	327
query78	9600	9686	8927	8927
query79	2024	874	605	605
query80	670	582	515	515
query81	489	263	236	236
query82	426	167	142	142
query83	269	269	268	268
query84	259	112	100	100
query85	1003	490	463	463
query86	333	322	295	295
query87	3664	3719	3637	3637
query88	3644	2241	2251	2241
query89	397	329	298	298
query90	1928	227	226	226
query91	183	168	140	140
query92	79	69	71	69
query93	1483	1030	685	685
query94	687	444	344	344
query95	408	328	312	312
query96	509	567	285	285
query97	2905	2957	2864	2864
query98	236	217	216	216
query99	1302	1415	1269	1269
Total cold run time: 275146 ms
Total hot run time: 187714 ms

@doris-robot
Copy link

doris-robot commented Nov 21, 2025

ClickBench: Total hot run time: 27.45 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 5e0e61daf8360b3253e1a93915f05a3defbf7848, data reload: false

query1	0.06	0.05	0.05
query2	0.09	0.04	0.05
query3	0.26	0.08	0.08
query4	1.63	0.11	0.11
query5	0.27	0.25	0.26
query6	1.19	0.64	0.64
query7	0.03	0.03	0.03
query8	0.06	0.04	0.05
query9	0.59	0.51	0.50
query10	0.57	0.57	0.58
query11	0.16	0.11	0.11
query12	0.15	0.12	0.12
query13	0.62	0.61	0.61
query14	1.01	1.01	0.98
query15	0.85	0.83	0.85
query16	0.39	0.40	0.38
query17	1.06	1.04	1.05
query18	0.21	0.20	0.20
query19	1.90	1.74	1.83
query20	0.01	0.01	0.01
query21	15.43	0.20	0.12
query22	4.98	0.07	0.06
query23	15.63	0.26	0.10
query24	2.91	0.96	0.43
query25	0.07	0.06	0.07
query26	0.16	0.15	0.14
query27	0.06	0.06	0.06
query28	4.09	1.17	0.94
query29	12.54	3.93	3.21
query30	0.28	0.13	0.13
query31	2.81	0.60	0.40
query32	3.22	0.55	0.48
query33	3.00	3.05	3.04
query34	15.91	5.15	4.54
query35	4.57	4.57	4.56
query36	0.68	0.51	0.49
query37	0.10	0.07	0.07
query38	0.07	0.05	0.04
query39	0.04	0.03	0.04
query40	0.18	0.14	0.13
query41	0.09	0.04	0.03
query42	0.04	0.03	0.03
query43	0.04	0.03	0.03
Total cold run time: 98.01 s
Total hot run time: 27.45 s

@hello-stephen
Copy link
Contributor

hello-stephen commented Nov 21, 2025

FE UT Coverage Report

Increment line coverage 100.00% (1/1) 🎉
Increment coverage report
Complete coverage report

@hello-stephen
Copy link
Contributor

hello-stephen commented Nov 21, 2025

FE Regression Coverage Report

Increment line coverage 100.00% (1/1) 🎉
Increment coverage report
Complete coverage report

@englefly
Copy link
Contributor Author

englefly commented Nov 24, 2025

run buildall

@englefly englefly marked this pull request as ready for review November 24, 2025 07:13
@doris-robot
Copy link

doris-robot commented Nov 24, 2025

TPC-H: Total hot run time: 34789 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 6c091c4cc1f11b4f2474568ed09d205651a97e10, data reload: false

------ Round 1 ----------------------------------
q1	17592	5032	4897	4897
q2	1990	310	203	203
q3	10273	1246	706	706
q4	10238	956	373	373
q5	7559	2435	2302	2302
q6	197	170	135	135
q7	940	788	636	636
q8	9347	1302	1104	1104
q9	7113	5369	5349	5349
q10	6882	2180	1790	1790
q11	478	309	296	296
q12	339	364	224	224
q13	17792	3593	2985	2985
q14	242	236	217	217
q15	583	504	500	500
q16	987	998	932	932
q17	577	865	367	367
q18	7592	7414	7710	7414
q19	1285	953	602	602
q20	353	345	235	235
q21	4348	3512	2492	2492
q22	1123	1098	1030	1030
Total cold run time: 107830 ms
Total hot run time: 34789 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5371	5198	5218	5198
q2	327	392	330	330
q3	2263	2862	2482	2482
q4	1433	1844	1453	1453
q5	4608	4431	4375	4375
q6	207	166	127	127
q7	1984	1918	1857	1857
q8	2637	2558	2487	2487
q9	7744	7482	7606	7482
q10	3051	3215	2733	2733
q11	555	531	489	489
q12	617	712	582	582
q13	3222	3598	3039	3039
q14	266	278	276	276
q15	545	497	490	490
q16	1033	1069	999	999
q17	1087	1457	1317	1317
q18	7205	7172	7028	7028
q19	754	769	974	769
q20	1910	1956	1818	1818
q21	4621	4330	4256	4256
q22	1076	1051	970	970
Total cold run time: 52516 ms
Total hot run time: 50557 ms

@doris-robot
Copy link

doris-robot commented Nov 24, 2025

TPC-DS: Total hot run time: 187830 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 6c091c4cc1f11b4f2474568ed09d205651a97e10, data reload: false

query1	1035	408	390	390
query2	6552	1692	1671	1671
query3	6757	240	237	237
query4	25855	23250	23114	23114
query5	4456	655	491	491
query6	350	246	231	231
query7	4647	497	302	302
query8	320	304	256	256
query9	8615	2611	2602	2602
query10	474	343	287	287
query11	15794	15068	14911	14911
query12	183	119	109	109
query13	1678	577	454	454
query14	11261	9285	9197	9197
query15	194	187	168	168
query16	7678	699	495	495
query17	1227	749	639	639
query18	2047	431	340	340
query19	220	202	187	187
query20	126	128	123	123
query21	220	141	117	117
query22	3953	4081	4112	4081
query23	33704	33015	33043	33015
query24	8430	2426	2450	2426
query25	639	574	484	484
query26	1236	282	165	165
query27	2737	507	354	354
query28	4367	2213	2205	2205
query29	815	641	515	515
query30	300	229	202	202
query31	914	814	723	723
query32	87	78	74	74
query33	612	393	354	354
query34	800	853	525	525
query35	820	838	756	756
query36	976	1004	910	910
query37	126	120	99	99
query38	3502	3535	3541	3535
query39	1479	1456	1403	1403
query40	233	139	124	124
query41	72	71	66	66
query42	136	118	122	118
query43	489	513	478	478
query44	1284	780	770	770
query45	187	190	184	184
query46	891	1001	665	665
query47	1715	1770	1727	1727
query48	391	430	330	330
query49	808	535	440	440
query50	669	736	415	415
query51	3843	3900	3857	3857
query52	115	115	109	109
query53	301	270	205	205
query54	299	295	275	275
query55	89	88	85	85
query56	344	323	316	316
query57	1203	1196	1087	1087
query58	283	272	268	268
query59	2529	2748	2579	2579
query60	343	346	335	335
query61	162	158	166	158
query62	818	729	634	634
query63	236	200	195	195
query64	4491	1179	888	888
query65	4001	3945	3910	3910
query66	1157	437	337	337
query67	15272	15307	14912	14912
query68	8462	907	626	626
query69	490	326	285	285
query70	1331	1192	1272	1192
query71	500	325	297	297
query72	5605	4861	4816	4816
query73	705	594	368	368
query74	8814	9136	8768	8768
query75	3992	3340	2755	2755
query76	3793	1160	749	749
query77	833	413	327	327
query78	9480	9726	8891	8891
query79	1954	862	618	618
query80	646	571	500	500
query81	490	266	227	227
query82	452	160	133	133
query83	271	262	249	249
query84	260	110	93	93
query85	931	493	445	445
query86	340	299	314	299
query87	3735	3765	3581	3581
query88	3000	2296	2282	2282
query89	386	324	303	303
query90	1954	232	220	220
query91	167	174	145	145
query92	86	67	68	67
query93	1124	1021	690	690
query94	719	460	346	346
query95	415	333	322	322
query96	495	564	286	286
query97	2947	2906	2865	2865
query98	227	210	209	209
query99	1674	1407	1262	1262
Total cold run time: 273825 ms
Total hot run time: 187830 ms

@doris-robot
Copy link

doris-robot commented Nov 24, 2025

ClickBench: Total hot run time: 27.25 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 6c091c4cc1f11b4f2474568ed09d205651a97e10, data reload: false

query1	0.06	0.05	0.05
query2	0.09	0.05	0.04
query3	0.26	0.08	0.08
query4	1.60	0.11	0.12
query5	0.26	0.25	0.26
query6	1.21	0.64	0.64
query7	0.03	0.02	0.02
query8	0.05	0.04	0.04
query9	0.58	0.52	0.53
query10	0.57	0.58	0.57
query11	0.16	0.12	0.11
query12	0.15	0.12	0.12
query13	0.62	0.61	0.61
query14	1.01	1.00	1.01
query15	0.85	0.84	0.84
query16	0.37	0.39	0.40
query17	1.02	0.98	1.02
query18	0.22	0.20	0.21
query19	1.93	1.80	1.86
query20	0.02	0.01	0.02
query21	15.46	0.21	0.12
query22	5.06	0.07	0.05
query23	15.67	0.25	0.09
query24	2.24	0.66	0.28
query25	0.07	0.06	0.05
query26	0.14	0.13	0.13
query27	0.06	0.06	0.06
query28	3.87	1.13	0.93
query29	12.59	3.83	3.26
query30	0.29	0.14	0.12
query31	2.82	0.58	0.39
query32	3.23	0.56	0.46
query33	3.01	3.00	3.03
query34	15.87	5.20	4.54
query35	4.58	4.55	4.53
query36	0.69	0.50	0.49
query37	0.10	0.07	0.07
query38	0.07	0.04	0.04
query39	0.04	0.04	0.03
query40	0.17	0.15	0.14
query41	0.10	0.03	0.03
query42	0.04	0.02	0.02
query43	0.04	0.04	0.04
Total cold run time: 97.27 s
Total hot run time: 27.25 s

@hello-stephen
Copy link
Contributor

hello-stephen commented Nov 24, 2025

FE UT Coverage Report

Increment line coverage 80.00% (4/5) 🎉
Increment coverage report
Complete coverage report

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Nov 24, 2025
@github-actions
Copy link
Contributor

github-actions bot commented Nov 24, 2025

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Contributor

github-actions bot commented Nov 24, 2025

PR approved by anyone and no changes requested.

@hello-stephen
Copy link
Contributor

hello-stephen commented Nov 24, 2025

FE Regression Coverage Report

Increment line coverage 80.00% (4/5) 🎉
Increment coverage report
Complete coverage report

@englefly englefly merged commit bf08030 into apache:master Nov 24, 2025
29 of 30 checks passed
@englefly englefly deleted the fix-prune-column-setoperator branch November 24, 2025 13:51
github-actions bot pushed a commit that referenced this pull request Nov 24, 2025
@englefly
[fix](nereids) setOperator column prunning bug (#58237)
ab38fc3 
### What problem does this PR solve?
column prunning through set operator may fail and no column are pruned for children of set operator
@github-actions github-actions bot mentioned this pull request Nov 24, 2025
Merged
yiguolei pushed a commit that referenced this pull request Nov 25, 2025
@github-actions @englefly
branch-4.0: [fix](nereids) setOperator column prunning bug #58237 (#5…
ec8678a 
…8318)

Cherry-picked from #58237

Co-authored-by: minghong <zhouminghong@selectdb.com>
@morrySnow morrySnow mentioned this pull request Nov 25, 2025
Merged
morrySnow pushed a commit that referenced this pull request Nov 26, 2025
@englefly
branch-3.1: [fix](nereids) setOperator column prunning bug #58237 (#5…
a466b4c 
…8343)

picked from #58237
nagisa-kunhah pushed a commit to nagisa-kunhah/doris that referenced this pull request Dec 14, 2025
@englefly @nagisa-kunhah
[fix](nereids) setOperator column prunning bug (apache#58237)
9141da1 
### What problem does this PR solve?
column prunning through set operator may fail and no column are pruned for children of set operator
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@924060929 924060929 924060929 approved these changes

@morrySnow morrySnow morrySnow approved these changes

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. dev/3.1.4-merged dev/4.0.2-merged reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

7 participants

@englefly @Thearas @doris-robot @hello-stephen @924060929 @morrySnow @yiguolei