Skip to content

branch-3.0: [opt](nereids) skip run PruneOlapScanTablet when exists lots of InPredicate #49388

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged

Conversation

924060929
Copy link
Contributor

cherry pick some parts from #47608 and #49386

@924060929 924060929 requested a review from dataroaring as a code owner March 24, 2025 05:22
@hello-stephen
Copy link
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@924060929
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 39879 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 575273ac454e5f1a994d6adc193bf3c2e3a44f07, data reload: false

------ Round 1 ----------------------------------
q1	17568	6679	6561	6561
q2	2061	174	171	171
q3	10602	1065	1174	1065
q4	10512	706	680	680
q5	7740	2865	2808	2808
q6	225	135	135	135
q7	961	630	604	604
q8	9353	1954	2003	1954
q9	6562	6374	6421	6374
q10	7040	2278	2267	2267
q11	455	256	264	256
q12	400	220	209	209
q13	17799	2993	3031	2993
q14	232	209	210	209
q15	515	461	455	455
q16	662	583	586	583
q17	972	551	551	551
q18	7165	6686	6672	6672
q19	1384	987	1028	987
q20	462	207	203	203
q21	4454	3267	3128	3128
q22	1110	1035	1014	1014
Total cold run time: 108234 ms
Total hot run time: 39879 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6581	6563	6569	6563
q2	332	241	236	236
q3	2907	2811	2892	2811
q4	2043	1769	1805	1769
q5	5747	5823	5789	5789
q6	208	132	130	130
q7	2219	1870	1856	1856
q8	3429	3580	3547	3547
q9	8774	8970	9013	8970
q10	3627	3592	3549	3549
q11	606	493	500	493
q12	823	608	614	608
q13	9735	3323	3220	3220
q14	303	280	270	270
q15	513	480	460	460
q16	692	662	655	655
q17	1853	1629	1633	1629
q18	8262	7958	8301	7958
q19	2760	1640	1577	1577
q20	2110	1941	1942	1941
q21	5849	5617	5537	5537
q22	1172	1082	1104	1082
Total cold run time: 70545 ms
Total hot run time: 60650 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 197771 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 575273ac454e5f1a994d6adc193bf3c2e3a44f07, data reload: false

query1	1296	934	881	881
query2	6252	2036	1992	1992
query3	10928	4378	4388	4378
query4	62830	31471	23495	23495
query5	4751	457	485	457
query6	409	179	178	178
query7	5486	307	309	307
query8	318	229	222	222
query9	8771	2614	2598	2598
query10	481	263	271	263
query11	16627	15288	15824	15288
query12	156	111	103	103
query13	1444	469	421	421
query14	10243	7207	7217	7207
query15	207	180	172	172
query16	7220	522	497	497
query17	1138	618	608	608
query18	1893	343	316	316
query19	202	164	162	162
query20	116	110	112	110
query21	203	102	107	102
query22	4668	4538	4623	4538
query23	34761	34263	33975	33975
query24	6255	2976	2955	2955
query25	542	435	443	435
query26	666	165	167	165
query27	1833	359	354	354
query28	3783	2485	2443	2443
query29	718	459	454	454
query30	244	172	168	168
query31	1000	834	830	830
query32	69	58	63	58
query33	409	340	320	320
query34	926	509	521	509
query35	861	758	737	737
query36	1102	1053	961	961
query37	118	73	72	72
query38	4202	4034	3999	3999
query39	1526	1487	1488	1487
query40	210	95	97	95
query41	49	47	47	47
query42	118	107	104	104
query43	547	497	494	494
query44	1174	819	814	814
query45	187	170	167	167
query46	1169	737	740	737
query47	2067	1945	1942	1942
query48	483	399	396	396
query49	740	400	390	390
query50	832	447	428	428
query51	7370	7409	7333	7333
query52	108	93	93	93
query53	267	192	194	192
query54	566	460	482	460
query55	82	80	79	79
query56	264	262	255	255
query57	1317	1150	1152	1150
query58	222	207	218	207
query59	3159	2867	2815	2815
query60	273	257	247	247
query61	110	102	107	102
query62	780	664	658	658
query63	217	188	184	184
query64	1369	671	636	636
query65	3254	3215	3182	3182
query66	610	281	323	281
query67	15842	15598	15675	15598
query68	4191	594	591	591
query69	431	268	263	263
query70	1128	1118	1131	1118
query71	332	258	254	254
query72	6630	4014	4030	4014
query73	765	354	354	354
query74	10208	8822	8914	8822
query75	3366	2625	2664	2625
query76	2518	1038	1168	1038
query77	561	272	284	272
query78	10499	9607	9592	9592
query79	1125	600	600	600
query80	839	421	427	421
query81	513	240	238	238
query82	338	91	89	89
query83	162	140	148	140
query84	293	73	78	73
query85	960	297	284	284
query86	442	273	278	273
query87	4417	4324	4329	4324
query88	4653	2385	2361	2361
query89	412	293	295	293
query90	2217	184	183	183
query91	183	147	144	144
query92	63	49	47	47
query93	2143	556	546	546
query94	764	300	294	294
query95	353	255	260	255
query96	617	283	284	283
query97	3310	3207	3183	3183
query98	216	201	193	193
query99	1559	1301	1303	1301
Total cold run time: 314248 ms
Total hot run time: 197771 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.65 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 575273ac454e5f1a994d6adc193bf3c2e3a44f07, data reload: false

query1	0.03	0.03	0.03
query2	0.07	0.03	0.03
query3	0.23	0.07	0.07
query4	1.62	0.10	0.11
query5	0.52	0.50	0.51
query6	1.13	0.72	0.72
query7	0.02	0.01	0.01
query8	0.04	0.03	0.02
query9	0.58	0.50	0.49
query10	0.56	0.53	0.54
query11	0.14	0.11	0.11
query12	0.14	0.11	0.11
query13	0.61	0.60	0.60
query14	2.73	2.75	2.75
query15	0.90	0.84	0.82
query16	0.38	0.37	0.38
query17	1.08	0.99	1.01
query18	0.24	0.23	0.22
query19	1.91	1.96	1.88
query20	0.01	0.01	0.01
query21	15.35	0.58	0.57
query22	2.50	2.84	1.97
query23	17.03	0.92	0.78
query24	3.41	0.28	1.28
query25	0.29	0.20	0.06
query26	0.36	0.15	0.13
query27	0.04	0.04	0.04
query28	10.84	0.49	0.46
query29	12.61	3.34	3.32
query30	0.24	0.06	0.06
query31	2.85	0.39	0.38
query32	3.26	0.48	0.45
query33	2.98	3.05	2.99
query34	16.97	4.47	4.50
query35	4.58	4.55	4.56
query36	0.67	0.48	0.48
query37	0.09	0.06	0.07
query38	0.05	0.04	0.03
query39	0.03	0.02	0.03
query40	0.16	0.13	0.13
query41	0.08	0.02	0.03
query42	0.03	0.02	0.02
query43	0.03	0.03	0.02
Total cold run time: 107.39 s
Total hot run time: 31.65 s

Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@dataroaring dataroaring merged commit 5d002d3 into apache:branch-3.0 Apr 22, 2025
23 of 24 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants