Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[opt](inverted index) reduce generation of the rowid_result if not necessary #35357

Merged
merged 1 commit into from
Jun 20, 2024
Merged

[opt](inverted index) reduce generation of the rowid_result if not necessary #35357

merged 1 commit into from
Jun 20, 2024

Conversation

zzzxl1993
Copy link
Contributor

…rocess

Proposed changes

  1. in the compound process, if all indexes are fully matched, the rowid_result will no longer be generated.

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@zzzxl1993
Copy link
Contributor Author

run buildall

@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TPC-H: Total hot run time: 40111 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 69751446a41c4c0e6ed1aea6e72d2d04e697d4ec, data reload: false

------ Round 1 ----------------------------------
q1	17825	4606	4365	4365
q2	2458	201	212	201
q3	12230	1195	1125	1125
q4	11101	838	878	838
q5	7552	2712	2672	2672
q6	227	139	138	138
q7	959	595	603	595
q8	9220	2063	2048	2048
q9	8911	6518	6493	6493
q10	8952	3712	3727	3712
q11	465	246	238	238
q12	415	221	217	217
q13	17819	3011	3029	3011
q14	251	222	217	217
q15	515	465	468	465
q16	500	386	387	386
q17	949	651	733	651
q18	7942	7485	7462	7462
q19	4787	1571	1531	1531
q20	648	306	315	306
q21	4960	3168	3241	3168
q22	337	272	275	272
Total cold run time: 119023 ms
Total hot run time: 40111 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4403	4262	4200	4200
q2	364	261	271	261
q3	3016	2724	2780	2724
q4	1871	1586	1605	1586
q5	5241	5249	5267	5249
q6	212	123	123	123
q7	2164	1782	1719	1719
q8	3205	3308	3313	3308
q9	8309	8287	8350	8287
q10	3841	3682	3664	3664
q11	572	498	490	490
q12	747	592	599	592
q13	16354	2984	2968	2968
q14	295	263	251	251
q15	535	476	470	470
q16	479	424	417	417
q17	1776	1470	1449	1449
q18	7671	7657	7459	7459
q19	1658	1589	1557	1557
q20	1966	1806	1792	1792
q21	9713	4726	4604	4604
q22	567	492	496	492
Total cold run time: 74959 ms
Total hot run time: 53662 ms

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 35.64% (9017/25297)
Line Coverage: 27.30% (74562/273084)
Region Coverage: 26.53% (38587/145449)
Branch Coverage: 23.39% (19684/84150)
Coverage Report: http://coverage.selectdb-in.cc/coverage/69751446a41c4c0e6ed1aea6e72d2d04e697d4ec_69751446a41c4c0e6ed1aea6e72d2d04e697d4ec/report/index.html

@doris-robot
Copy link

TPC-DS: Total hot run time: 169992 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 69751446a41c4c0e6ed1aea6e72d2d04e697d4ec, data reload: false

query1	916	385	366	366
query2	6452	2368	2379	2368
query3	6660	211	204	204
query4	19920	17397	17465	17397
query5	4133	432	427	427
query6	245	160	152	152
query7	4597	286	290	286
query8	234	186	181	181
query9	8631	2398	2383	2383
query10	447	276	263	263
query11	10627	10196	10045	10045
query12	135	94	90	90
query13	1638	365	366	365
query14	9287	7691	7166	7166
query15	225	177	175	175
query16	7903	260	256	256
query17	1850	533	552	533
query18	1963	271	266	266
query19	209	159	155	155
query20	92	85	87	85
query21	189	127	130	127
query22	4380	4095	3936	3936
query23	33807	33263	32833	32833
query24	12093	2878	2859	2859
query25	655	352	375	352
query26	1813	159	153	153
query27	2950	326	333	326
query28	7505	2045	2065	2045
query29	1088	609	609	609
query30	312	170	173	170
query31	959	737	779	737
query32	89	52	54	52
query33	756	268	259	259
query34	1009	473	481	473
query35	743	634	614	614
query36	1077	922	910	910
query37	286	68	71	68
query38	2940	2764	2768	2764
query39	861	796	804	796
query40	283	128	130	128
query41	46	43	44	43
query42	104	99	94	94
query43	571	558	537	537
query44	1211	729	742	729
query45	188	168	165	165
query46	1057	737	730	730
query47	1857	1788	1821	1788
query48	367	300	304	300
query49	1194	379	381	379
query50	781	407	398	398
query51	6780	6720	6849	6720
query52	108	87	94	87
query53	355	287	281	281
query54	1007	423	425	423
query55	76	69	70	69
query56	260	240	268	240
query57	1147	1066	1041	1041
query58	235	211	201	201
query59	3392	3231	3239	3231
query60	272	254	253	253
query61	88	93	90	90
query62	620	452	446	446
query63	304	283	279	279
query64	9839	2202	1739	1739
query65	3193	3634	3103	3103
query66	1366	329	331	329
query67	15420	14850	14723	14723
query68	4616	526	554	526
query69	486	268	268	268
query70	1172	1112	1170	1112
query71	433	265	270	265
query72	7549	6273	2714	2714
query73	711	320	320	320
query74	6001	5751	5639	5639
query75	3469	2650	2659	2650
query76	2827	1085	984	984
query77	457	268	262	262
query78	10290	9940	9810	9810
query79	2078	522	505	505
query80	867	439	430	430
query81	525	242	245	242
query82	797	92	93	92
query83	257	170	169	169
query84	230	89	82	82
query85	1594	313	353	313
query86	470	324	301	301
query87	3286	3138	3116	3116
query88	4095	2343	2340	2340
query89	477	394	382	382
query90	1996	183	187	183
query91	128	94	97	94
query92	58	48	46	46
query93	2263	515	506	506
query94	1236	185	182	182
query95	415	309	306	306
query96	589	267	269	267
query97	3185	3096	3039	3039
query98	237	220	211	211
query99	1147	859	844	844
Total cold run time: 275738 ms
Total hot run time: 169992 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.61 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 69751446a41c4c0e6ed1aea6e72d2d04e697d4ec, data reload: false

query1	0.05	0.04	0.04
query2	0.08	0.04	0.04
query3	0.23	0.06	0.05
query4	1.69	0.09	0.10
query5	0.50	0.49	0.50
query6	1.12	0.72	0.72
query7	0.02	0.02	0.02
query8	0.05	0.04	0.05
query9	0.53	0.49	0.49
query10	0.55	0.55	0.55
query11	0.15	0.12	0.11
query12	0.14	0.12	0.11
query13	0.60	0.59	0.58
query14	0.77	0.80	0.78
query15	0.83	0.80	0.81
query16	0.37	0.35	0.37
query17	0.99	1.03	0.96
query18	0.21	0.24	0.24
query19	1.74	1.68	1.73
query20	0.01	0.01	0.01
query21	15.45	0.69	0.69
query22	4.64	6.67	2.01
query23	18.29	1.31	1.19
query24	1.33	0.35	0.24
query25	0.14	0.09	0.08
query26	0.26	0.17	0.17
query27	0.08	0.07	0.07
query28	13.46	1.01	1.00
query29	12.69	3.34	3.31
query30	0.24	0.07	0.06
query31	2.86	0.38	0.37
query32	3.31	0.46	0.47
query33	2.80	2.90	2.93
query34	17.12	4.41	4.37
query35	4.50	4.51	4.48
query36	0.64	0.46	0.46
query37	0.17	0.15	0.16
query38	0.14	0.15	0.14
query39	0.04	0.04	0.04
query40	0.16	0.14	0.14
query41	0.09	0.04	0.05
query42	0.05	0.05	0.05
query43	0.04	0.04	0.04
Total cold run time: 109.13 s
Total hot run time: 30.61 s

airborne12
airborne12 previously approved these changes May 25, 2024
View reviewed changes
Copy link
Member

@airborne12 airborne12 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions GitHub Actions
Copy link
Contributor

PR approved by anyone and no changes requested.

@xiaokang xiaokang changed the title [opt](inverted index) simplification of the rowid_result generation p… [opt](inverted index) simplification of the rowid_result generation May 26, 2024
@xiaokang xiaokang added usercase Important user case type label dev/2.0.x labels May 26, 2024
@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@zzzxl1993
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 40752 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 97bfb237f28e5398f0d30100068cf3121564f4de, data reload: false

------ Round 1 ----------------------------------
q1	17622	4381	4431	4381
q2	2024	192	193	192
q3	10489	1309	1220	1220
q4	10514	906	809	809
q5	7817	2677	2800	2677
q6	222	131	133	131
q7	963	605	612	605
q8	9441	2079	2123	2079
q9	9247	6617	6632	6617
q10	9148	3702	3667	3667
q11	458	247	254	247
q12	415	219	212	212
q13	17772	2954	2954	2954
q14	259	217	221	217
q15	512	486	467	467
q16	520	371	382	371
q17	965	765	675	675
q18	8073	7362	7392	7362
q19	4741	1516	1494	1494
q20	649	299	302	299
q21	4886	3796	3821	3796
q22	335	282	280	280
Total cold run time: 117072 ms
Total hot run time: 40752 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4308	4207	4217	4207
q2	358	264	261	261
q3	2931	2754	2717	2717
q4	1813	1569	1577	1569
q5	5237	5258	5287	5258
q6	219	123	125	123
q7	2167	1699	1682	1682
q8	3146	3291	3279	3279
q9	8317	8249	8322	8249
q10	3884	3624	3651	3624
q11	584	499	497	497
q12	737	578	607	578
q13	17240	2960	2993	2960
q14	316	260	257	257
q15	515	479	475	475
q16	475	414	421	414
q17	1767	1505	1449	1449
q18	7638	7458	7281	7281
q19	1648	1511	1545	1511
q20	2018	1771	1791	1771
q21	4809	4795	4712	4712
q22	560	503	488	488
Total cold run time: 70687 ms
Total hot run time: 53362 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 170550 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 97bfb237f28e5398f0d30100068cf3121564f4de, data reload: false

query1	908	381	365	365
query2	7416	2372	2254	2254
query3	6651	220	219	219
query4	19144	17391	17168	17168
query5	4230	417	432	417
query6	246	156	154	154
query7	4585	309	306	306
query8	262	180	186	180
query9	8568	2411	2401	2401
query10	463	283	265	265
query11	10604	10016	9990	9990
query12	139	94	91	91
query13	1645	365	364	364
query14	10026	6039	6577	6039
query15	213	168	164	164
query16	7811	263	262	262
query17	1366	535	501	501
query18	1953	268	263	263
query19	190	149	148	148
query20	94	82	87	82
query21	200	131	127	127
query22	4325	3995	3932	3932
query23	33512	33082	33017	33017
query24	12440	2795	2796	2795
query25	655	353	358	353
query26	1792	151	153	151
query27	3033	319	326	319
query28	7818	2091	2050	2050
query29	1133	605	593	593
query30	289	146	153	146
query31	979	735	745	735
query32	92	50	53	50
query33	767	263	260	260
query34	1051	473	477	473
query35	751	612	590	590
query36	1064	901	918	901
query37	279	63	66	63
query38	2927	2774	2788	2774
query39	859	785	768	768
query40	273	122	123	122
query41	47	43	50	43
query42	101	97	95	95
query43	602	552	538	538
query44	1227	742	754	742
query45	185	160	192	160
query46	1069	747	734	734
query47	1858	1749	1801	1749
query48	374	290	291	290
query49	1190	374	381	374
query50	772	394	380	380
query51	6889	6750	6566	6566
query52	106	91	88	88
query53	358	287	285	285
query54	954	429	410	410
query55	76	76	72	72
query56	255	235	241	235
query57	1130	1010	1034	1010
query58	229	210	203	203
query59	3300	2976	3163	2976
query60	271	255	251	251
query61	86	87	84	84
query62	652	433	468	433
query63	319	286	284	284
query64	9816	2211	1791	1791
query65	3185	3116	3121	3116
query66	1373	333	327	327
query67	15193	14728	15085	14728
query68	4497	533	532	532
query69	435	271	269	269
query70	1162	1139	1086	1086
query71	416	273	256	256
query72	7411	5411	5740	5411
query73	718	327	315	315
query74	5931	5640	5525	5525
query75	3510	2643	2640	2640
query76	2939	943	1033	943
query77	436	267	264	264
query78	10156	9999	9734	9734
query79	2024	529	519	519
query80	1193	462	450	450
query81	517	225	222	222
query82	621	88	96	88
query83	251	178	183	178
query84	243	93	85	85
query85	1865	330	311	311
query86	478	310	304	304
query87	3334	3126	3173	3126
query88	4172	2358	2352	2352
query89	487	388	390	388
query90	2005	208	191	191
query91	135	109	105	105
query92	56	50	51	50
query93	2218	517	505	505
query94	1285	193	196	193
query95	402	314	313	313
query96	591	266	264	264
query97	3187	3048	3139	3048
query98	237	223	219	219
query99	1179	848	875	848
Total cold run time: 276461 ms
Total hot run time: 170550 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.39 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 97bfb237f28e5398f0d30100068cf3121564f4de, data reload: false

query1	0.03	0.03	0.04
query2	0.09	0.04	0.05
query3	0.23	0.06	0.06
query4	1.66	0.08	0.10
query5	0.50	0.51	0.50
query6	1.13	0.72	0.71
query7	0.01	0.01	0.02
query8	0.05	0.05	0.04
query9	0.54	0.49	0.48
query10	0.53	0.54	0.52
query11	0.16	0.11	0.11
query12	0.14	0.11	0.11
query13	0.59	0.59	0.60
query14	0.75	0.79	0.76
query15	0.82	0.81	0.80
query16	0.37	0.37	0.37
query17	1.02	1.01	1.02
query18	0.24	0.23	0.27
query19	1.91	1.68	1.79
query20	0.01	0.01	0.01
query21	15.44	0.68	0.67
query22	4.60	7.15	1.87
query23	18.29	1.24	1.32
query24	1.93	0.23	0.21
query25	0.13	0.10	0.08
query26	0.27	0.16	0.16
query27	0.08	0.08	0.08
query28	13.29	1.02	0.99
query29	12.67	3.30	3.31
query30	0.25	0.06	0.05
query31	2.88	0.39	0.38
query32	3.27	0.47	0.46
query33	2.89	2.86	2.93
query34	17.14	4.38	4.46
query35	4.49	4.55	4.47
query36	0.66	0.46	0.46
query37	0.17	0.15	0.16
query38	0.15	0.14	0.14
query39	0.04	0.04	0.04
query40	0.15	0.14	0.15
query41	0.09	0.04	0.04
query42	0.06	0.05	0.05
query43	0.04	0.04	0.03
Total cold run time: 109.76 s
Total hot run time: 30.39 s

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 35.76% (9009/25191)
Line Coverage: 27.37% (74565/272460)
Region Coverage: 26.59% (38579/145100)
Branch Coverage: 23.45% (19672/83884)
Coverage Report: http://coverage.selectdb-in.cc/coverage/97bfb237f28e5398f0d30100068cf3121564f4de_97bfb237f28e5398f0d30100068cf3121564f4de/report/index.html

@zzzxl1993
Copy link
Contributor Author

run buildall

@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 35.77% (9004/25173)
Line Coverage: 27.40% (74563/272173)
Region Coverage: 26.62% (38575/144930)
Branch Coverage: 23.49% (19678/83786)
Coverage Report: http://coverage.selectdb-in.cc/coverage/fd61f63de7a8b2cb7fb543c24faf37ce4d8d619a_fd61f63de7a8b2cb7fb543c24faf37ce4d8d619a/report/index.html

@doris-robot
Copy link

TPC-H: Total hot run time: 40146 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit fd61f63de7a8b2cb7fb543c24faf37ce4d8d619a, data reload: false

------ Round 1 ----------------------------------
q1	17694	4277	4258	4258
q2	2021	200	193	193
q3	10591	1348	1237	1237
q4	10260	743	851	743
q5	7549	2657	2712	2657
q6	220	134	137	134
q7	952	612	606	606
q8	10162	2081	2098	2081
q9	9271	6614	6560	6560
q10	8998	3772	3692	3692
q11	449	248	241	241
q12	466	221	223	221
q13	18744	2990	2988	2988
q14	261	225	224	224
q15	515	478	480	478
q16	479	389	388	388
q17	949	690	688	688
q18	7993	7580	7321	7321
q19	1626	1551	1531	1531
q20	657	301	303	301
q21	4987	3331	3935	3331
q22	364	273	284	273
Total cold run time: 115208 ms
Total hot run time: 40146 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4259	4220	4232	4220
q2	379	280	266	266
q3	2982	2733	2729	2729
q4	1832	1592	1605	1592
q5	5269	5310	5262	5262
q6	210	125	126	125
q7	2076	1707	1774	1707
q8	3160	3293	3290	3290
q9	8308	8294	8358	8294
q10	3939	3694	3695	3694
q11	588	484	480	480
q12	775	601	577	577
q13	16484	3006	3023	3006
q14	309	276	258	258
q15	513	488	476	476
q16	482	407	433	407
q17	1780	1478	1470	1470
q18	7736	7486	7396	7396
q19	4684	1598	1512	1512
q20	1973	1792	1773	1773
q21	4859	4646	4641	4641
q22	579	492	494	492
Total cold run time: 73176 ms
Total hot run time: 53667 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 169801 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit fd61f63de7a8b2cb7fb543c24faf37ce4d8d619a, data reload: false

query1	928	386	387	386
query2	6458	2374	2325	2325
query3	6657	205	201	201
query4	19984	17339	17331	17331
query5	4142	422	416	416
query6	255	159	146	146
query7	4608	298	290	290
query8	235	180	188	180
query9	8645	2457	2440	2440
query10	446	283	279	279
query11	10590	10165	9962	9962
query12	136	87	86	86
query13	1649	380	360	360
query14	9597	6839	7654	6839
query15	212	173	166	166
query16	7852	272	260	260
query17	1838	525	514	514
query18	1954	272	272	272
query19	198	152	157	152
query20	93	88	88	88
query21	197	131	127	127
query22	4210	4087	3880	3880
query23	33716	33001	32976	32976
query24	11875	2882	2823	2823
query25	660	349	381	349
query26	1774	158	155	155
query27	3027	322	327	322
query28	7347	2083	2113	2083
query29	1079	619	597	597
query30	273	152	150	150
query31	972	756	755	755
query32	90	52	54	52
query33	756	285	264	264
query34	1003	465	476	465
query35	759	616	623	616
query36	1108	904	888	888
query37	159	67	68	67
query38	2933	2757	2762	2757
query39	836	795	784	784
query40	272	127	125	125
query41	48	43	43	43
query42	103	95	96	95
query43	560	527	521	521
query44	1259	737	755	737
query45	180	160	162	160
query46	1075	706	753	706
query47	1857	1765	1775	1765
query48	381	312	298	298
query49	1189	398	418	398
query50	765	381	387	381
query51	6864	6831	6864	6831
query52	103	86	91	86
query53	351	288	284	284
query54	1103	448	427	427
query55	73	72	73	72
query56	261	253	275	253
query57	1161	1076	1080	1076
query58	242	212	217	212
query59	3335	3205	3372	3205
query60	273	259	249	249
query61	91	87	91	87
query62	650	450	474	450
query63	310	290	280	280
query64	9795	2296	1755	1755
query65	3190	3123	3132	3123
query66	1379	333	326	326
query67	15310	15093	14984	14984
query68	4514	555	552	552
query69	446	271	273	271
query70	1129	1141	1162	1141
query71	391	278	270	270
query72	7565	5346	2732	2732
query73	725	328	325	325
query74	6161	5645	5601	5601
query75	3346	2620	2689	2620
query76	2326	932	935	932
query77	435	260	266	260
query78	10260	9774	9763	9763
query79	2039	506	505	505
query80	934	430	436	430
query81	515	217	223	217
query82	648	94	89	89
query83	241	185	166	166
query84	240	89	84	84
query85	1873	286	255	255
query86	493	317	313	313
query87	3258	3052	3120	3052
query88	4136	2422	2418	2418
query89	477	388	371	371
query90	1954	194	192	192
query91	126	96	100	96
query92	57	50	47	47
query93	2327	529	517	517
query94	1262	195	190	190
query95	413	330	375	330
query96	592	279	263	263
query97	3195	3005	3081	3005
query98	240	228	218	218
query99	1090	844	883	844
Total cold run time: 274839 ms
Total hot run time: 169801 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.58 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit fd61f63de7a8b2cb7fb543c24faf37ce4d8d619a, data reload: false

query1	0.04	0.03	0.03
query2	0.08	0.04	0.04
query3	0.23	0.05	0.05
query4	1.67	0.07	0.07
query5	0.49	0.49	0.50
query6	1.14	0.73	0.73
query7	0.02	0.01	0.01
query8	0.05	0.04	0.04
query9	0.52	0.50	0.49
query10	0.55	0.55	0.55
query11	0.15	0.12	0.11
query12	0.14	0.12	0.12
query13	0.60	0.58	0.60
query14	0.78	0.78	0.81
query15	0.82	0.81	0.82
query16	0.37	0.35	0.37
query17	1.02	0.96	1.00
query18	0.20	0.25	0.24
query19	1.87	1.75	1.72
query20	0.02	0.01	0.01
query21	15.65	0.68	0.66
query22	3.70	8.13	2.04
query23	18.36	1.50	1.32
query24	2.02	0.24	0.21
query25	0.16	0.08	0.07
query26	0.27	0.15	0.17
query27	0.07	0.08	0.07
query28	13.20	1.02	1.00
query29	13.18	3.27	3.32
query30	0.24	0.08	0.07
query31	2.84	0.38	0.37
query32	3.28	0.47	0.47
query33	2.91	2.85	2.88
query34	17.23	4.38	4.45
query35	4.45	4.43	4.48
query36	0.64	0.46	0.45
query37	0.18	0.15	0.15
query38	0.16	0.15	0.15
query39	0.05	0.03	0.04
query40	0.16	0.14	0.14
query41	0.09	0.05	0.05
query42	0.05	0.04	0.05
query43	0.04	0.04	0.03
Total cold run time: 109.69 s
Total hot run time: 30.58 s

@xiaokang xiaokang changed the title [opt](inverted index) simplification of the rowid_result generation [opt](inverted index) reduce generation of the rowid_result if not necessary May 29, 2024
xiaokang
xiaokang reviewed May 29, 2024
View reviewed changes
be/src/olap/rowset/segment_v2/segment_iterator.cpp Outdated
@@ -515,6 +515,9 @@ Status SegmentIterator::_get_row_ranges_by_column_conditions() {
}
}
_col_preds_except_leafnode_of_andnode.clear();
for (auto& iter : _rowid_result_for_index) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

add comment to explain its effect and related code

xiaokang
xiaokang previously approved these changes May 30, 2024
View reviewed changes
Copy link
Contributor

@xiaokang xiaokang left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@doris-robot
Copy link

TPC-H: Total hot run time: 40117 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 93ec42b014fd13c879e3b919df73e0b737e70f14, data reload: false

------ Round 1 ----------------------------------
q1	17620	4255	4200	4200
q2	2021	206	200	200
q3	10429	1091	1112	1091
q4	10205	825	765	765
q5	7467	2679	2620	2620
q6	228	135	138	135
q7	940	606	608	606
q8	9226	2062	2041	2041
q9	8732	6461	6486	6461
q10	8982	3712	3722	3712
q11	460	245	241	241
q12	469	232	227	227
q13	17754	2970	2978	2970
q14	267	216	216	216
q15	519	474	456	456
q16	519	370	366	366
q17	958	675	647	647
q18	7886	7350	7368	7350
q19	2902	1556	1354	1354
q20	642	312	319	312
q21	4870	3817	3880	3817
q22	389	345	330	330
Total cold run time: 113485 ms
Total hot run time: 40117 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4413	4234	4196	4196
q2	368	270	268	268
q3	2978	2704	2760	2704
q4	1982	1685	1722	1685
q5	5520	5636	5485	5485
q6	221	126	133	126
q7	2200	1833	1788	1788
q8	3315	3460	3434	3434
q9	8669	8644	8724	8644
q10	4119	3862	3914	3862
q11	587	471	482	471
q12	781	603	619	603
q13	17121	3179	3227	3179
q14	308	275	286	275
q15	520	497	485	485
q16	486	447	423	423
q17	1770	1540	1512	1512
q18	8042	7878	7722	7722
q19	3956	1655	1638	1638
q20	2154	1872	1959	1872
q21	5101	4944	4963	4944
q22	610	581	541	541
Total cold run time: 75221 ms
Total hot run time: 55857 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 173278 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 93ec42b014fd13c879e3b919df73e0b737e70f14, data reload: false

query1	911	380	362	362
query2	6451	2442	2393	2393
query3	6631	214	215	214
query4	19045	17340	17184	17184
query5	3614	463	474	463
query6	255	163	155	155
query7	4585	292	293	292
query8	310	285	274	274
query9	8681	2389	2366	2366
query10	559	307	276	276
query11	10599	10173	9877	9877
query12	118	87	86	86
query13	1640	362	361	361
query14	9483	7730	6505	6505
query15	225	191	186	186
query16	7833	268	260	260
query17	1924	552	515	515
query18	1913	266	268	266
query19	198	150	159	150
query20	93	81	84	81
query21	208	134	121	121
query22	4195	4033	4307	4033
query23	33898	33647	33611	33611
query24	11184	2859	2818	2818
query25	591	380	370	370
query26	725	151	155	151
query27	2346	319	328	319
query28	6473	2124	2108	2108
query29	858	630	620	620
query30	262	155	159	155
query31	965	783	775	775
query32	90	51	60	51
query33	776	305	295	295
query34	1056	494	514	494
query35	766	622	629	622
query36	1091	962	988	962
query37	157	74	75	74
query38	2969	2837	2780	2780
query39	950	842	819	819
query40	210	130	131	130
query41	55	57	53	53
query42	116	100	102	100
query43	618	574	534	534
query44	1190	725	717	717
query45	194	167	160	160
query46	1074	711	714	711
query47	1910	1760	1768	1760
query48	369	298	295	295
query49	833	403	428	403
query50	762	394	391	391
query51	6808	6690	6731	6690
query52	98	93	88	88
query53	355	290	293	290
query54	895	441	447	441
query55	74	71	75	71
query56	276	251	255	251
query57	1131	1067	1070	1067
query58	247	250	247	247
query59	3486	3351	3161	3161
query60	284	266	266	266
query61	106	93	91	91
query62	611	433	442	433
query63	316	291	292	291
query64	8655	2352	1828	1828
query65	3140	3086	3117	3086
query66	761	334	341	334
query67	15564	14822	14988	14822
query68	5626	549	549	549
query69	605	476	414	414
query70	1121	1110	1072	1072
query71	462	273	274	273
query72	7651	5939	5871	5871
query73	788	331	328	328
query74	5935	5538	5419	5419
query75	3684	2761	2704	2704
query76	3508	982	953	953
query77	631	303	308	303
query78	10442	9849	9680	9680
query79	7186	564	510	510
query80	1566	467	455	455
query81	586	223	230	223
query82	1490	105	104	104
query83	223	169	166	166
query84	272	82	87	82
query85	1413	282	263	263
query86	475	328	310	310
query87	3240	3050	3086	3050
query88	4333	2365	2353	2353
query89	487	377	372	372
query90	1769	194	189	189
query91	128	103	98	98
query92	63	47	50	47
query93	5735	520	504	504
query94	1106	186	189	186
query95	411	317	325	317
query96	611	266	266	266
query97	3189	3018	3071	3018
query98	217	195	189	189
query99	1243	855	861	855
Total cold run time: 280962 ms
Total hot run time: 173278 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.54 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 93ec42b014fd13c879e3b919df73e0b737e70f14, data reload: false

query1	0.04	0.04	0.04
query2	0.08	0.05	0.04
query3	0.23	0.06	0.05
query4	1.66	0.10	0.07
query5	0.50	0.49	0.48
query6	1.13	0.73	0.73
query7	0.02	0.02	0.01
query8	0.05	0.04	0.05
query9	0.55	0.48	0.50
query10	0.55	0.56	0.54
query11	0.15	0.12	0.11
query12	0.15	0.12	0.12
query13	0.60	0.59	0.61
query14	0.77	0.77	0.81
query15	0.82	0.81	0.82
query16	0.36	0.36	0.37
query17	1.06	1.03	0.99
query18	0.24	0.23	0.26
query19	1.81	1.82	1.67
query20	0.01	0.01	0.02
query21	15.42	0.65	0.65
query22	4.27	7.04	1.89
query23	18.26	1.42	1.24
query24	2.09	0.25	0.22
query25	0.16	0.08	0.08
query26	0.28	0.18	0.18
query27	0.09	0.08	0.08
query28	13.23	1.02	1.00
query29	12.65	3.32	3.26
query30	0.26	0.06	0.06
query31	2.86	0.38	0.39
query32	3.26	0.50	0.48
query33	2.89	2.96	2.88
query34	17.03	4.46	4.50
query35	4.54	4.50	4.55
query36	0.65	0.47	0.46
query37	0.18	0.15	0.15
query38	0.15	0.14	0.15
query39	0.05	0.03	0.03
query40	0.18	0.14	0.15
query41	0.10	0.05	0.04
query42	0.06	0.04	0.05
query43	0.05	0.04	0.04
Total cold run time: 109.49 s
Total hot run time: 30.54 s

@zzzxl1993
Copy link
Contributor Author

run buildall

@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 36.46% (9003/24696)
Line Coverage: 28.00% (73772/263458)
Region Coverage: 27.48% (38323/139439)
Branch Coverage: 24.17% (19528/80792)
Coverage Report: http://coverage.selectdb-in.cc/coverage/05df443e5f0a26652a0538ba01f10a2fb265deea_05df443e5f0a26652a0538ba01f10a2fb265deea/report/index.html

@doris-robot
Copy link

TPC-H: Total hot run time: 40852 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 05df443e5f0a26652a0538ba01f10a2fb265deea, data reload: false

------ Round 1 ----------------------------------
q1	18105	4735	4430	4430
q2	3000	196	203	196
q3	11872	1115	1144	1115
q4	10785	881	841	841
q5	7851	2761	2714	2714
q6	231	141	142	141
q7	979	608	591	591
q8	9229	2061	2116	2061
q9	8831	6532	6470	6470
q10	8854	3709	3754	3709
q11	455	241	238	238
q12	410	239	227	227
q13	17770	3041	2981	2981
q14	271	222	235	222
q15	514	487	485	485
q16	531	396	378	378
q17	954	690	741	690
q18	7936	7419	7421	7419
q19	6499	1505	1521	1505
q20	651	309	323	309
q21	4909	3789	3945	3789
q22	405	342	341	341
Total cold run time: 121042 ms
Total hot run time: 40852 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4432	4302	4280	4280
q2	377	280	290	280
q3	2982	2695	2777	2695
q4	1839	1646	1593	1593
q5	5242	5303	5286	5286
q6	219	128	132	128
q7	2119	1760	1740	1740
q8	3202	3362	3326	3326
q9	8345	8287	8338	8287
q10	3885	3704	3661	3661
q11	587	489	476	476
q12	766	603	594	594
q13	17375	3003	3023	3003
q14	296	280	269	269
q15	518	482	471	471
q16	499	436	428	428
q17	1800	1518	1490	1490
q18	7662	7548	7444	7444
q19	1734	1527	1497	1497
q20	1982	1822	1797	1797
q21	4891	4632	4790	4632
q22	619	541	547	541
Total cold run time: 71371 ms
Total hot run time: 53918 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 171886 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 05df443e5f0a26652a0538ba01f10a2fb265deea, data reload: false

query1	913	375	364	364
query2	6459	2608	2380	2380
query3	6646	208	211	208
query4	18671	17259	17280	17259
query5	4190	468	467	467
query6	251	164	159	159
query7	4590	300	293	293
query8	323	291	280	280
query9	8554	2393	2371	2371
query10	604	314	281	281
query11	10441	9943	9835	9835
query12	141	90	84	84
query13	1631	394	357	357
query14	9447	7111	6649	6649
query15	233	191	190	190
query16	7629	266	254	254
query17	1484	534	520	520
query18	1702	265	266	265
query19	188	158	154	154
query20	91	85	81	81
query21	205	132	124	124
query22	4254	4153	3987	3987
query23	33555	33005	33390	33005
query24	12435	2898	2810	2810
query25	668	341	351	341
query26	1776	155	154	154
query27	3033	313	308	308
query28	7552	2031	2017	2017
query29	1109	632	602	602
query30	279	151	146	146
query31	959	707	762	707
query32	95	54	55	54
query33	761	280	285	280
query34	1016	478	457	457
query35	750	616	631	616
query36	1078	932	909	909
query37	307	105	67	67
query38	2865	2738	2719	2719
query39	857	802	793	793
query40	288	123	126	123
query41	53	52	52	52
query42	117	96	100	96
query43	565	530	549	530
query44	1242	718	714	714
query45	197	167	167	167
query46	1101	711	715	711
query47	1851	1781	1769	1769
query48	367	290	292	290
query49	1202	404	393	393
query50	760	394	384	384
query51	6803	6703	6640	6640
query52	117	91	94	91
query53	358	283	287	283
query54	959	439	437	437
query55	73	73	74	73
query56	270	255	327	255
query57	1129	1042	1075	1042
query58	262	238	264	238
query59	3475	3264	3197	3197
query60	293	295	261	261
query61	92	92	90	90
query62	674	440	453	440
query63	334	296	284	284
query64	9908	2269	1731	1731
query65	3223	3123	3179	3123
query66	1334	332	335	332
query67	15449	14991	14816	14816
query68	4525	538	549	538
query69	521	403	377	377
query70	1062	1173	1137	1137
query71	400	273	274	273
query72	7285	5900	5589	5589
query73	760	321	319	319
query74	5853	5567	5534	5534
query75	3512	2648	2657	2648
query76	2853	986	928	928
query77	544	293	286	286
query78	10273	9777	9663	9663
query79	2449	511	505	505
query80	893	468	455	455
query81	571	220	216	216
query82	804	103	102	102
query83	262	166	166	166
query84	239	87	83	83
query85	1197	275	261	261
query86	432	328	301	301
query87	3300	3111	3073	3073
query88	4148	2324	2303	2303
query89	467	376	382	376
query90	1815	195	189	189
query91	128	99	100	99
query92	70	48	48	48
query93	1903	520	500	500
query94	1265	182	179	179
query95	397	316	371	316
query96	582	263	261	261
query97	3241	3007	3084	3007
query98	229	209	202	202
query99	1285	843	843	843
Total cold run time: 273503 ms
Total hot run time: 171886 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.95 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 05df443e5f0a26652a0538ba01f10a2fb265deea, data reload: false

query1	0.04	0.03	0.03
query2	0.09	0.04	0.04
query3	0.22	0.06	0.05
query4	1.66	0.07	0.07
query5	0.50	0.49	0.48
query6	1.14	0.73	0.72
query7	0.02	0.01	0.01
query8	0.06	0.05	0.04
query9	0.56	0.49	0.50
query10	0.54	0.54	0.53
query11	0.15	0.11	0.11
query12	0.14	0.12	0.11
query13	0.59	0.59	0.60
query14	0.79	0.77	0.78
query15	0.85	0.81	0.81
query16	0.36	0.34	0.35
query17	0.96	1.00	0.98
query18	0.24	0.25	0.24
query19	1.74	1.76	1.72
query20	0.01	0.01	0.01
query21	15.41	0.65	0.66
query22	4.22	6.07	2.22
query23	18.29	1.54	1.27
query24	2.16	0.22	0.21
query25	0.14	0.09	0.08
query26	0.28	0.17	0.17
query27	0.08	0.09	0.08
query28	13.20	1.01	1.00
query29	12.64	3.31	3.32
query30	0.26	0.06	0.06
query31	2.87	0.39	0.39
query32	3.26	0.48	0.47
query33	2.85	2.93	2.93
query34	17.26	4.47	4.43
query35	4.51	4.49	4.56
query36	0.66	0.49	0.46
query37	0.19	0.16	0.15
query38	0.14	0.14	0.15
query39	0.05	0.04	0.03
query40	0.17	0.14	0.14
query41	0.09	0.05	0.05
query42	0.06	0.04	0.04
query43	0.04	0.04	0.04
Total cold run time: 109.49 s
Total hot run time: 30.95 s

@zzzxl1993 zzzxl1993 mentioned this pull request Jun 19, 2024
Merged
@zzzxl1993
Copy link
Contributor Author

run p0

@zzzxl1993 zzzxl1993 mentioned this pull request Jun 20, 2024
Merged
@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jun 20, 2024
@github-actions GitHub Actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

xiaokang
xiaokang approved these changes Jun 20, 2024
View reviewed changes
Copy link
Contributor

@xiaokang xiaokang left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@xiaokang xiaokang merged commit 2c795a9 into apache:master Jun 20, 2024
25 of 29 checks passed
iszhangpch pushed a commit to iszhangpch/doris-p that referenced this pull request Jun 21, 2024
@zzzxl1993
[opt](inverted index) reduce generation of the rowid_result if not ne…
768347a 
…cessary (apache#35357)

in the compound process, if all indexes are fully matched, the
rowid_result will no longer be generated.
dataroaring pushed a commit that referenced this pull request Jun 21, 2024
@zzzxl1993 @dataroaring
[opt](inverted index) reduce generation of the rowid_result if not ne…
ba92161 
…cessary (#35357)

in the compound process, if all indexes are fully matched, the
rowid_result will no longer be generated.
xiaokang pushed a commit that referenced this pull request Jul 6, 2024
@zzzxl1993
[opt](inverted index) reduce generation of the rowid_result if not ne…
c399a0e 
…cessary #35357 (#36569)
xiaokang pushed a commit that referenced this pull request Jul 7, 2024
@zzzxl1993
[opt](inverted index) reduce generation of the rowid_result if not ne…
a420cf6 
…cessary #35357 (#36571)
@yiguolei yiguolei mentioned this pull request Jul 19, 2024
Open
1 task
mongo360 pushed a commit to mongo360/doris that referenced this pull request Aug 16, 2024
@zzzxl1993
[opt](inverted index) reduce generation of the rowid_result if not ne…
0c9b0e6 
…cessary apache#35357 (apache#36571)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@qidaye qidaye qidaye approved these changes

@xiaokang xiaokang xiaokang approved these changes

@airborne12 airborne12 airborne12 left review comments

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by one committer. dev/2.0.13-merged dev/2.1.5-merged dev/3.0.0-merged reviewed usercase Important user case type label
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
6 participants
@zzzxl1993 @doris-robot @xiaokang @airborne12 @qidaye @dataroaring