Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Improvement](inverted index) Remove the check for inverted index file exists #36945

Merged
merged 1 commit into from
Jul 2, 2024
Merged

[Improvement](inverted index) Remove the check for inverted index file exists #36945

merged 1 commit into from
Jul 2, 2024

Conversation

csun5285
Copy link
Contributor

Proposed changes

#36929

Issue Number: close #xxx

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@csun5285 csun5285 mentioned this pull request Jun 27, 2024
Merged
@csun5285
Copy link
Contributor Author

run buildall

@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TPC-H: Total hot run time: 40461 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 8dea62212e7a0a71f4a1bacf8927c381b4e60121, data reload: false

------ Round 1 ----------------------------------
q1	17871	4667	4457	4457
q2	2626	194	190	190
q3	10884	1230	1111	1111
q4	11202	848	810	810
q5	7817	2704	2704	2704
q6	224	141	141	141
q7	968	614	623	614
q8	9546	2177	2128	2128
q9	9062	6580	6561	6561
q10	8975	3739	3732	3732
q11	450	243	235	235
q12	443	229	228	228
q13	17766	2980	2964	2964
q14	260	220	223	220
q15	523	476	472	472
q16	494	393	381	381
q17	990	703	701	701
q18	8160	7593	7438	7438
q19	1891	1484	1418	1418
q20	646	306	304	304
q21	4988	3840	3320	3320
q22	394	332	335	332
Total cold run time: 116180 ms
Total hot run time: 40461 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4338	4308	4273	4273
q2	382	274	270	270
q3	2981	2800	2764	2764
q4	1867	1599	1590	1590
q5	5299	5313	5285	5285
q6	219	137	131	131
q7	2122	1772	1766	1766
q8	3213	3377	3344	3344
q9	8352	8359	8333	8333
q10	3857	3684	3695	3684
q11	599	491	483	483
q12	798	614	599	599
q13	16773	3012	2972	2972
q14	307	262	276	262
q15	531	482	493	482
q16	475	433	426	426
q17	1783	1500	1505	1500
q18	7700	7489	7371	7371
q19	1719	1636	1610	1610
q20	1975	1761	1819	1761
q21	4980	4718	4801	4718
q22	615	549	534	534
Total cold run time: 70885 ms
Total hot run time: 54158 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 172517 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 8dea62212e7a0a71f4a1bacf8927c381b4e60121, data reload: false

query1	919	385	367	367
query2	6474	2386	2280	2280
query3	6645	205	209	205
query4	19416	17688	17224	17224
query5	4259	483	461	461
query6	285	176	164	164
query7	4603	297	289	289
query8	303	273	271	271
query9	8796	2436	2401	2401
query10	612	300	279	279
query11	10431	10056	9969	9969
query12	128	92	79	79
query13	1658	364	369	364
query14	11294	6747	7566	6747
query15	220	185	187	185
query16	7819	264	261	261
query17	1918	583	510	510
query18	1986	266	267	266
query19	189	147	156	147
query20	86	82	81	81
query21	207	131	128	128
query22	4399	4172	4129	4129
query23	33741	32820	33006	32820
query24	11030	2849	2834	2834
query25	640	362	356	356
query26	1608	159	151	151
query27	2992	315	314	314
query28	7609	2092	2077	2077
query29	1012	632	612	612
query30	281	150	155	150
query31	954	720	745	720
query32	86	54	53	53
query33	771	279	284	279
query34	976	468	466	466
query35	738	624	609	609
query36	1108	937	927	927
query37	154	69	69	69
query38	2866	2721	2779	2721
query39	861	803	784	784
query40	292	124	130	124
query41	54	54	56	54
query42	129	102	102	102
query43	573	560	516	516
query44	1270	735	728	728
query45	195	156	168	156
query46	1080	758	726	726
query47	1850	1761	1753	1753
query48	367	298	299	298
query49	1153	410	412	410
query50	772	379	376	376
query51	6846	6786	6773	6773
query52	109	91	94	91
query53	353	294	288	288
query54	854	442	440	440
query55	75	74	72	72
query56	277	259	263	259
query57	1155	1059	1058	1058
query58	265	249	248	248
query59	3511	2954	3035	2954
query60	290	271	294	271
query61	91	95	96	95
query62	640	441	445	441
query63	321	294	285	285
query64	9757	2241	1792	1792
query65	3183	3116	3109	3109
query66	1187	352	388	352
query67	15523	15064	15108	15064
query68	4927	550	566	550
query69	603	454	383	383
query70	1119	1178	1080	1080
query71	441	275	277	275
query72	7108	5752	5532	5532
query73	761	325	325	325
query74	5865	5442	5535	5442
query75	3745	2698	2696	2696
query76	3585	953	1008	953
query77	680	310	306	306
query78	10348	9846	9724	9724
query79	2372	523	527	523
query80	1335	466	460	460
query81	564	222	221	221
query82	763	103	108	103
query83	212	170	174	170
query84	274	84	85	84
query85	1445	290	269	269
query86	405	311	332	311
query87	3288	3099	3076	3076
query88	4206	2371	2368	2368
query89	474	379	382	379
query90	1843	191	190	190
query91	129	107	100	100
query92	60	52	50	50
query93	3244	513	517	513
query94	1170	188	188	188
query95	401	315	308	308
query96	604	272	271	271
query97	3227	3066	3042	3042
query98	233	200	193	193
query99	1272	862	848	848
Total cold run time: 278949 ms
Total hot run time: 172517 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.06 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 8dea62212e7a0a71f4a1bacf8927c381b4e60121, data reload: false

query1	0.04	0.04	0.03
query2	0.09	0.04	0.05
query3	0.23	0.05	0.05
query4	1.67	0.07	0.08
query5	0.50	0.47	0.49
query6	1.14	0.73	0.72
query7	0.02	0.02	0.01
query8	0.05	0.04	0.05
query9	0.56	0.50	0.51
query10	0.55	0.55	0.55
query11	0.15	0.11	0.12
query12	0.15	0.12	0.12
query13	0.60	0.58	0.60
query14	0.77	0.76	0.79
query15	0.84	0.84	0.81
query16	0.37	0.37	0.38
query17	0.99	1.03	0.99
query18	0.20	0.25	0.26
query19	1.92	1.84	1.80
query20	0.02	0.01	0.01
query21	15.45	0.76	0.66
query22	3.86	7.19	2.33
query23	18.29	1.35	1.24
query24	2.13	0.23	0.22
query25	0.15	0.09	0.08
query26	0.28	0.18	0.17
query27	0.09	0.08	0.08
query28	13.26	1.03	1.00
query29	12.63	3.38	3.30
query30	0.26	0.07	0.05
query31	2.85	0.39	0.39
query32	3.24	0.47	0.47
query33	2.90	2.88	2.95
query34	17.34	4.43	4.43
query35	4.44	4.44	4.49
query36	0.65	0.46	0.47
query37	0.17	0.15	0.14
query38	0.15	0.15	0.14
query39	0.04	0.04	0.04
query40	0.18	0.15	0.14
query41	0.09	0.05	0.05
query42	0.06	0.04	0.04
query43	0.04	0.03	0.04
Total cold run time: 109.41 s
Total hot run time: 31.06 s

@csun5285
Copy link
Contributor Author

run buildall

@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TPC-H: Total hot run time: 39982 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 80a1c56016247b5b423e45d2f31a418fcf4ddcb9, data reload: false

------ Round 1 ----------------------------------
q1	17604	4395	4281	4281
q2	2015	200	202	200
q3	10432	1270	1253	1253
q4	10195	913	792	792
q5	7490	2640	2640	2640
q6	217	131	132	131
q7	955	593	603	593
q8	9233	2117	2075	2075
q9	9151	6564	6467	6467
q10	9026	3770	3763	3763
q11	472	237	237	237
q12	419	235	225	225
q13	17775	2975	2984	2975
q14	267	226	217	217
q15	526	477	463	463
q16	524	387	378	378
q17	996	665	642	642
q18	8120	7497	7420	7420
q19	8034	1548	1420	1420
q20	665	342	334	334
q21	5005	3142	3870	3142
q22	387	334	343	334
Total cold run time: 119508 ms
Total hot run time: 39982 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4396	4254	4275	4254
q2	373	254	264	254
q3	3125	2890	2952	2890
q4	2003	1749	1680	1680
q5	5540	5508	5516	5508
q6	223	134	133	133
q7	2250	1866	1874	1866
q8	3301	3464	3398	3398
q9	8725	8769	8719	8719
q10	4192	3740	3848	3740
q11	590	501	496	496
q12	820	653	640	640
q13	15986	3200	3185	3185
q14	290	291	269	269
q15	535	475	486	475
q16	499	439	484	439
q17	1835	1561	1512	1512
q18	8109	8056	7738	7738
q19	1828	1703	1670	1670
q20	3034	1910	1885	1885
q21	8380	5058	4860	4860
q22	636	561	587	561
Total cold run time: 76670 ms
Total hot run time: 56172 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 174771 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 80a1c56016247b5b423e45d2f31a418fcf4ddcb9, data reload: false

query1	934	376	382	376
query2	6452	2429	2314	2314
query3	6634	202	208	202
query4	19638	17485	17400	17400
query5	3662	492	482	482
query6	257	186	169	169
query7	4590	291	294	291
query8	314	291	299	291
query9	8577	2443	2454	2443
query10	571	314	310	310
query11	10715	10049	10118	10049
query12	118	94	83	83
query13	1639	377	364	364
query14	9448	6799	7586	6799
query15	242	198	188	188
query16	7732	271	272	271
query17	1910	564	540	540
query18	1973	286	280	280
query19	199	157	160	157
query20	96	85	85	85
query21	221	137	129	129
query22	4273	3954	4024	3954
query23	33892	33756	33765	33756
query24	10687	2912	2980	2912
query25	632	420	408	408
query26	731	166	162	162
query27	2252	332	326	326
query28	5987	2154	2146	2146
query29	936	661	656	656
query30	261	163	158	158
query31	983	774	761	761
query32	92	60	56	56
query33	674	311	305	305
query34	896	471	504	471
query35	759	671	635	635
query36	1124	996	1010	996
query37	152	79	81	79
query38	3004	2869	2891	2869
query39	889	853	853	853
query40	213	135	134	134
query41	61	54	57	54
query42	107	105	110	105
query43	600	538	553	538
query44	1097	739	729	729
query45	200	169	168	168
query46	1082	738	736	736
query47	1870	1769	1787	1769
query48	370	310	308	308
query49	858	429	428	428
query50	769	390	492	390
query51	6890	6857	6731	6731
query52	109	91	96	91
query53	365	307	304	304
query54	875	442	446	442
query55	75	74	76	74
query56	276	256	273	256
query57	1131	1062	1053	1053
query58	256	260	257	257
query59	3427	3135	3378	3135
query60	302	296	283	283
query61	96	108	90	90
query62	596	452	438	438
query63	313	291	295	291
query64	8460	2248	1757	1757
query65	3163	3117	3099	3099
query66	736	330	333	330
query67	15666	14833	14868	14833
query68	7343	549	547	547
query69	709	552	445	445
query70	1139	1130	1118	1118
query71	504	281	283	281
query72	8694	5676	5579	5579
query73	827	321	325	321
query74	6048	5501	5442	5442
query75	5086	2652	2698	2652
query76	4352	1009	921	921
query77	787	303	301	301
query78	10486	9857	10098	9857
query79	8209	527	518	518
query80	1004	482	473	473
query81	559	223	228	223
query82	770	108	107	107
query83	352	176	172	172
query84	273	87	89	87
query85	1290	279	270	270
query86	460	313	327	313
query87	3339	3152	3077	3077
query88	5176	2448	2439	2439
query89	524	384	389	384
query90	1861	193	195	193
query91	133	105	100	100
query92	61	52	51	51
query93	6459	513	506	506
query94	1172	193	189	189
query95	416	327	320	320
query96	610	271	267	267
query97	3209	3010	3032	3010
query98	214	238	205	205
query99	1143	854	855	854
Total cold run time: 287318 ms
Total hot run time: 174771 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.27 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 80a1c56016247b5b423e45d2f31a418fcf4ddcb9, data reload: false

query1	0.04	0.03	0.03
query2	0.08	0.04	0.04
query3	0.22	0.05	0.05
query4	1.68	0.08	0.08
query5	0.49	0.49	0.48
query6	1.13	0.73	0.73
query7	0.02	0.01	0.01
query8	0.06	0.05	0.05
query9	0.56	0.52	0.50
query10	0.54	0.54	0.56
query11	0.16	0.11	0.11
query12	0.15	0.12	0.13
query13	0.61	0.59	0.61
query14	0.80	0.78	0.80
query15	0.87	0.84	0.84
query16	0.38	0.36	0.38
query17	1.02	1.07	1.04
query18	0.23	0.23	0.24
query19	1.85	1.73	1.72
query20	0.01	0.01	0.01
query21	15.44	0.77	0.66
query22	3.92	6.98	2.19
query23	18.32	1.44	1.35
query24	2.18	0.23	0.22
query25	0.17	0.08	0.08
query26	0.27	0.18	0.17
query27	0.09	0.08	0.08
query28	13.14	1.06	1.03
query29	12.61	3.30	3.28
query30	0.25	0.06	0.06
query31	2.87	0.40	0.40
query32	3.28	0.48	0.50
query33	2.95	2.98	2.86
query34	17.06	4.53	4.47
query35	4.58	4.53	4.57
query36	0.64	0.48	0.49
query37	0.17	0.15	0.15
query38	0.15	0.14	0.15
query39	0.04	0.04	0.04
query40	0.18	0.15	0.16
query41	0.09	0.05	0.05
query42	0.05	0.05	0.05
query43	0.04	0.05	0.04
Total cold run time: 109.39 s
Total hot run time: 31.27 s

@csun5285 csun5285 changed the title [fix](inverted index) Remove the check for inverted index file exists [Improvement](inverted index) Remove the check for inverted index file exists Jun 28, 2024
airborne12 pushed a commit that referenced this pull request Jun 28, 2024
@csun5285
[Improvement](inverted index) Remove the check for inverted index fil…
132df2d 
…e exists (#36929)

## Proposed changes

backport #36945
@csun5285
Copy link
Contributor Author

run buildall

@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TPC-H: Total hot run time: 39670 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 8e07bd669a153e6a84764f9d5bc59ba44758b8b3, data reload: false

------ Round 1 ----------------------------------
q1	17892	4637	4250	4250
q2	2030	192	196	192
q3	10557	1198	1140	1140
q4	10223	826	651	651
q5	7485	2676	2658	2658
q6	218	137	135	135
q7	956	604	592	592
q8	9229	2101	2091	2091
q9	9030	6495	6514	6495
q10	8929	3665	3714	3665
q11	458	244	242	242
q12	470	230	224	224
q13	17762	2965	3001	2965
q14	275	222	225	222
q15	522	480	495	480
q16	507	385	381	381
q17	974	657	762	657
q18	8148	7430	7401	7401
q19	3506	1507	1450	1450
q20	657	312	323	312
q21	5053	3125	4042	3125
q22	417	347	342	342
Total cold run time: 115298 ms
Total hot run time: 39670 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4330	4259	4208	4208
q2	368	265	275	265
q3	3003	2795	2925	2795
q4	2015	1731	1637	1637
q5	5614	5522	5502	5502
q6	229	129	127	127
q7	2184	1835	1882	1835
q8	3270	3421	3452	3421
q9	8661	8724	8826	8724
q10	4072	3945	3766	3766
q11	591	490	494	490
q12	774	626	678	626
q13	17376	3182	3172	3172
q14	308	289	276	276
q15	539	490	490	490
q16	493	429	423	423
q17	1828	1528	1506	1506
q18	8155	7867	7717	7717
q19	4255	1516	1676	1516
q20	2049	1923	1849	1849
q21	5119	4892	4803	4803
q22	628	568	560	560
Total cold run time: 75861 ms
Total hot run time: 55708 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 174618 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 8e07bd669a153e6a84764f9d5bc59ba44758b8b3, data reload: false

query1	931	382	380	380
query2	6449	2482	2383	2383
query3	6632	207	213	207
query4	18932	17437	17156	17156
query5	3672	461	473	461
query6	252	186	171	171
query7	4597	310	298	298
query8	326	298	298	298
query9	8632	2448	2425	2425
query10	583	311	285	285
query11	10619	10174	9988	9988
query12	118	90	83	83
query13	1634	371	368	368
query14	10225	7629	7547	7547
query15	300	195	195	195
query16	8045	279	269	269
query17	1905	540	535	535
query18	2116	281	282	281
query19	208	154	157	154
query20	96	87	82	82
query21	210	131	123	123
query22	4289	3950	3994	3950
query23	33895	33818	33715	33715
query24	10948	2889	2960	2889
query25	639	382	401	382
query26	1325	156	158	156
query27	2990	328	322	322
query28	7734	2151	2146	2146
query29	934	635	651	635
query30	258	157	162	157
query31	974	770	745	745
query32	95	55	56	55
query33	795	315	299	299
query34	920	473	484	473
query35	766	667	635	635
query36	1121	983	943	943
query37	164	73	81	73
query38	2978	2861	2807	2807
query39	891	815	786	786
query40	218	129	126	126
query41	59	49	57	49
query42	121	103	108	103
query43	615	569	538	538
query44	1199	748	735	735
query45	202	164	165	164
query46	1071	748	729	729
query47	1875	1784	1810	1784
query48	390	301	299	299
query49	876	490	416	416
query50	768	376	380	376
query51	6984	6796	6833	6796
query52	103	89	97	89
query53	355	288	285	285
query54	909	462	445	445
query55	76	73	72	72
query56	286	253	262	253
query57	1144	1050	1068	1050
query58	244	252	235	235
query59	3415	3023	3007	3007
query60	289	273	288	273
query61	97	92	110	92
query62	605	452	451	451
query63	322	280	280	280
query64	9295	2264	1791	1791
query65	3178	3086	3105	3086
query66	905	327	323	323
query67	15654	15014	14828	14828
query68	8369	546	553	546
query69	689	516	415	415
query70	1186	1120	1139	1120
query71	511	272	277	272
query72	9181	5749	5492	5492
query73	1389	325	322	322
query74	5864	5535	5550	5535
query75	5313	2646	2667	2646
query76	5002	937	944	937
query77	774	302	311	302
query78	10495	9863	9841	9841
query79	7635	510	511	510
query80	967	471	471	471
query81	572	230	226	226
query82	784	105	103	103
query83	368	177	172	172
query84	317	87	86	86
query85	1358	279	266	266
query86	397	330	330	330
query87	3315	3115	3107	3107
query88	4511	2372	2332	2332
query89	488	383	381	381
query90	1908	192	193	192
query91	129	101	98	98
query92	57	51	57	51
query93	5629	519	512	512
query94	1154	194	190	190
query95	420	322	319	319
query96	602	265	264	264
query97	3252	3072	3013	3013
query98	217	205	196	196
query99	1207	839	861	839
Total cold run time: 293513 ms
Total hot run time: 174618 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.63 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 8e07bd669a153e6a84764f9d5bc59ba44758b8b3, data reload: false

query1	0.04	0.04	0.03
query2	0.08	0.03	0.04
query3	0.22	0.05	0.05
query4	1.68	0.07	0.07
query5	0.48	0.47	0.48
query6	1.14	0.73	0.73
query7	0.02	0.01	0.01
query8	0.05	0.05	0.05
query9	0.56	0.52	0.49
query10	0.56	0.55	0.55
query11	0.15	0.12	0.11
query12	0.15	0.13	0.12
query13	0.59	0.58	0.60
query14	0.79	0.78	0.78
query15	0.85	0.84	0.81
query16	0.36	0.36	0.35
query17	1.00	0.96	0.95
query18	0.20	0.26	0.24
query19	1.89	1.70	1.76
query20	0.01	0.01	0.01
query21	15.54	0.75	0.66
query22	4.55	6.48	1.94
query23	18.26	1.41	1.33
query24	2.16	0.23	0.22
query25	0.16	0.08	0.09
query26	0.27	0.18	0.18
query27	0.08	0.09	0.08
query28	13.16	1.03	1.00
query29	12.61	3.33	3.27
query30	0.26	0.06	0.05
query31	2.87	0.40	0.39
query32	3.25	0.47	0.47
query33	2.84	2.87	2.91
query34	17.10	4.42	4.47
query35	4.56	4.49	4.50
query36	0.65	0.49	0.47
query37	0.19	0.15	0.14
query38	0.15	0.14	0.14
query39	0.05	0.04	0.04
query40	0.18	0.15	0.14
query41	0.09	0.05	0.04
query42	0.05	0.04	0.04
query43	0.04	0.04	0.04
Total cold run time: 109.89 s
Total hot run time: 30.63 s

xiaokang
xiaokang reviewed Jun 28, 2024
View reviewed changes
Copy link
Contributor

@xiaokang xiaokang left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

pls add testcase

@csun5285
fix exists
4886c43 
add clucene

fix v2

remove unused code

add test
@csun5285
Copy link
Contributor Author

csun5285 commented Jul 1, 2024

run buildall

@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Jul 1, 2024

clang-tidy review says "All clean, LGTM! 👍"

1 similar comment
@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Jul 1, 2024

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TPC-H: Total hot run time: 39877 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 4886c4342cfebbd915bdea6595e77e3f2e13df39, data reload: false

------ Round 1 ----------------------------------
q1	17599	4544	4260	4260
q2	2031	198	197	197
q3	10507	1236	1093	1093
q4	10202	801	723	723
q5	7550	2691	2671	2671
q6	221	139	140	139
q7	962	584	604	584
q8	9211	2076	2102	2076
q9	9106	6493	6459	6459
q10	8931	3705	3736	3705
q11	457	237	236	236
q12	482	231	232	231
q13	17780	2974	2992	2974
q14	265	225	222	222
q15	518	483	473	473
q16	512	375	375	375
q17	968	633	671	633
q18	8199	7522	7480	7480
q19	7300	1515	1526	1515
q20	664	317	332	317
q21	4912	3182	3249	3182
q22	397	332	345	332
Total cold run time: 118774 ms
Total hot run time: 39877 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4381	4234	4235	4234
q2	369	261	262	261
q3	2960	2946	2884	2884
q4	2010	1663	1745	1663
q5	5603	5504	5471	5471
q6	228	145	132	132
q7	2186	1894	1843	1843
q8	3257	3460	3446	3446
q9	8742	8755	8795	8755
q10	4168	3756	3728	3728
q11	584	504	493	493
q12	838	648	647	647
q13	16018	3129	3147	3129
q14	291	296	283	283
q15	538	514	488	488
q16	498	437	437	437
q17	1849	1537	1535	1535
q18	8194	8037	7870	7870
q19	1766	1716	1681	1681
q20	2257	1865	1873	1865
q21	10112	5016	4823	4823
q22	607	567	561	561
Total cold run time: 77456 ms
Total hot run time: 56229 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 172496 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 4886c4342cfebbd915bdea6595e77e3f2e13df39, data reload: false

query1	919	387	375	375
query2	6450	2393	2433	2393
query3	6639	202	210	202
query4	20044	17367	17146	17146
query5	3683	487	479	479
query6	257	157	159	157
query7	4602	291	288	288
query8	320	279	301	279
query9	8712	2408	2399	2399
query10	585	309	275	275
query11	10566	10004	10028	10004
query12	115	87	85	85
query13	1646	371	362	362
query14	9416	6330	6945	6330
query15	240	189	198	189
query16	7741	266	272	266
query17	1614	593	551	551
query18	1915	285	277	277
query19	209	160	157	157
query20	88	82	81	81
query21	210	128	124	124
query22	4243	3973	3936	3936
query23	33974	33480	33910	33480
query24	10885	2848	2918	2848
query25	655	401	384	384
query26	717	162	154	154
query27	2328	333	326	326
query28	6249	2150	2138	2138
query29	917	656	662	656
query30	265	159	163	159
query31	998	762	768	762
query32	97	58	61	58
query33	703	318	321	318
query34	965	474	484	474
query35	760	628	641	628
query36	1158	954	938	938
query37	142	78	80	78
query38	2954	2928	2813	2813
query39	900	863	804	804
query40	207	122	122	122
query41	53	50	51	50
query42	114	98	100	98
query43	590	535	537	535
query44	1083	718	725	718
query45	194	174	168	168
query46	1087	709	713	709
query47	1863	1726	1772	1726
query48	369	298	288	288
query49	839	405	407	405
query50	763	386	375	375
query51	6886	6827	6725	6725
query52	110	91	94	91
query53	351	284	281	281
query54	856	438	436	436
query55	72	73	75	73
query56	282	258	259	258
query57	1104	1060	1034	1034
query58	242	246	242	242
query59	3555	3266	3205	3205
query60	299	276	278	276
query61	91	91	90	90
query62	614	442	444	442
query63	322	291	289	289
query64	8479	2228	1735	1735
query65	3180	3105	3076	3076
query66	750	324	351	324
query67	15545	14834	15002	14834
query68	8141	545	603	545
query69	691	435	350	350
query70	1153	1055	1062	1055
query71	509	276	281	276
query72	7289	5320	5927	5320
query73	837	323	323	323
query74	5901	5512	5438	5438
query75	4390	2659	2692	2659
query76	4622	945	882	882
query77	711	303	309	303
query78	10555	9722	9805	9722
query79	7706	508	507	507
query80	1480	468	470	468
query81	551	225	220	220
query82	790	149	105	105
query83	200	164	171	164
query84	278	83	87	83
query85	1232	270	262	262
query86	465	312	302	302
query87	3306	3089	3077	3077
query88	5082	2353	2341	2341
query89	502	372	373	372
query90	1831	184	187	184
query91	126	97	95	95
query92	63	50	50	50
query93	6004	506	502	502
query94	1046	184	188	184
query95	402	311	315	311
query96	611	274	264	264
query97	3229	3102	3099	3099
query98	218	201	191	191
query99	1218	864	867	864
Total cold run time: 285920 ms
Total hot run time: 172496 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.62 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 4886c4342cfebbd915bdea6595e77e3f2e13df39, data reload: false

query1	0.04	0.04	0.03
query2	0.08	0.03	0.04
query3	0.23	0.05	0.06
query4	1.65	0.07	0.12
query5	0.50	0.49	0.50
query6	1.14	0.72	0.72
query7	0.02	0.01	0.01
query8	0.05	0.04	0.04
query9	0.55	0.51	0.49
query10	0.54	0.55	0.54
query11	0.15	0.12	0.11
query12	0.14	0.12	0.11
query13	0.59	0.58	0.59
query14	0.76	0.79	0.78
query15	0.84	0.82	0.82
query16	0.37	0.37	0.36
query17	0.95	0.99	0.97
query18	0.24	0.23	0.23
query19	1.77	1.66	1.66
query20	0.02	0.01	0.01
query21	15.43	0.76	0.63
query22	4.80	6.33	2.08
query23	18.29	1.32	1.31
query24	2.15	0.21	0.21
query25	0.15	0.08	0.08
query26	0.27	0.18	0.17
query27	0.09	0.08	0.08
query28	13.28	1.00	0.99
query29	12.63	3.25	3.24
query30	0.25	0.06	0.05
query31	2.87	0.38	0.38
query32	3.26	0.47	0.47
query33	2.86	2.87	2.92
query34	17.02	4.42	4.46
query35	4.49	4.50	4.47
query36	0.66	0.47	0.47
query37	0.18	0.14	0.15
query38	0.16	0.14	0.14
query39	0.04	0.04	0.04
query40	0.18	0.15	0.15
query41	0.09	0.04	0.04
query42	0.05	0.05	0.05
query43	0.05	0.04	0.04
Total cold run time: 109.88 s
Total hot run time: 30.62 s

xiaokang
xiaokang approved these changes Jul 2, 2024
View reviewed changes
Copy link
Contributor

@xiaokang xiaokang left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jul 2, 2024
@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Jul 2, 2024

PR approved by at least one committer and no changes requested.

@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Jul 2, 2024

PR approved by anyone and no changes requested.

zzzxl1993
zzzxl1993 approved these changes Jul 2, 2024
View reviewed changes
Copy link
Contributor

@zzzxl1993 zzzxl1993 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@xiaokang xiaokang merged commit a25ac14 into apache:master Jul 2, 2024
25 of 29 checks passed
@csun5285 csun5285 mentioned this pull request Jul 8, 2024
Merged
xiaokang pushed a commit that referenced this pull request Jul 11, 2024
@csun5285
[cherry-pick] (branch-2.1) Remove the check for inverted index file e…
79a2082 
…xists #36945 (#37423)
@csun5285 csun5285 mentioned this pull request Jul 17, 2024
Closed
dataroaring pushed a commit that referenced this pull request Jul 17, 2024
@csun5285 @dataroaring
[Improvement](inverted index) Remove the check for inverted index fil…
d7adfb5 
…e exists (#36945)

Remove the check for inverted index file exists to avoid latency of remote fs eg. s3
@yiguolei yiguolei mentioned this pull request Jul 19, 2024
Open
1 task
mongo360 pushed a commit to mongo360/doris that referenced this pull request Aug 16, 2024
@csun5285
[Improvement](inverted index) Remove the check for inverted index fil…
9977edd 
…e exists (apache#36929)

## Proposed changes

backport apache#36945
@gavinchou gavinchou mentioned this pull request Aug 19, 2024
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@zzzxl1993 zzzxl1993 zzzxl1993 approved these changes

@xiaokang xiaokang xiaokang approved these changes

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by one committer. dev/2.0.13-merged dev/2.1.5-merged dev/3.0.1-merged reviewed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
6 participants
@csun5285 @doris-robot @xiaokang @zzzxl1993 @wm1581066 @dataroaring