Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[feature](move-memtable) support for inverted index file #35891

Merged
merged 7 commits into from
Jul 5, 2024
Merged

[feature](move-memtable) support for inverted index file #35891

merged 7 commits into from
Jul 5, 2024

Conversation

csun5285
Copy link
Contributor

@csun5285 csun5285 commented Jun 5, 2024

Proposed changes

support for inverted index file

Issue Number: close #xxx

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@csun5285
Copy link
Contributor Author

csun5285 commented Jun 5, 2024

run p0

github-actions[bot]
github-actions bot reviewed Jun 5, 2024
View reviewed changes
Copy link
Contributor

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

clang-tidy made some suggestions

be/src/olap/rowset/segment_creator.h Show resolved Hide resolved
@csun5285
Copy link
Contributor Author

csun5285 commented Jun 5, 2024

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 41957 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 75eddd0bfee340e7aa257f3c3d4a4e8b2e80a34c, data reload: false

------ Round 1 ----------------------------------
q1	17606	5086	4337	4337
q2	2038	198	190	190
q3	10456	1153	1168	1153
q4	10185	871	841	841
q5	7497	2686	2745	2686
q6	222	134	141	134
q7	963	612	621	612
q8	9217	2164	2121	2121
q9	9180	6718	6768	6718
q10	9515	3895	3884	3884
q11	433	236	247	236
q12	443	250	234	234
q13	18805	3221	3031	3031
q14	285	228	235	228
q15	526	508	462	462
q16	508	397	389	389
q17	1018	677	731	677
q18	8593	7947	7999	7947
q19	6944	1525	1436	1436
q20	660	314	319	314
q21	5128	4111	3986	3986
q22	398	346	341	341
Total cold run time: 120620 ms
Total hot run time: 41957 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4623	4457	4413	4413
q2	384	271	278	271
q3	3227	2825	2758	2758
q4	1931	1620	1704	1620
q5	5503	5524	5515	5515
q6	225	125	123	123
q7	2183	1847	1790	1790
q8	3262	3390	3365	3365
q9	8636	8625	8696	8625
q10	4117	3747	3733	3733
q11	593	498	491	491
q12	824	640	606	606
q13	16134	3106	3186	3106
q14	306	284	257	257
q15	532	492	496	492
q16	523	435	452	435
q17	1821	1509	1496	1496
q18	7915	7443	7444	7443
q19	1703	1610	1576	1576
q20	2055	1764	1753	1753
q21	10356	4694	4700	4694
q22	589	534	537	534
Total cold run time: 77442 ms
Total hot run time: 55096 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 170426 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 75eddd0bfee340e7aa257f3c3d4a4e8b2e80a34c, data reload: false

query1	921	394	370	370
query2	6472	2450	2267	2267
query3	6649	208	211	208
query4	19400	17504	17267	17267
query5	4128	446	449	446
query6	246	150	147	147
query7	4577	297	286	286
query8	320	286	295	286
query9	8453	2361	2379	2361
query10	469	295	276	276
query11	10644	10124	10128	10124
query12	138	90	89	89
query13	1648	378	377	377
query14	10082	7502	7475	7475
query15	239	190	179	179
query16	7844	261	270	261
query17	1828	549	510	510
query18	1940	267	265	265
query19	196	152	153	152
query20	91	81	85	81
query21	208	133	125	125
query22	4586	4234	4126	4126
query23	33587	33005	32966	32966
query24	11037	2837	2906	2837
query25	612	355	356	355
query26	1225	151	153	151
query27	2459	309	311	309
query28	7211	2039	2051	2039
query29	881	611	603	603
query30	277	150	156	150
query31	967	755	724	724
query32	92	51	53	51
query33	757	282	276	276
query34	970	476	461	461
query35	744	601	590	590
query36	1080	952	905	905
query37	160	72	71	71
query38	2936	2811	2761	2761
query39	857	795	776	776
query40	206	129	125	125
query41	57	54	53	53
query42	121	95	98	95
query43	586	537	522	522
query44	1182	747	754	747
query45	195	164	170	164
query46	1078	710	731	710
query47	1866	1763	1814	1763
query48	375	298	305	298
query49	1093	419	413	413
query50	779	392	391	391
query51	6814	6720	6660	6660
query52	103	90	92	90
query53	354	281	295	281
query54	916	457	444	444
query55	73	74	74	74
query56	287	267	276	267
query57	1155	1057	1044	1044
query58	273	277	265	265
query59	3375	3266	3202	3202
query60	307	299	293	293
query61	161	85	89	85
query62	652	460	451	451
query63	310	286	284	284
query64	8871	2182	1788	1788
query65	3162	3069	3143	3069
query66	807	322	328	322
query67	15539	15104	15012	15012
query68	4535	550	533	533
query69	459	300	301	300
query70	1135	1075	1121	1075
query71	403	280	287	280
query72	7078	4726	2533	2533
query73	724	321	322	321
query74	5985	5605	5665	5605
query75	3388	2642	2691	2642
query76	2838	899	960	899
query77	480	298	288	288
query78	10273	9823	9721	9721
query79	2075	519	508	508
query80	2538	452	443	443
query81	598	214	221	214
query82	1047	99	104	99
query83	318	172	170	170
query84	264	87	86	86
query85	1280	262	264	262
query86	483	306	310	306
query87	3292	3096	3096	3096
query88	3396	2422	2428	2422
query89	483	391	379	379
query90	1800	186	185	185
query91	125	97	101	97
query92	68	51	50	50
query93	2005	509	495	495
query94	1276	182	183	182
query95	404	309	312	309
query96	593	274	328	274
query97	3179	2994	3016	2994
query98	252	220	208	208
query99	1172	835	868	835
Total cold run time: 271574 ms
Total hot run time: 170426 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.8 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 75eddd0bfee340e7aa257f3c3d4a4e8b2e80a34c, data reload: false

query1	0.04	0.03	0.03
query2	0.08	0.04	0.04
query3	0.23	0.05	0.04
query4	1.68	0.09	0.08
query5	0.51	0.47	0.51
query6	1.11	0.72	0.73
query7	0.02	0.01	0.01
query8	0.05	0.04	0.04
query9	0.53	0.50	0.48
query10	0.55	0.54	0.54
query11	0.16	0.12	0.11
query12	0.14	0.11	0.12
query13	0.59	0.58	0.59
query14	0.81	0.75	0.78
query15	0.82	0.80	0.81
query16	0.37	0.37	0.36
query17	0.95	0.95	1.02
query18	0.22	0.24	0.25
query19	1.82	1.79	1.74
query20	0.01	0.01	0.02
query21	15.44	0.70	0.67
query22	4.65	6.57	2.20
query23	18.30	1.40	1.29
query24	1.91	0.23	0.20
query25	0.15	0.09	0.09
query26	0.27	0.17	0.17
query27	0.08	0.08	0.09
query28	13.28	1.02	0.99
query29	13.32	3.28	3.26
query30	0.23	0.06	0.06
query31	2.89	0.40	0.39
query32	3.25	0.47	0.46
query33	2.90	2.88	2.94
query34	17.10	4.45	4.41
query35	4.45	4.51	4.71
query36	0.64	0.48	0.47
query37	0.17	0.15	0.15
query38	0.14	0.14	0.14
query39	0.04	0.03	0.04
query40	0.16	0.14	0.14
query41	0.08	0.04	0.05
query42	0.05	0.05	0.04
query43	0.04	0.03	0.03
Total cold run time: 110.23 s
Total hot run time: 30.8 s

@csun5285
Copy link
Contributor Author

csun5285 commented Jun 5, 2024

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 40965 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit b6b5592a326061e6e134e59a45b519b918b6b905, data reload: false

------ Round 1 ----------------------------------
q1	17608	4446	4281	4281
q2	2035	197	194	194
q3	10448	1210	1100	1100
q4	10199	826	915	826
q5	7491	2731	2723	2723
q6	224	137	135	135
q7	989	619	632	619
q8	9221	2138	2107	2107
q9	9449	6694	6688	6688
q10	9377	3975	3947	3947
q11	439	236	236	236
q12	449	226	235	226
q13	18490	3213	3209	3209
q14	268	218	230	218
q15	520	472	483	472
q16	505	396	392	392
q17	982	617	659	617
q18	8467	7879	7910	7879
q19	7043	1377	1206	1206
q20	636	309	309	309
q21	5164	4034	3250	3250
q22	393	331	354	331
Total cold run time: 120397 ms
Total hot run time: 40965 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4677	4420	4398	4398
q2	371	282	281	281
q3	3118	3016	2811	2811
q4	1888	1607	1690	1607
q5	5494	5502	5568	5502
q6	218	127	126	126
q7	2189	1794	1803	1794
q8	3250	3428	3381	3381
q9	8601	8712	8689	8689
q10	4121	3869	3719	3719
q11	580	482	472	472
q12	806	637	669	637
q13	17207	3137	3174	3137
q14	328	263	271	263
q15	542	496	479	479
q16	499	452	428	428
q17	1831	1521	1510	1510
q18	8013	7482	7516	7482
q19	1697	1477	1574	1477
q20	2073	1781	1777	1777
q21	7670	4695	4734	4695
q22	612	524	543	524
Total cold run time: 75785 ms
Total hot run time: 55189 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 173367 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit b6b5592a326061e6e134e59a45b519b918b6b905, data reload: false

query1	943	384	374	374
query2	7813	2552	2526	2526
query3	6644	206	207	206
query4	19186	17505	17385	17385
query5	4107	460	456	456
query6	251	173	153	153
query7	4601	305	293	293
query8	318	278	275	275
query9	8509	2399	2387	2387
query10	451	342	282	282
query11	10469	10084	10039	10039
query12	130	93	84	84
query13	1655	388	407	388
query14	9876	6739	7011	6739
query15	243	181	182	181
query16	7215	278	271	271
query17	1383	520	497	497
query18	1909	270	277	270
query19	213	151	151	151
query20	92	82	81	81
query21	204	134	126	126
query22	4380	3997	3948	3948
query23	33766	33342	33275	33275
query24	11312	2851	2794	2794
query25	635	358	364	358
query26	1387	155	155	155
query27	3039	323	327	323
query28	7623	2051	2047	2047
query29	997	623	625	623
query30	288	152	152	152
query31	978	764	732	732
query32	102	57	62	57
query33	777	323	293	293
query34	914	481	487	481
query35	792	631	604	604
query36	1095	950	902	902
query37	159	67	70	67
query38	2867	2717	2741	2717
query39	859	808	793	793
query40	219	128	123	123
query41	53	50	52	50
query42	123	101	100	100
query43	595	569	556	556
query44	1261	738	773	738
query45	192	165	162	162
query46	1073	723	724	723
query47	1846	1787	1765	1765
query48	380	297	305	297
query49	1088	433	412	412
query50	781	380	396	380
query51	6829	6723	6709	6709
query52	99	98	89	89
query53	356	289	302	289
query54	860	465	435	435
query55	75	77	74	74
query56	272	252	262	252
query57	1129	1031	1079	1031
query58	255	253	262	253
query59	3741	3461	3336	3336
query60	292	277	267	267
query61	116	88	85	85
query62	657	459	436	436
query63	323	301	285	285
query64	8993	2189	1718	1718
query65	3183	3126	3164	3126
query66	1308	320	332	320
query67	15535	14987	15035	14987
query68	4525	559	548	548
query69	469	301	304	301
query70	1168	1123	1158	1123
query71	404	273	281	273
query72	7238	5509	5830	5509
query73	745	328	325	325
query74	6004	5572	5476	5476
query75	3470	2669	2640	2640
query76	2361	938	933	933
query77	480	304	306	304
query78	10323	9808	9891	9808
query79	2493	527	517	517
query80	1078	478	467	467
query81	600	225	229	225
query82	839	107	107	107
query83	241	177	172	172
query84	242	85	93	85
query85	1460	292	262	262
query86	478	318	322	318
query87	3316	3155	3089	3089
query88	4365	2352	2357	2352
query89	479	393	394	393
query90	1731	192	190	190
query91	125	98	98	98
query92	69	51	50	50
query93	1817	521	506	506
query94	1232	184	187	184
query95	406	314	319	314
query96	589	274	265	265
query97	3161	3025	3030	3025
query98	225	207	200	200
query99	1273	846	825	825
Total cold run time: 273222 ms
Total hot run time: 173367 ms

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 36.37% (8986/24710)
Line Coverage: 27.90% (73440/263227)
Region Coverage: 27.33% (38006/139085)
Branch Coverage: 23.92% (19291/80646)
Coverage Report: http://coverage.selectdb-in.cc/coverage/b6b5592a326061e6e134e59a45b519b918b6b905_b6b5592a326061e6e134e59a45b519b918b6b905/report/index.html

@doris-robot
Copy link

ClickBench: Total hot run time: 30.75 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit b6b5592a326061e6e134e59a45b519b918b6b905, data reload: false

query1	0.04	0.03	0.03
query2	0.08	0.04	0.05
query3	0.23	0.05	0.05
query4	1.66	0.07	0.07
query5	0.51	0.47	0.48
query6	1.13	0.72	0.72
query7	0.02	0.02	0.01
query8	0.05	0.04	0.04
query9	0.55	0.50	0.50
query10	0.54	0.55	0.55
query11	0.15	0.11	0.11
query12	0.14	0.12	0.12
query13	0.60	0.59	0.62
query14	0.78	0.77	0.80
query15	0.83	0.82	0.82
query16	0.35	0.37	0.37
query17	1.03	0.98	1.04
query18	0.21	0.27	0.22
query19	1.77	1.68	1.70
query20	0.02	0.01	0.01
query21	15.56	0.69	0.67
query22	4.78	6.86	2.00
query23	18.28	1.30	1.35
query24	2.13	0.24	0.21
query25	0.16	0.08	0.08
query26	0.26	0.17	0.17
query27	0.09	0.08	0.07
query28	13.16	1.02	1.01
query29	13.33	3.23	3.24
query30	0.24	0.05	0.06
query31	2.85	0.38	0.39
query32	3.27	0.47	0.47
query33	2.90	2.88	2.92
query34	17.30	4.49	4.45
query35	4.50	4.58	4.68
query36	0.64	0.47	0.47
query37	0.16	0.15	0.16
query38	0.14	0.14	0.14
query39	0.04	0.04	0.03
query40	0.17	0.15	0.17
query41	0.09	0.04	0.04
query42	0.05	0.05	0.04
query43	0.04	0.03	0.03
Total cold run time: 110.83 s
Total hot run time: 30.75 s

@csun5285
Copy link
Contributor Author

csun5285 commented Jun 6, 2024

run buildall

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 36.37% (8986/24709)
Line Coverage: 27.91% (73459/263237)
Region Coverage: 27.33% (38024/139116)
Branch Coverage: 23.93% (19296/80644)
Coverage Report: http://coverage.selectdb-in.cc/coverage/f8e98bc2eae02f176c7d3c415df1fde2bf020217_f8e98bc2eae02f176c7d3c415df1fde2bf020217/report/index.html

@csun5285
Copy link
Contributor Author

csun5285 commented Jun 6, 2024

run p0

@csun5285
Copy link
Contributor Author

csun5285 commented Jun 6, 2024

run p2

@csun5285
Copy link
Contributor Author

csun5285 commented Jun 6, 2024

run buildall

@csun5285
Copy link
Contributor Author

csun5285 commented Jun 6, 2024

run buildall

1 similar comment
@csun5285
Copy link
Contributor Author

csun5285 commented Jun 6, 2024

run buildall

github-actions[bot]
github-actions bot reviewed Jun 6, 2024
View reviewed changes
Copy link
Contributor

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

clang-tidy made some suggestions

be/src/runtime/load_stream_writer.h Show resolved Hide resolved
@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 36.43% (8973/24629)
Line Coverage: 27.97% (73366/262289)
Region Coverage: 27.38% (38033/138916)
Branch Coverage: 23.99% (19311/80486)
Coverage Report: http://coverage.selectdb-in.cc/coverage/ddadcae89d3878648bf2bcf069fec11d60023480_ddadcae89d3878648bf2bcf069fec11d60023480/report/index.html

@doris-robot
Copy link

TPC-H: Total hot run time: 40874 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit ddadcae89d3878648bf2bcf069fec11d60023480, data reload: false

------ Round 1 ----------------------------------
q1	17618	4349	4371	4349
q2	2033	193	194	193
q3	10465	1219	1113	1113
q4	10196	793	804	793
q5	7469	2751	2751	2751
q6	221	136	136	136
q7	960	623	616	616
q8	9234	2131	2123	2123
q9	9117	6719	6667	6667
q10	9097	3926	3836	3836
q11	471	238	238	238
q12	540	230	228	228
q13	18379	3215	3199	3199
q14	277	236	225	225
q15	537	477	469	469
q16	506	416	398	398
q17	973	583	650	583
q18	8408	7873	7799	7799
q19	7175	1256	1464	1256
q20	675	335	340	335
q21	5018	3927	3231	3231
q22	411	336	338	336
Total cold run time: 119780 ms
Total hot run time: 40874 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4538	4439	4595	4439
q2	370	271	276	271
q3	3140	2976	2881	2881
q4	2060	1654	1581	1581
q5	5334	5547	5489	5489
q6	224	128	129	128
q7	2188	1860	1796	1796
q8	3224	3387	3372	3372
q9	8641	8546	8658	8546
q10	4114	3842	3786	3786
q11	570	480	466	466
q12	737	583	604	583
q13	17034	3132	3101	3101
q14	316	272	279	272
q15	514	470	476	470
q16	474	415	434	415
q17	1839	1486	1494	1486
q18	8111	7795	7457	7457
q19	1717	1542	1638	1542
q20	3019	1784	1793	1784
q21	5197	4606	4865	4606
q22	726	528	545	528
Total cold run time: 74087 ms
Total hot run time: 54999 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 172320 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit ddadcae89d3878648bf2bcf069fec11d60023480, data reload: false

query1	932	376	386	376
query2	6467	2319	2432	2319
query3	6646	205	207	205
query4	20640	17440	17483	17440
query5	4116	430	446	430
query6	245	153	149	149
query7	4607	294	289	289
query8	308	282	273	273
query9	8463	2462	2420	2420
query10	461	291	267	267
query11	10598	10191	10074	10074
query12	135	83	82	82
query13	1630	357	354	354
query14	9968	6753	6827	6753
query15	228	183	180	180
query16	7135	270	264	264
query17	1016	519	513	513
query18	1905	269	269	269
query19	199	144	157	144
query20	86	80	88	80
query21	200	125	122	122
query22	4650	4433	4153	4153
query23	33949	33031	33051	33031
query24	11104	2883	2776	2776
query25	617	352	375	352
query26	1224	153	154	153
query27	3084	316	330	316
query28	7617	2059	2051	2051
query29	928	597	599	597
query30	285	149	145	145
query31	939	754	719	719
query32	91	50	53	50
query33	763	281	270	270
query34	965	469	464	464
query35	749	594	620	594
query36	1113	923	933	923
query37	155	66	75	66
query38	2885	2774	2741	2741
query39	839	780	787	780
query40	200	125	124	124
query41	53	51	51	51
query42	116	98	99	98
query43	555	525	533	525
query44	1145	713	732	713
query45	195	160	167	160
query46	1077	708	749	708
query47	1883	1776	1786	1776
query48	380	298	288	288
query49	991	401	408	401
query50	783	376	395	376
query51	6769	6737	6556	6556
query52	104	94	89	89
query53	364	285	291	285
query54	835	480	433	433
query55	75	74	75	74
query56	291	264	256	256
query57	1142	1059	1110	1059
query58	257	245	243	243
query59	3408	3140	3203	3140
query60	284	272	274	272
query61	100	84	87	84
query62	655	462	437	437
query63	315	288	286	286
query64	9046	2248	1728	1728
query65	3207	3115	3108	3108
query66	1218	333	332	332
query67	15556	14964	15062	14964
query68	4538	554	535	535
query69	473	318	310	310
query70	1172	1162	1166	1162
query71	356	269	322	269
query72	7144	5557	5180	5180
query73	760	326	324	324
query74	5964	5522	5522	5522
query75	3411	2676	2624	2624
query76	2261	943	876	876
query77	444	299	289	289
query78	10239	10015	9835	9835
query79	1982	509	506	506
query80	2145	451	452	451
query81	568	223	225	223
query82	966	102	99	99
query83	304	163	166	163
query84	273	83	94	83
query85	1327	260	261	260
query86	483	327	301	301
query87	3259	3066	3085	3066
query88	3484	2447	2430	2430
query89	482	400	370	370
query90	1699	191	189	189
query91	129	107	107	107
query92	72	51	50	50
query93	1573	508	499	499
query94	1246	196	198	196
query95	413	327	322	322
query96	584	276	272	272
query97	3217	3043	3010	3010
query98	236	204	201	201
query99	1309	846	862	846
Total cold run time: 271459 ms
Total hot run time: 172320 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.42 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit ddadcae89d3878648bf2bcf069fec11d60023480, data reload: false

query1	0.04	0.03	0.03
query2	0.08	0.03	0.04
query3	0.23	0.05	0.06
query4	1.66	0.08	0.10
query5	0.53	0.49	0.50
query6	1.13	0.74	0.72
query7	0.02	0.01	0.02
query8	0.05	0.04	0.04
query9	0.53	0.50	0.49
query10	0.54	0.56	0.54
query11	0.16	0.11	0.12
query12	0.15	0.12	0.12
query13	0.59	0.58	0.59
query14	0.78	0.78	0.78
query15	0.83	0.81	0.81
query16	0.37	0.38	0.38
query17	1.03	1.01	0.99
query18	0.22	0.23	0.27
query19	1.91	1.74	1.81
query20	0.01	0.01	0.01
query21	15.70	0.64	0.65
query22	4.34	7.79	1.67
query23	18.27	1.41	1.33
query24	2.15	0.22	0.22
query25	0.15	0.09	0.09
query26	0.25	0.18	0.18
query27	0.08	0.08	0.08
query28	13.19	1.01	1.00
query29	13.34	3.31	3.27
query30	0.24	0.06	0.05
query31	2.88	0.38	0.39
query32	3.25	0.47	0.46
query33	2.89	2.90	2.86
query34	17.24	4.50	4.44
query35	4.56	4.50	4.59
query36	0.68	0.48	0.46
query37	0.18	0.15	0.16
query38	0.15	0.14	0.13
query39	0.04	0.03	0.03
query40	0.17	0.16	0.14
query41	0.09	0.04	0.04
query42	0.05	0.05	0.04
query43	0.04	0.03	0.04
Total cold run time: 110.79 s
Total hot run time: 30.42 s

@csun5285
Copy link
Contributor Author

csun5285 commented Jun 7, 2024

run buildall

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jun 28, 2024
@github-actions GitHub Actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions GitHub Actions
Copy link
Contributor

PR approved by anyone and no changes requested.

@doris-robot
Copy link

TPC-H: Total hot run time: 39640 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 7d9f9a636726bc2b51dce8829e74e0b6de44b003, data reload: false

------ Round 1 ----------------------------------
q1	18686	4920	4308	4308
q2	2019	198	193	193
q3	10539	1144	1111	1111
q4	10239	745	735	735
q5	7501	2675	2587	2587
q6	220	135	132	132
q7	955	597	608	597
q8	9222	2105	2081	2081
q9	8750	6531	6501	6501
q10	9020	3686	3711	3686
q11	445	240	229	229
q12	475	225	228	225
q13	18164	2974	2970	2970
q14	253	227	210	210
q15	530	476	474	474
q16	502	376	373	373
q17	979	622	670	622
q18	8064	7361	7425	7361
q19	3724	1450	1480	1450
q20	644	310	335	310
q21	5108	3148	3334	3148
q22	393	348	337	337
Total cold run time: 116432 ms
Total hot run time: 39640 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4358	4258	4214	4214
q2	374	281	276	276
q3	2971	2778	2853	2778
q4	1958	1715	1709	1709
q5	5677	5523	5488	5488
q6	224	125	133	125
q7	2164	1781	1889	1781
q8	3254	3417	3425	3417
q9	8728	8645	8796	8645
q10	4088	3955	3707	3707
q11	590	502	514	502
q12	823	646	676	646
q13	16089	3188	3128	3128
q14	320	282	269	269
q15	561	500	494	494
q16	498	434	455	434
q17	1839	1528	1514	1514
q18	8014	8171	7780	7780
q19	5164	1611	1713	1611
q20	3036	1873	1850	1850
q21	9444	4904	4975	4904
q22	638	558	565	558
Total cold run time: 80812 ms
Total hot run time: 55830 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 174384 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 7d9f9a636726bc2b51dce8829e74e0b6de44b003, data reload: false

query1	928	383	380	380
query2	6455	2474	2378	2378
query3	6644	210	218	210
query4	18805	17479	17341	17341
query5	3643	457	476	457
query6	264	167	165	165
query7	4595	303	301	301
query8	306	308	293	293
query9	8684	2358	2328	2328
query10	576	298	283	283
query11	10492	9971	9954	9954
query12	118	89	90	89
query13	1642	366	369	366
query14	10044	7659	7460	7460
query15	230	190	193	190
query16	7765	277	303	277
query17	1865	566	556	556
query18	1983	285	279	279
query19	225	158	161	158
query20	92	85	81	81
query21	212	126	121	121
query22	4301	4013	4050	4013
query23	33959	33627	33496	33496
query24	10679	2816	2982	2816
query25	620	413	397	397
query26	720	157	163	157
query27	2322	321	324	321
query28	6262	2124	2134	2124
query29	927	650	662	650
query30	276	163	165	163
query31	972	774	752	752
query32	102	59	59	59
query33	663	298	311	298
query34	914	476	498	476
query35	763	656	661	656
query36	1141	978	943	943
query37	147	81	77	77
query38	2976	2893	2789	2789
query39	894	834	806	806
query40	216	129	126	126
query41	57	53	54	53
query42	122	98	106	98
query43	601	564	559	559
query44	1091	747	740	740
query45	197	172	172	172
query46	1072	745	709	709
query47	1891	1778	1857	1778
query48	375	308	318	308
query49	836	396	410	396
query50	775	386	379	379
query51	6922	6695	6713	6695
query52	105	90	90	90
query53	352	288	286	286
query54	876	436	437	436
query55	74	72	73	72
query56	276	254	279	254
query57	1136	1056	1050	1050
query58	236	255	236	236
query59	3396	3180	3216	3180
query60	309	279	268	268
query61	91	94	90	90
query62	641	435	448	435
query63	312	287	291	287
query64	8482	2269	1734	1734
query65	3159	3101	3139	3101
query66	743	314	326	314
query67	15740	15024	14873	14873
query68	8892	547	546	546
query69	764	437	407	407
query70	1494	1114	1114	1114
query71	529	276	273	273
query72	8777	5676	5491	5491
query73	2295	327	324	324
query74	5862	5455	5467	5455
query75	5587	2661	2694	2661
query76	5454	969	984	969
query77	786	302	303	302
query78	10546	9814	9742	9742
query79	7681	526	519	519
query80	892	465	501	465
query81	554	221	223	221
query82	256	103	105	103
query83	341	174	170	170
query84	271	90	86	86
query85	876	277	268	268
query86	355	324	329	324
query87	3299	3072	3090	3072
query88	4740	2470	2465	2465
query89	528	373	379	373
query90	1932	198	189	189
query91	125	98	99	98
query92	59	49	91	49
query93	5538	514	500	500
query94	1274	188	183	183
query95	396	310	315	310
query96	613	271	281	271
query97	3160	3015	3082	3015
query98	227	197	197	197
query99	1143	853	843	843
Total cold run time: 289535 ms
Total hot run time: 174384 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.38 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 7d9f9a636726bc2b51dce8829e74e0b6de44b003, data reload: false

query1	0.04	0.04	0.04
query2	0.08	0.04	0.04
query3	0.22	0.05	0.06
query4	1.67	0.08	0.07
query5	0.48	0.50	0.48
query6	1.18	0.73	0.73
query7	0.02	0.02	0.01
query8	0.06	0.05	0.04
query9	0.54	0.48	0.49
query10	0.56	0.54	0.53
query11	0.15	0.11	0.11
query12	0.15	0.12	0.12
query13	0.58	0.58	0.59
query14	0.77	0.80	0.81
query15	0.84	0.82	0.82
query16	0.37	0.35	0.36
query17	0.98	0.94	1.04
query18	0.22	0.23	0.24
query19	1.89	1.68	1.70
query20	0.01	0.01	0.01
query21	15.42	0.75	0.67
query22	4.19	7.13	1.91
query23	18.28	1.39	1.25
query24	2.04	0.23	0.22
query25	0.15	0.09	0.08
query26	0.26	0.16	0.17
query27	0.07	0.07	0.08
query28	13.25	1.03	1.00
query29	12.67	3.28	3.28
query30	0.26	0.06	0.06
query31	2.89	0.40	0.38
query32	3.25	0.47	0.49
query33	2.90	2.86	2.90
query34	17.15	4.44	4.37
query35	4.61	4.52	4.41
query36	0.66	0.50	0.47
query37	0.19	0.17	0.16
query38	0.16	0.14	0.15
query39	0.04	0.04	0.03
query40	0.16	0.15	0.14
query41	0.10	0.05	0.05
query42	0.06	0.05	0.05
query43	0.05	0.04	0.04
Total cold run time: 109.62 s
Total hot run time: 30.38 s

xiaokang
xiaokang reviewed Jun 30, 2024
View reviewed changes
be/src/runtime/load_stream.cpp Show resolved Hide resolved
be/src/runtime/load_stream_writer.cpp Outdated Show resolved Hide resolved
be/src/util/thrift_util.cpp Show resolved Hide resolved
be/src/olap/rowset/rowset_writer_context.h Outdated Show resolved Hide resolved
be/src/olap/rowset/beta_rowset_writer.cpp Outdated Show resolved Hide resolved
be/src/olap/rowset/segment_v2/inverted_index_fs_directory.cpp Show resolved Hide resolved
be/src/olap/rowset/segment_v2/inverted_index_fs_directory.cpp Show resolved Hide resolved
@github-actions github-actions bot removed the approved Indicates a PR has been approved by one committer. label Jun 30, 2024
@csun5285
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 40169 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit a26e9de7f3ccd81080a0177fa81bc5c81608bbb6, data reload: false

------ Round 1 ----------------------------------
q1	18246	4527	4369	4369
q2	2017	195	194	194
q3	10547	1229	1115	1115
q4	10228	793	805	793
q5	7490	2671	2672	2671
q6	223	141	142	141
q7	961	592	610	592
q8	9235	2105	2077	2077
q9	8997	6548	6502	6502
q10	8934	3718	3727	3718
q11	467	245	245	245
q12	416	243	243	243
q13	17785	3012	2996	2996
q14	271	230	212	212
q15	538	474	489	474
q16	521	380	391	380
q17	993	703	745	703
q18	8029	7661	7389	7389
q19	6427	1490	1529	1490
q20	678	333	346	333
q21	5041	3198	4038	3198
q22	403	344	334	334
Total cold run time: 118447 ms
Total hot run time: 40169 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4528	4273	4289	4273
q2	365	266	264	264
q3	3119	2902	2968	2902
q4	1944	1783	1780	1780
q5	5502	5548	5448	5448
q6	234	143	149	143
q7	2297	1854	1882	1854
q8	3316	3469	3405	3405
q9	8781	8836	8762	8762
q10	4145	3719	3851	3719
q11	595	513	503	503
q12	829	647	651	647
q13	17228	3178	3218	3178
q14	299	288	269	269
q15	547	490	493	490
q16	502	440	447	440
q17	1852	1551	1531	1531
q18	8114	7976	7813	7813
q19	1817	1619	1596	1596
q20	2188	1901	1859	1859
q21	10610	4943	4774	4774
q22	644	577	577	577
Total cold run time: 79456 ms
Total hot run time: 56227 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 174428 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit a26e9de7f3ccd81080a0177fa81bc5c81608bbb6, data reload: false

query1	928	382	381	381
query2	6471	2495	2357	2357
query3	6637	208	218	208
query4	19431	17488	17589	17488
query5	3718	489	506	489
query6	269	186	165	165
query7	4594	306	299	299
query8	323	299	298	298
query9	8436	2424	2406	2406
query10	580	319	302	302
query11	10648	10013	9939	9939
query12	114	90	87	87
query13	1661	374	382	374
query14	9616	8125	6875	6875
query15	254	195	198	195
query16	7692	280	270	270
query17	1882	578	552	552
query18	1482	285	282	282
query19	205	158	167	158
query20	90	84	86	84
query21	211	141	133	133
query22	4220	3942	3964	3942
query23	34021	33748	33788	33748
query24	10931	2912	2956	2912
query25	611	426	421	421
query26	710	164	169	164
query27	2262	326	338	326
query28	6120	2165	2171	2165
query29	914	695	688	688
query30	231	160	155	155
query31	971	760	793	760
query32	97	63	60	60
query33	727	330	304	304
query34	941	498	482	482
query35	786	703	697	697
query36	1142	953	977	953
query37	138	107	81	81
query38	2986	2841	2810	2810
query39	881	790	803	790
query40	214	133	132	132
query41	57	53	54	53
query42	130	105	103	103
query43	607	571	566	566
query44	1212	736	745	736
query45	194	165	170	165
query46	1092	754	735	735
query47	1868	1781	1745	1745
query48	376	298	303	298
query49	861	525	420	420
query50	781	390	388	388
query51	6866	6680	6782	6680
query52	103	98	99	98
query53	363	297	302	297
query54	898	462	450	450
query55	79	73	72	72
query56	314	269	267	267
query57	1151	1029	1049	1029
query58	260	250	277	250
query59	3514	3146	3221	3146
query60	336	292	279	279
query61	92	90	89	89
query62	594	433	448	433
query63	325	292	302	292
query64	8518	2278	1759	1759
query65	3208	3100	3142	3100
query66	754	336	329	329
query67	15499	14947	14906	14906
query68	8812	557	557	557
query69	783	493	345	345
query70	1449	1137	1069	1069
query71	524	287	276	276
query72	9460	5791	5402	5402
query73	2237	335	328	328
query74	5907	5493	5501	5493
query75	5378	2632	2690	2632
query76	5245	956	968	956
query77	850	321	312	312
query78	10488	9967	9784	9784
query79	7668	513	528	513
query80	879	465	468	465
query81	547	222	225	222
query82	261	108	107	107
query83	326	174	172	172
query84	278	85	88	85
query85	886	289	278	278
query86	351	306	319	306
query87	3283	3075	3072	3072
query88	4896	2383	2409	2383
query89	513	375	377	375
query90	2027	197	188	188
query91	129	99	98	98
query92	62	50	51	50
query93	5606	527	515	515
query94	1367	189	189	189
query95	400	318	321	318
query96	608	270	269	269
query97	3249	3038	3015	3015
query98	218	202	200	200
query99	1116	837	846	837
Total cold run time: 290000 ms
Total hot run time: 174428 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.96 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit a26e9de7f3ccd81080a0177fa81bc5c81608bbb6, data reload: false

query1	0.03	0.03	0.03
query2	0.08	0.04	0.04
query3	0.22	0.05	0.05
query4	1.66	0.09	0.10
query5	0.49	0.50	0.49
query6	1.13	0.73	0.72
query7	0.02	0.02	0.01
query8	0.05	0.04	0.04
query9	0.55	0.49	0.49
query10	0.55	0.55	0.55
query11	0.16	0.11	0.11
query12	0.16	0.13	0.13
query13	0.59	0.59	0.58
query14	0.78	0.81	0.77
query15	0.85	0.82	0.81
query16	0.36	0.36	0.35
query17	0.96	0.96	0.95
query18	0.21	0.23	0.24
query19	1.78	1.71	1.73
query20	0.02	0.01	0.01
query21	15.44	0.73	0.64
query22	4.01	6.88	2.27
query23	18.27	1.44	1.28
query24	2.18	0.21	0.20
query25	0.16	0.08	0.08
query26	0.27	0.17	0.17
query27	0.08	0.08	0.08
query28	13.24	1.02	0.99
query29	12.59	3.32	3.32
query30	0.25	0.07	0.05
query31	2.86	0.39	0.39
query32	3.27	0.47	0.47
query33	2.86	2.86	2.95
query34	17.14	4.45	4.51
query35	4.49	4.51	4.55
query36	0.66	0.46	0.46
query37	0.18	0.15	0.15
query38	0.15	0.16	0.14
query39	0.05	0.03	0.04
query40	0.16	0.14	0.13
query41	0.09	0.04	0.04
query42	0.06	0.05	0.05
query43	0.05	0.04	0.04
Total cold run time: 109.16 s
Total hot run time: 30.96 s

xiaokang
xiaokang approved these changes Jul 2, 2024
View reviewed changes
Copy link
Contributor

@xiaokang xiaokang left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jul 2, 2024
@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Jul 2, 2024

PR approved by at least one committer and no changes requested.

kaijchen
kaijchen approved these changes Jul 4, 2024
View reviewed changes
Copy link
Contributor

@kaijchen kaijchen left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@xiaokang xiaokang merged commit bf51d5b into apache:master Jul 5, 2024
30 of 38 checks passed
@csun5285 csun5285 mentioned this pull request Jul 17, 2024
Merged
airborne12 pushed a commit that referenced this pull request Jul 17, 2024
@csun5285
[Pick 3.0](move-memtable) support for inverted index file (#37975)
a925586 
## Proposed changes

pick from master #35891
pick from master #37621
dataroaring pushed a commit that referenced this pull request Jul 17, 2024
@csun5285 @dataroaring
[feature](move-memtable) support for inverted index file (#35891)
c8db63a 
support for inverted index file in move-memtable
@gavinchou gavinchou mentioned this pull request Aug 19, 2024
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@github-actions github-actions[bot] github-actions[bot] left review comments

@xiaokang xiaokang xiaokang approved these changes

@kaijchen kaijchen kaijchen approved these changes

@airborne12 airborne12 airborne12 left review comments

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by one committer. dev/3.0.0-merged dev/3.0.1-merged meta-change reviewed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
6 participants
@csun5285 @doris-robot @xiaokang @airborne12 @kaijchen @dataroaring