Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[feature](index compaction)support index compaction in cloud mode #33168

Merged
merged 5 commits into from
Apr 4, 2024
Merged

[feature](index compaction)support index compaction in cloud mode #33168

merged 5 commits into from
Apr 4, 2024

Conversation

qidaye
Copy link
Contributor

@qidaye qidaye commented Apr 2, 2024 •
edited

Proposed changes

  1. Add index compaction step in cloud cumulative compaction flows
  2. Move construct_skip_inverted_index,do_inverted_index_compaction function from CompactionMixin to Compaction class
  3. Add cloudMode check in regression cases. Since cloud compaction dose not compact 0-1 rowset.

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@qidaye
Copy link
Contributor Author

qidaye commented Apr 2, 2024

run buildall

github-actions[bot]
github-actions bot reviewed Apr 2, 2024
View reviewed changes
Copy link
Contributor

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

clang-tidy made some suggestions

be/src/olap/compaction.cpp Show resolved Hide resolved
be/src/olap/compaction.cpp Show resolved Hide resolved
@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 35.64% (8879/24911)
Line Coverage: 27.38% (72905/266309)
Region Coverage: 26.56% (37709/141993)
Branch Coverage: 23.34% (19216/82318)
Coverage Report: http://coverage.selectdb-in.cc/coverage/eeec2216671a669442ceb570715b75f5f0a3b705_eeec2216671a669442ceb570715b75f5f0a3b705/report/index.html

@doris-robot
Copy link

TPC-H: Total hot run time: 38598 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit eeec2216671a669442ceb570715b75f5f0a3b705, data reload: false

------ Round 1 ----------------------------------
q1	17615	4073	4067	4067
q2	2012	193	185	185
q3	10476	1234	1391	1234
q4	10207	886	968	886
q5	7474	2950	2896	2896
q6	219	135	135	135
q7	1106	628	610	610
q8	9400	2039	2020	2020
q9	6726	6172	6101	6101
q10	8435	3503	3522	3503
q11	416	249	255	249
q12	390	222	220	220
q13	17786	2879	2916	2879
q14	268	242	240	240
q15	532	492	489	489
q16	512	386	378	378
q17	952	927	891	891
q18	7236	6374	6441	6374
q19	1624	1537	1524	1524
q20	596	322	327	322
q21	3558	3099	3085	3085
q22	373	310	310	310
Total cold run time: 107913 ms
Total hot run time: 38598 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4204	4014	4057	4014
q2	339	218	218	218
q3	2980	2973	2948	2948
q4	1872	1840	1828	1828
q5	5234	5217	5222	5217
q6	208	126	125	125
q7	2238	1757	1801	1757
q8	3209	3278	3278	3278
q9	8458	8487	8469	8469
q10	3746	3961	4012	3961
q11	562	478	479	478
q12	747	595	582	582
q13	16822	3126	3100	3100
q14	294	278	275	275
q15	540	495	521	495
q16	510	470	472	470
q17	1783	1758	1743	1743
q18	8202	7683	7817	7683
q19	1686	1676	1663	1663
q20	2066	1826	1871	1826
q21	5174	4984	4984	4984
q22	534	444	417	417
Total cold run time: 71408 ms
Total hot run time: 55531 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 182752 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit eeec2216671a669442ceb570715b75f5f0a3b705, data reload: false

query1	1255	1115	1115	1115
query2	6341	2001	2110	2001
query3	6666	220	218	218
query4	24694	21750	21385	21385
query5	4256	405	411	405
query6	277	192	185	185
query7	4599	316	302	302
query8	234	165	186	165
query9	8486	2310	2333	2310
query10	464	256	266	256
query11	14990	14444	14395	14395
query12	145	102	96	96
query13	1632	387	391	387
query14	8542	7017	6959	6959
query15	224	186	185	185
query16	6855	290	285	285
query17	965	647	584	584
query18	1856	295	299	295
query19	207	169	172	169
query20	101	95	95	95
query21	202	138	133	133
query22	4936	4845	4765	4765
query23	33713	32860	32897	32860
query24	13344	3251	3183	3183
query25	719	449	459	449
query26	1862	180	178	178
query27	3183	385	372	372
query28	7178	1926	1904	1904
query29	1329	666	628	628
query30	323	176	174	174
query31	1010	764	794	764
query32	102	63	64	63
query33	740	269	268	268
query34	1110	529	526	526
query35	923	737	773	737
query36	987	871	887	871
query37	267	88	90	88
query38	3750	3662	3598	3598
query39	1066	1033	1084	1033
query40	239	154	144	144
query41	53	47	51	47
query42	117	110	121	110
query43	473	414	414	414
query44	1125	730	723	723
query45	285	279	287	279
query46	1136	877	818	818
query47	1964	1886	1903	1886
query48	398	321	323	321
query49	950	362	375	362
query50	838	434	433	433
query51	7193	6902	6919	6902
query52	117	108	107	107
query53	383	303	305	303
query54	308	252	251	251
query55	90	84	84	84
query56	253	233	239	233
query57	1279	1180	1189	1180
query58	249	236	234	234
query59	2863	2540	2504	2504
query60	258	253	245	245
query61	93	96	92	92
query62	671	458	449	449
query63	319	289	294	289
query64	5820	3098	3100	3098
query65	3049	3045	3027	3027
query66	1302	316	310	310
query67	15382	14828	14840	14828
query68	8478	585	597	585
query69	568	333	339	333
query70	1226	1098	1116	1098
query71	499	277	272	272
query72	6253	2591	2445	2445
query73	802	335	333	333
query74	6767	6311	6309	6309
query75	3481	2294	2354	2294
query76	5131	1041	1233	1041
query77	574	259	260	259
query78	10964	10165	10205	10165
query79	8597	542	528	528
query80	1285	454	445	445
query81	500	238	237	237
query82	725	110	104	104
query83	201	168	162	162
query84	275	89	89	89
query85	1322	296	294	294
query86	407	276	271	271
query87	3677	3510	3483	3483
query88	4016	2357	2371	2357
query89	550	374	380	374
query90	1938	179	188	179
query91	135	108	108	108
query92	62	51	52	51
query93	6940	532	524	524
query94	1024	203	192	192
query95	448	343	344	343
query96	614	277	274	274
query97	2670	2522	2505	2505
query98	233	215	215	215
query99	1283	824	828	824
Total cold run time: 298385 ms
Total hot run time: 182752 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.22 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit eeec2216671a669442ceb570715b75f5f0a3b705, data reload: false

query1	0.04	0.03	0.03
query2	0.07	0.04	0.04
query3	0.23	0.05	0.04
query4	1.68	0.06	0.06
query5	0.48	0.47	0.48
query6	1.14	0.65	0.66
query7	0.02	0.01	0.01
query8	0.05	0.04	0.05
query9	0.58	0.51	0.51
query10	0.56	0.58	0.57
query11	0.15	0.12	0.11
query12	0.14	0.11	0.12
query13	0.61	0.59	0.60
query14	0.78	0.78	0.79
query15	0.86	0.85	0.84
query16	0.35	0.35	0.36
query17	0.98	0.96	0.96
query18	0.25	0.26	0.25
query19	1.80	1.71	1.76
query20	0.01	0.00	0.00
query21	15.40	0.75	0.72
query22	2.86	5.50	1.79
query23	18.07	1.24	1.16
query24	1.33	0.21	0.40
query25	0.14	0.09	0.08
query26	0.30	0.17	0.20
query27	0.08	0.08	0.09
query28	13.51	0.95	0.93
query29	12.67	3.61	3.38
query30	0.26	0.06	0.07
query31	2.84	0.39	0.39
query32	3.28	0.49	0.47
query33	2.91	2.85	2.86
query34	15.48	4.34	4.33
query35	4.36	4.39	4.40
query36	0.68	0.46	0.47
query37	0.18	0.17	0.16
query38	0.17	0.16	0.17
query39	0.04	0.04	0.03
query40	0.17	0.14	0.16
query41	0.10	0.06	0.05
query42	0.06	0.05	0.05
query43	0.05	0.04	0.04
Total cold run time: 105.72 s
Total hot run time: 30.22 s

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit eeec2216671a669442ceb570715b75f5f0a3b705 with default session variables
Stream load json:         19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc:          59 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       16.1 seconds inserted 10000000 Rows, about 621K ops/s

platoneko
platoneko reviewed Apr 2, 2024
View reviewed changes
be/src/olap/compaction.h Outdated Show resolved Hide resolved
platoneko
platoneko requested changes Apr 2, 2024
View reviewed changes
Copy link
Contributor

@platoneko platoneko left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

在 Compaction 这个公共基类做了 Tablet 类型强转，会有未定义错误

@qidaye qidaye requested a review from platoneko April 3, 2024 03:09
@qidaye
Copy link
Contributor Author

qidaye commented Apr 3, 2024

run buildall

github-actions[bot]
github-actions bot reviewed Apr 3, 2024
View reviewed changes
Copy link
Contributor

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

clang-tidy made some suggestions

be/src/olap/base_tablet.h Outdated Show resolved Hide resolved
be/src/olap/compaction.cpp Show resolved Hide resolved
be/src/olap/compaction.cpp Show resolved Hide resolved
@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 35.64% (8880/24918)
Line Coverage: 27.37% (72911/266404)
Region Coverage: 26.54% (37702/142058)
Branch Coverage: 23.34% (19217/82344)
Coverage Report: http://coverage.selectdb-in.cc/coverage/acb2ef7c7ce1e97d2bc4df2c7850b46e99f5690a_acb2ef7c7ce1e97d2bc4df2c7850b46e99f5690a/report/index.html

@doris-robot
Copy link

TPC-H: Total hot run time: 38709 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit acb2ef7c7ce1e97d2bc4df2c7850b46e99f5690a, data reload: false

------ Round 1 ----------------------------------
q1	17633	4145	4146	4145
q2	2008	192	182	182
q3	10470	1227	1417	1227
q4	10204	856	988	856
q5	7462	3016	2951	2951
q6	218	135	134	134
q7	1114	642	613	613
q8	9399	2062	2047	2047
q9	6642	6177	6121	6121
q10	8481	3509	3484	3484
q11	417	237	241	237
q12	391	233	220	220
q13	17776	2890	2892	2890
q14	276	240	243	240
q15	530	487	490	487
q16	506	386	376	376
q17	966	919	895	895
q18	7276	6472	6376	6376
q19	1610	1559	1533	1533
q20	603	327	308	308
q21	3597	3177	3086	3086
q22	369	302	301	301
Total cold run time: 107948 ms
Total hot run time: 38709 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4091	4053	4046	4046
q2	326	212	217	212
q3	2968	2958	2960	2958
q4	1909	1844	1823	1823
q5	5284	5253	5236	5236
q6	209	125	127	125
q7	2241	1793	1775	1775
q8	3245	3310	3313	3310
q9	8558	8503	8546	8503
q10	3764	4000	4024	4000
q11	567	464	470	464
q12	771	598	641	598
q13	12853	3033	3091	3033
q14	316	272	268	268
q15	543	488	493	488
q16	512	449	458	449
q17	1782	1762	1691	1691
q18	8279	7785	7963	7785
q19	1915	1692	1692	1692
q20	2053	1815	1865	1815
q21	5198	4949	5025	4949
q22	523	473	449	449
Total cold run time: 67907 ms
Total hot run time: 55669 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 182609 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit acb2ef7c7ce1e97d2bc4df2c7850b46e99f5690a, data reload: false

query1	1217	1118	1118	1118
query2	6354	2031	1946	1946
query3	6657	219	208	208
query4	24829	21411	21460	21411
query5	4172	394	411	394
query6	278	189	188	188
query7	4603	315	311	311
query8	227	176	176	176
query9	8431	2296	2296	2296
query10	454	269	254	254
query11	15010	14500	14490	14490
query12	145	96	96	96
query13	1635	393	389	389
query14	8478	6992	6851	6851
query15	232	178	182	178
query16	6861	286	281	281
query17	968	619	579	579
query18	1852	290	285	285
query19	208	163	167	163
query20	102	93	94	93
query21	196	131	131	131
query22	4952	4815	4698	4698
query23	33381	32683	32557	32557
query24	12754	3214	3212	3212
query25	728	437	440	437
query26	1936	174	177	174
query27	3336	385	417	385
query28	6994	1952	1906	1906
query29	1384	642	636	636
query30	324	172	181	172
query31	1026	785	779	779
query32	99	66	65	65
query33	738	271	262	262
query34	1426	548	540	540
query35	872	750	736	736
query36	1004	884	890	884
query37	239	88	83	83
query38	3687	3620	3588	3588
query39	1653	1615	1618	1615
query40	240	148	150	148
query41	48	48	50	48
query42	121	118	118	118
query43	456	413	410	410
query44	1128	740	729	729
query45	298	283	274	274
query46	1102	856	813	813
query47	1978	1874	1880	1874
query48	403	317	324	317
query49	959	374	388	374
query50	855	423	425	423
query51	7025	6870	6951	6870
query52	108	98	101	98
query53	368	298	296	296
query54	308	235	254	235
query55	92	82	82	82
query56	239	222	233	222
query57	1295	1204	1164	1164
query58	250	239	242	239
query59	2746	2252	2320	2252
query60	273	243	240	240
query61	91	87	88	87
query62	677	452	485	452
query63	312	284	288	284
query64	5825	3089	3398	3089
query65	3036	2994	2973	2973
query66	1300	340	317	317
query67	15362	15003	15046	15003
query68	9551	583	598	583
query69	608	328	329	328
query70	1394	1130	1132	1130
query71	540	276	270	270
query72	6436	2577	2414	2414
query73	1570	329	326	326
query74	6772	6263	6397	6263
query75	3807	2292	2301	2292
query76	5819	1143	1209	1143
query77	562	252	255	252
query78	10884	10098	10034	10034
query79	10521	559	549	549
query80	1958	442	437	437
query81	527	245	237	237
query82	463	110	106	106
query83	209	164	165	164
query84	269	90	88	88
query85	964	321	305	305
query86	356	281	295	281
query87	3744	3527	3488	3488
query88	3640	2359	2373	2359
query89	557	368	374	368
query90	1994	178	178	178
query91	134	104	108	104
query92	61	50	53	50
query93	6835	544	538	538
query94	1202	190	190	190
query95	439	324	335	324
query96	603	277	274	274
query97	2708	2510	2494	2494
query98	227	221	214	214
query99	1266	828	828	828
Total cold run time: 302957 ms
Total hot run time: 182609 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.86 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit acb2ef7c7ce1e97d2bc4df2c7850b46e99f5690a, data reload: false

query1	0.04	0.04	0.04
query2	0.07	0.04	0.04
query3	0.24	0.04	0.05
query4	1.68	0.07	0.07
query5	0.48	0.48	0.49
query6	1.12	0.65	0.65
query7	0.02	0.02	0.01
query8	0.04	0.04	0.04
query9	0.56	0.53	0.51
query10	0.57	0.57	0.55
query11	0.16	0.12	0.11
query12	0.15	0.12	0.12
query13	0.61	0.60	0.60
query14	0.76	0.80	0.79
query15	0.85	0.83	0.85
query16	0.35	0.35	0.34
query17	0.99	0.98	0.98
query18	0.24	0.26	0.25
query19	1.79	1.70	1.75
query20	0.01	0.02	0.01
query21	15.41	0.72	0.71
query22	3.16	5.47	2.47
query23	17.71	1.37	1.08
query24	1.39	0.22	0.23
query25	0.14	0.09	0.09
query26	0.28	0.18	0.18
query27	0.08	0.08	0.08
query28	13.90	0.95	0.93
query29	12.53	3.38	3.37
query30	0.26	0.06	0.06
query31	2.86	0.40	0.38
query32	3.27	0.48	0.48
query33	2.83	2.86	2.87
query34	15.48	4.33	4.32
query35	4.40	4.39	4.36
query36	0.67	0.47	0.47
query37	0.19	0.18	0.17
query38	0.17	0.15	0.16
query39	0.05	0.04	0.04
query40	0.17	0.15	0.15
query41	0.10	0.05	0.05
query42	0.07	0.04	0.06
query43	0.04	0.04	0.04
Total cold run time: 105.89 s
Total hot run time: 30.86 s

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit acb2ef7c7ce1e97d2bc4df2c7850b46e99f5690a with default session variables
Stream load json:         18 seconds loaded 2358488459 Bytes, about 124 MB/s
Stream load orc:          59 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet:      32 seconds loaded 861443392 Bytes, about 25 MB/s
Insert into select:       17.1 seconds inserted 10000000 Rows, about 584K ops/s

@qidaye
Copy link
Contributor Author

qidaye commented Apr 3, 2024

run buildall

github-actions[bot]
github-actions bot reviewed Apr 3, 2024
View reviewed changes
Copy link
Contributor

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

clang-tidy made some suggestions

be/src/olap/compaction.cpp Show resolved Hide resolved
be/src/olap/compaction.cpp Show resolved Hide resolved
@doris-robot
Copy link

TPC-H: Total hot run time: 39045 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 9dbd7e4d9e316d63f910ee9cee01062f5cadbe31, data reload: false

------ Round 1 ----------------------------------
q1	18223	4258	4222	4222
q2	2702	194	202	194
q3	10898	1212	1424	1212
q4	10205	846	989	846
q5	7463	3017	2947	2947
q6	219	136	135	135
q7	1130	630	618	618
q8	9405	2045	2052	2045
q9	6671	6245	6242	6242
q10	8424	3550	3507	3507
q11	423	242	233	233
q12	386	225	221	221
q13	17764	2890	2915	2890
q14	272	242	244	242
q15	532	488	471	471
q16	505	391	387	387
q17	969	921	915	915
q18	7277	6491	6458	6458
q19	1613	1539	1551	1539
q20	610	335	310	310
q21	3582	3101	3202	3101
q22	370	313	310	310
Total cold run time: 109643 ms
Total hot run time: 39045 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4100	4065	4112	4065
q2	328	219	219	219
q3	2999	2991	2937	2937
q4	1904	1867	1847	1847
q5	5272	5232	5237	5232
q6	207	128	128	128
q7	2265	1827	1858	1827
q8	3267	3326	3328	3326
q9	8528	8531	8524	8524
q10	3778	3837	3836	3836
q11	551	464	458	458
q12	716	569	534	534
q13	12630	2926	2934	2926
q14	302	264	265	264
q15	517	470	479	470
q16	463	413	399	399
q17	1733	1671	1691	1671
q18	7722	7373	7220	7220
q19	1645	1655	1653	1653
q20	1942	1741	1724	1724
q21	5066	4805	4797	4797
q22	488	427	441	427
Total cold run time: 66423 ms
Total hot run time: 54484 ms

@qidaye
Copy link
Contributor Author

qidaye commented Apr 3, 2024

run buildall

github-actions[bot]
github-actions bot reviewed Apr 3, 2024
View reviewed changes
Copy link
Contributor

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

clang-tidy made some suggestions

be/src/olap/compaction.cpp Show resolved Hide resolved
be/src/olap/compaction.cpp Show resolved Hide resolved
@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 35.65% (8883/24915)
Line Coverage: 27.37% (72913/266354)
Region Coverage: 26.55% (37711/142022)
Branch Coverage: 23.35% (19212/82290)
Coverage Report: http://coverage.selectdb-in.cc/coverage/c17947f2c10b4461a6a1b817942370ed682842f2_c17947f2c10b4461a6a1b817942370ed682842f2/report/index.html

@doris-robot
Copy link

TPC-H: Total hot run time: 39229 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit c17947f2c10b4461a6a1b817942370ed682842f2, data reload: false

------ Round 1 ----------------------------------
q1	17937	4230	4215	4215
q2	2335	198	192	192
q3	10744	1363	1447	1363
q4	10845	931	1001	931
q5	7788	2979	2938	2938
q6	218	133	143	133
q7	1125	620	610	610
q8	9392	2080	2043	2043
q9	6767	6223	6231	6223
q10	8437	3538	3524	3524
q11	416	239	229	229
q12	386	219	215	215
q13	17776	2909	2899	2899
q14	271	246	252	246
q15	533	501	478	478
q16	492	388	382	382
q17	958	931	935	931
q18	7412	6484	6458	6458
q19	1616	1553	1545	1545
q20	616	328	302	302
q21	3605	3064	3087	3064
q22	378	309	308	308
Total cold run time: 110047 ms
Total hot run time: 39229 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4083	4068	4076	4068
q2	328	220	223	220
q3	2955	2963	2960	2960
q4	1906	1847	1837	1837
q5	5240	5257	5229	5229
q6	211	128	125	125
q7	2244	1806	1794	1794
q8	3244	3286	3322	3286
q9	8448	8480	8476	8476
q10	3767	3807	3833	3807
q11	535	451	447	447
q12	719	553	538	538
q13	16807	2890	2874	2874
q14	292	267	278	267
q15	510	463	479	463
q16	450	392	426	392
q17	1722	1701	1677	1677
q18	7669	7302	7279	7279
q19	1646	1643	1652	1643
q20	1950	1715	1728	1715
q21	4993	4739	4774	4739
q22	486	433	430	430
Total cold run time: 70205 ms
Total hot run time: 54266 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 182523 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit c17947f2c10b4461a6a1b817942370ed682842f2, data reload: false

query1	1250	1122	1124	1122
query2	6487	1890	1840	1840
query3	6655	217	211	211
query4	24096	21355	21386	21355
query5	4169	406	414	406
query6	285	184	185	184
query7	4591	302	290	290
query8	238	178	180	178
query9	8453	2241	2261	2241
query10	581	262	252	252
query11	14843	14594	14431	14431
query12	150	99	97	97
query13	1645	390	392	390
query14	8569	6820	6829	6820
query15	208	184	177	177
query16	7162	275	273	273
query17	1005	613	590	590
query18	1914	292	292	292
query19	212	161	166	161
query20	98	96	95	95
query21	199	142	125	125
query22	5089	4852	4919	4852
query23	33846	33145	32968	32968
query24	12636	3204	3101	3101
query25	680	385	388	385
query26	1892	169	156	156
query27	3017	326	338	326
query28	6774	1828	1824	1824
query29	1392	585	578	578
query30	305	164	171	164
query31	987	717	715	715
query32	96	61	60	60
query33	690	251	232	232
query34	1064	492	529	492
query35	844	697	702	697
query36	976	876	873	873
query37	282	75	75	75
query38	3556	3378	3406	3378
query39	1573	1548	1546	1546
query40	306	129	133	129
query41	47	44	45	44
query42	116	101	103	101
query43	440	411	398	398
query44	1093	701	710	701
query45	285	263	260	260
query46	1080	823	787	787
query47	1908	1824	1822	1822
query48	377	303	304	303
query49	1172	368	367	367
query50	798	392	394	392
query51	6516	6620	6574	6574
query52	113	96	104	96
query53	359	291	286	286
query54	328	244	253	244
query55	91	78	84	78
query56	244	225	226	225
query57	1254	1125	1126	1125
query58	250	227	229	227
query59	2533	2448	2417	2417
query60	269	241	257	241
query61	112	109	108	108
query62	712	446	453	446
query63	316	284	291	284
query64	6490	3393	3163	3163
query65	3072	3050	3006	3006
query66	1461	333	333	333
query67	15520	15094	15390	15094
query68	5341	568	574	568
query69	544	350	329	329
query70	1165	1099	1092	1092
query71	432	296	280	280
query72	6494	2737	2479	2479
query73	726	324	324	324
query74	6850	6378	6425	6378
query75	3091	2285	2319	2285
query76	3639	1122	1228	1122
query77	566	252	264	252
query78	10992	10324	10000	10000
query79	8685	548	530	530
query80	1770	434	423	423
query81	521	247	238	238
query82	1378	109	108	108
query83	303	159	161	159
query84	269	90	90	90
query85	1622	292	294	292
query86	469	278	283	278
query87	3661	3495	3498	3495
query88	4380	2303	2304	2303
query89	581	379	373	373
query90	1985	178	187	178
query91	138	108	105	105
query92	63	57	54	54
query93	7174	538	529	529
query94	1212	186	188	186
query95	1097	1094	1089	1089
query96	619	271	269	269
query97	2667	2463	2478	2463
query98	239	219	216	216
query99	1285	833	842	833
Total cold run time: 295729 ms
Total hot run time: 182523 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.59 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit c17947f2c10b4461a6a1b817942370ed682842f2, data reload: false

query1	0.03	0.04	0.03
query2	0.07	0.05	0.04
query3	0.23	0.05	0.05
query4	1.74	0.07	0.07
query5	0.48	0.49	0.49
query6	1.13	0.65	0.65
query7	0.02	0.01	0.01
query8	0.05	0.05	0.04
query9	0.59	0.50	0.51
query10	0.55	0.57	0.56
query11	0.14	0.11	0.11
query12	0.14	0.11	0.11
query13	0.61	0.60	0.59
query14	0.76	0.79	0.79
query15	0.87	0.84	0.85
query16	0.35	0.34	0.36
query17	0.98	1.01	0.99
query18	0.25	0.25	0.26
query19	1.78	1.73	1.75
query20	0.02	0.01	0.01
query21	15.44	0.75	0.72
query22	3.53	5.19	2.36
query23	17.76	1.21	0.97
query24	1.60	0.23	0.27
query25	0.12	0.09	0.09
query26	0.28	0.17	0.20
query27	0.09	0.09	0.08
query28	13.60	0.96	0.95
query29	12.53	3.24	3.48
query30	0.25	0.06	0.06
query31	2.86	0.38	0.39
query32	3.35	0.47	0.48
query33	2.86	2.85	2.92
query34	15.48	4.35	4.31
query35	4.37	4.38	4.39
query36	0.66	0.48	0.47
query37	0.19	0.18	0.19
query38	0.17	0.16	0.15
query39	0.05	0.04	0.04
query40	0.17	0.14	0.15
query41	0.09	0.05	0.05
query42	0.06	0.06	0.06
query43	0.04	0.04	0.04
Total cold run time: 106.34 s
Total hot run time: 30.59 s

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit c17947f2c10b4461a6a1b817942370ed682842f2 with default session variables
Stream load json:         18 seconds loaded 2358488459 Bytes, about 124 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      32 seconds loaded 861443392 Bytes, about 25 MB/s
Insert into select:       16.4 seconds inserted 10000000 Rows, about 609K ops/s

@qidaye
Copy link
Contributor Author

qidaye commented Apr 3, 2024

run p0

@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Apr 3, 2024

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added approved Indicates a PR has been approved by one committer. reviewed labels Apr 3, 2024
@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Apr 3, 2024

PR approved by anyone and no changes requested.

@qidaye
Copy link
Contributor Author

qidaye commented Apr 4, 2024

run p0

dataroaring
dataroaring approved these changes Apr 4, 2024
View reviewed changes
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@dataroaring dataroaring merged commit 39b5336 into apache:master Apr 4, 2024
27 of 31 checks passed
@qidaye qidaye deleted the cloud_index_compaction branch April 7, 2024 02:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@github-actions github-actions[bot] github-actions left review comments

@luwei16 luwei16 luwei16 approved these changes

@platoneko platoneko platoneko approved these changes

@dataroaring dataroaring dataroaring approved these changes

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by one committer. reviewed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
6 participants
@qidaye @doris-robot @luwei16 @platoneko @dataroaring @luennng