Skip to content

[Fix](compaction) Fix full compaction count info#33735

Merged
zhannngchen merged 1 commit intoapache:masterfrom
Yukang-Lian:Fix_Full_Compaction
Apr 17, 2024
Merged

[Fix](compaction) Fix full compaction count info#33735
zhannngchen merged 1 commit intoapache:masterfrom
Yukang-Lian:Fix_Full_Compaction

Conversation

@Yukang-Lian
Copy link
Collaborator

When doing compaction, meta-service will check cumu compaction count. If it doesn't fit, the compaction fail. This PR fix it.

Proposed changes

Issue Number: close #xxx

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

… If it doesn't fit, the compaction fail. This PR fix it.
@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@Yukang-Lian
Copy link
Collaborator Author

run buildall

@github-actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TPC-H: Total hot run time: 38593 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit f20458b81fafadc8f46431930b74894c6bd5f26f, data reload: false

------ Round 1 ----------------------------------
q1	17611	4270	4299	4270
q2	2011	184	185	184
q3	10468	1223	1219	1219
q4	10823	844	801	801
q5	7582	2739	2692	2692
q6	221	137	133	133
q7	1070	638	575	575
q8	9282	2098	2066	2066
q9	7385	6745	6627	6627
q10	8637	3556	3498	3498
q11	446	239	222	222
q12	434	219	209	209
q13	17769	2920	2914	2914
q14	267	231	231	231
q15	537	488	477	477
q16	530	375	369	369
q17	955	595	693	595
q18	7403	6880	6631	6631
q19	2108	1513	1527	1513
q20	636	328	310	310
q21	3567	2757	2899	2757
q22	371	300	305	300
Total cold run time: 110113 ms
Total hot run time: 38593 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4301	4217	4240	4217
q2	368	288	264	264
q3	3008	2785	2744	2744
q4	1835	1559	1597	1559
q5	5365	5329	5287	5287
q6	208	122	122	122
q7	2239	1853	1876	1853
q8	3201	3372	3364	3364
q9	8570	8562	8539	8539
q10	3931	3774	3735	3735
q11	576	473	487	473
q12	767	595	593	593
q13	16312	2909	2916	2909
q14	303	262	274	262
q15	522	473	476	473
q16	490	414	425	414
q17	1767	1488	1469	1469
q18	7595	7714	7420	7420
q19	1687	1510	1523	1510
q20	1966	1771	1722	1722
q21	4965	4696	4662	4662
q22	543	435	456	435
Total cold run time: 70519 ms
Total hot run time: 54026 ms

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 35.49% (8906/25092)
Line Coverage: 27.21% (73130/268726)
Region Coverage: 26.34% (37805/143548)
Branch Coverage: 23.12% (19268/83332)
Coverage Report: http://coverage.selectdb-in.cc/coverage/f20458b81fafadc8f46431930b74894c6bd5f26f_f20458b81fafadc8f46431930b74894c6bd5f26f/report/index.html

@doris-robot
Copy link

TPC-DS: Total hot run time: 184522 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit f20458b81fafadc8f46431930b74894c6bd5f26f, data reload: false

query1	896	365	359	359
query2	6492	2736	2416	2416
query3	6674	201	208	201
query4	23139	21254	21335	21254
query5	4163	411	421	411
query6	271	175	169	169
query7	4600	290	283	283
query8	222	184	182	182
query9	8385	2360	2350	2350
query10	580	254	254	254
query11	15061	14242	14204	14204
query12	137	91	91	91
query13	1636	380	347	347
query14	10125	7840	7546	7546
query15	297	182	182	182
query16	8241	257	259	257
query17	2003	577	538	538
query18	2127	280	266	266
query19	350	147	149	147
query20	96	83	86	83
query21	197	118	128	118
query22	5043	4916	4855	4855
query23	34278	33159	33112	33112
query24	11512	2902	2877	2877
query25	619	371	379	371
query26	1618	163	152	152
query27	2892	313	317	313
query28	7554	2005	2008	2005
query29	965	594	583	583
query30	320	172	173	172
query31	955	698	740	698
query32	91	52	54	52
query33	739	248	249	248
query34	1042	471	475	471
query35	847	696	728	696
query36	1080	888	911	888
query37	135	68	71	68
query38	3354	3190	3207	3190
query39	1589	1520	1533	1520
query40	272	125	122	122
query41	50	42	42	42
query42	101	92	94	92
query43	567	540	555	540
query44	1227	713	707	707
query45	289	252	257	252
query46	1071	749	734	734
query47	1990	1913	1879	1879
query48	359	297	286	286
query49	1100	366	367	366
query50	760	385	382	382
query51	6671	6622	6635	6622
query52	101	90	85	85
query53	342	285	287	285
query54	298	231	234	231
query55	77	72	75	72
query56	238	224	248	224
query57	1216	1124	1147	1124
query58	224	194	197	194
query59	3216	3093	3108	3093
query60	254	231	236	231
query61	93	90	90	90
query62	664	430	446	430
query63	303	275	271	271
query64	6274	4013	3879	3879
query65	3093	3029	3003	3003
query66	1390	359	356	356
query67	15575	14975	15218	14975
query68	5603	546	547	546
query69	522	308	309	308
query70	1271	1182	1124	1124
query71	1408	1266	1274	1266
query72	6581	2652	2438	2438
query73	721	316	321	316
query74	6914	6408	6549	6408
query75	3444	2610	2632	2610
query76	3573	954	947	947
query77	615	270	279	270
query78	10902	10217	10118	10118
query79	8367	521	504	504
query80	1476	438	435	435
query81	522	250	250	250
query82	1583	97	92	92
query83	200	167	187	167
query84	253	84	92	84
query85	1473	273	265	265
query86	496	302	284	284
query87	3477	3319	3400	3319
query88	5172	2396	2401	2396
query89	548	366	378	366
query90	1963	173	181	173
query91	125	100	97	97
query92	59	49	47	47
query93	6903	519	493	493
query94	1183	177	177	177
query95	395	301	296	296
query96	599	268	260	260
query97	3101	2902	2910	2902
query98	244	212	223	212
query99	1251	870	856	856
Total cold run time: 299084 ms
Total hot run time: 184522 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 29.64 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit f20458b81fafadc8f46431930b74894c6bd5f26f, data reload: false

query1	0.04	0.03	0.03
query2	0.08	0.04	0.04
query3	0.23	0.05	0.05
query4	1.70	0.07	0.07
query5	0.49	0.50	0.50
query6	1.48	0.73	0.73
query7	0.02	0.02	0.02
query8	0.04	0.04	0.05
query9	0.54	0.49	0.50
query10	0.55	0.57	0.53
query11	0.16	0.12	0.12
query12	0.14	0.12	0.12
query13	0.58	0.58	0.59
query14	0.75	0.77	0.79
query15	0.82	0.82	0.80
query16	0.36	0.37	0.37
query17	0.96	1.03	0.99
query18	0.22	0.23	0.24
query19	1.79	1.77	1.70
query20	0.01	0.01	0.01
query21	15.41	0.64	0.63
query22	4.88	7.84	1.22
query23	18.28	1.32	1.26
query24	1.62	0.30	0.21
query25	0.14	0.09	0.08
query26	0.26	0.17	0.17
query27	0.08	0.08	0.08
query28	13.36	1.00	0.98
query29	12.64	3.28	3.24
query30	0.26	0.06	0.05
query31	2.86	0.39	0.37
query32	3.27	0.46	0.47
query33	2.84	2.80	2.86
query34	17.05	4.43	4.41
query35	4.53	4.46	4.47
query36	0.65	0.48	0.46
query37	0.18	0.15	0.15
query38	0.15	0.14	0.15
query39	0.04	0.03	0.04
query40	0.18	0.15	0.13
query41	0.09	0.05	0.05
query42	0.06	0.04	0.05
query43	0.04	0.04	0.03
Total cold run time: 109.83 s
Total hot run time: 29.64 s

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit f20458b81fafadc8f46431930b74894c6bd5f26f with default session variables
Stream load json:         18 seconds loaded 2358488459 Bytes, about 124 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      32 seconds loaded 861443392 Bytes, about 25 MB/s
Insert into select:       13.5 seconds inserted 10000000 Rows, about 740K ops/s

@github-actions
Copy link
Contributor

PR approved by anyone and no changes requested.

Copy link
Contributor

@zhannngchen zhannngchen left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Apr 17, 2024
@github-actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@zhannngchen zhannngchen merged commit 64ff8fe into apache:master Apr 17, 2024
Yukang-Lian added a commit to Yukang-Lian/doris that referenced this pull request Apr 17, 2024
zhannngchen pushed a commit that referenced this pull request Apr 18, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by one committer. reviewed

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants