Skip to content

branch-3.1: [fix](mow) fix agg delete bitmap for stale rowsets #53611#53662

Merged
morrySnow merged 1 commit intobranch-3.1from
auto-pick-53611-branch-3.1
Jul 22, 2025
Merged

branch-3.1: [fix](mow) fix agg delete bitmap for stale rowsets #53611#53662
morrySnow merged 1 commit intobranch-3.1from
auto-pick-53611-branch-3.1

Conversation

@github-actions
Copy link
Copy Markdown
Contributor

@github-actions github-actions bot commented Jul 21, 2025

Cherry-picked from #53611

@mymeiyi
[fix](mow) fix agg delete bitmap for stale rowsets (#53611)
b64b014 
### What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:
```
F20250718 18:41:03.517045 377331 base_tablet.cpp:1752] Check failed: start_version < end_version . start_version: 147, end_version: 147
*** Check failure stack trace: ***
    @     0x561faa3a6df6  google::LogMessage::SendToLog()
    @     0x561faa3a3840  google::LogMessage::Flush()
    @     0x561faa3a7639  google::LogMessageFatal::~LogMessageFatal()
    @     0x561f6e3567d0  doris::BaseTablet::agg_delete_bitmap_for_stale_rowsets()
    @     0x561faa25c065  doris::CloudTablet::delete_expired_stale_rowsets()
    @     0x561faa2aa50b  doris::CloudTabletMgr::vacuum_stale_rowsets()
    @     0x561faa17b028  doris::CloudStorageEngine::_vacuum_stale_rowsets_thread_callback()
```
@github-actions github-actions bot requested a review from morrySnow as a code owner July 21, 2025 11:32
@Thearas
Copy link
Copy Markdown
Contributor

Thearas commented Jul 21, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@dataroaring dataroaring reopened this Jul 21, 2025
@Thearas
Copy link
Copy Markdown
Contributor

Thearas commented Jul 21, 2025

run buildall

@doris-robot
Copy link
Copy Markdown

doris-robot commented Jul 21, 2025

TPC-H: Total hot run time: 33024 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit b64b014284220742747622a9c23c13d01d48e85f, data reload: false

------ Round 1 ----------------------------------
q1	17606	5523	5410	5410
q2	2077	302	197	197
q3	10499	1246	792	792
q4	10307	882	463	463
q5	9097	2393	2157	2157
q6	207	168	143	143
q7	904	766	633	633
q8	9357	1458	1196	1196
q9	5324	4941	4882	4882
q10	6848	2271	1848	1848
q11	515	302	286	286
q12	361	367	235	235
q13	17774	3677	3030	3030
q14	240	246	235	235
q15	572	514	517	514
q16	464	447	397	397
q17	616	879	372	372
q18	7014	6438	6438	6438
q19	1418	987	608	608
q20	360	367	222	222
q21	2913	2209	1991	1991
q22	1050	1054	975	975
Total cold run time: 105523 ms
Total hot run time: 33024 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5602	5525	5585	5525
q2	260	349	252	252
q3	2277	2587	2327	2327
q4	1349	1813	1311	1311
q5	4469	4990	4973	4973
q6	187	177	139	139
q7	2089	1973	1828	1828
q8	2642	2883	2746	2746
q9	7267	7132	7194	7132
q10	3057	3316	2838	2838
q11	595	515	494	494
q12	669	785	616	616
q13	3357	3791	3191	3191
q14	301	307	308	307
q15	562	512	501	501
q16	459	511	450	450
q17	1269	1768	1276	1276
q18	7734	7459	7489	7459
q19	874	1118	1190	1118
q20	2068	2160	1921	1921
q21	5357	4856	4734	4734
q22	1132	1110	1071	1071
Total cold run time: 53576 ms
Total hot run time: 52209 ms

@doris-robot
Copy link
Copy Markdown

doris-robot commented Jul 21, 2025

BE UT Coverage Report

Increment line coverage 16.67% (1/6) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 45.27% (12560/27744)
Line Coverage 36.19% (111848/309087)
Region Coverage 35.25% (57829/164040)
Branch Coverage 32.43% (31422/96888)

@hello-stephen
Copy link
Copy Markdown
Contributor

hello-stephen commented Jul 21, 2025

BE Regression && UT Coverage Report

Increment line coverage 66.67% (4/6) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 76.14% (20872/27411)
Line Coverage 69.50% (214637/308850)
Region Coverage 67.54% (128417/190138)
Branch Coverage 61.08% (66763/109306)

@doris-robot
Copy link
Copy Markdown

doris-robot commented Jul 22, 2025

TPC-H: Total hot run time: 32267 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit b64b014284220742747622a9c23c13d01d48e85f, data reload: false

------ Round 1 ----------------------------------
q1	17581	5436	5464	5436
q2	2043	304	168	168
q3	10506	1265	739	739
q4	10236	882	450	450
q5	8363	2383	2053	2053
q6	178	159	132	132
q7	906	751	598	598
q8	9325	1389	1147	1147
q9	5204	4902	4884	4884
q10	6826	2271	1834	1834
q11	490	295	265	265
q12	346	360	218	218
q13	17802	3607	3038	3038
q14	220	226	207	207
q15	536	464	479	464
q16	448	431	369	369
q17	603	854	351	351
q18	6724	6428	6251	6251
q19	1558	959	553	553
q20	335	341	200	200
q21	2771	2140	1928	1928
q22	1035	1045	982	982
Total cold run time: 104036 ms
Total hot run time: 32267 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5551	5435	5501	5435
q2	233	333	249	249
q3	2236	2635	2317	2317
q4	1324	1765	1346	1346
q5	4440	4786	4937	4786
q6	169	158	123	123
q7	2052	1984	1819	1819
q8	2593	2819	2665	2665
q9	7334	7164	7224	7164
q10	3003	3295	2800	2800
q11	581	501	484	484
q12	644	739	549	549
q13	3350	3797	3186	3186
q14	271	297	279	279
q15	545	458	462	458
q16	444	492	450	450
q17	1219	1727	1277	1277
q18	7516	7341	7456	7341
q19	839	1185	1080	1080
q20	2020	2049	1877	1877
q21	5211	4921	4738	4738
q22	1117	1051	986	986
Total cold run time: 52692 ms
Total hot run time: 51409 ms

@doris-robot
Copy link
Copy Markdown

doris-robot commented Jul 22, 2025

TPC-DS: Total hot run time: 198926 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit b64b014284220742747622a9c23c13d01d48e85f, data reload: false

query1	1273	956	927	927
query2	6220	2027	2021	2021
query3	10936	4545	4688	4545
query4	32812	23598	23692	23598
query5	3960	607	451	451
query6	283	199	190	190
query7	3988	495	324	324
query8	292	260	235	235
query9	9363	2589	2573	2573
query10	492	339	259	259
query11	17955	15491	15220	15220
query12	159	104	102	102
query13	1556	569	419	419
query14	10031	7935	7915	7915
query15	229	204	188	188
query16	8065	688	498	498
query17	1603	761	601	601
query18	2140	447	325	325
query19	228	194	168	168
query20	129	120	118	118
query21	204	128	115	115
query22	4655	4531	4467	4467
query23	35534	34075	33994	33994
query24	7473	2746	2788	2746
query25	508	501	428	428
query26	853	293	180	180
query27	2191	491	356	356
query28	5195	2240	2173	2173
query29	713	587	473	473
query30	243	184	161	161
query31	1002	907	854	854
query32	81	65	58	58
query33	482	382	315	315
query34	758	854	534	534
query35	799	819	771	771
query36	1019	1059	959	959
query37	103	101	70	70
query38	4119	4045	4081	4045
query39	1549	1447	1497	1447
query40	206	119	103	103
query41	48	48	50	48
query42	121	104	102	102
query43	527	540	490	490
query44	1396	875	864	864
query45	187	181	169	169
query46	912	1084	706	706
query47	1954	1978	1925	1925
query48	410	453	364	364
query49	732	489	410	410
query50	725	709	455	455
query51	7495	7441	7206	7206
query52	106	108	95	95
query53	249	274	195	195
query54	549	569	490	490
query55	80	77	78	77
query56	288	275	261	261
query57	1274	1249	1211	1211
query58	233	236	226	226
query59	3234	3369	3193	3193
query60	303	295	272	272
query61	114	106	118	106
query62	807	765	701	701
query63	245	206	198	198
query64	3608	999	636	636
query65	3353	3331	3317	3317
query66	778	411	304	304
query67	15888	15661	15556	15556
query68	6688	859	550	550
query69	503	312	265	265
query70	1230	1149	1102	1102
query71	403	294	268	268
query72	5794	3823	3774	3774
query73	636	748	359	359
query74	10413	9198	9269	9198
query75	3222	3153	2637	2637
query76	3102	1187	774	774
query77	505	383	295	295
query78	10320	10482	9619	9619
query79	3663	896	585	585
query80	735	527	461	461
query81	490	256	220	220
query82	669	119	95	95
query83	168	163	150	150
query84	246	103	77	77
query85	776	355	299	299
query86	392	319	288	288
query87	4332	4367	4217	4217
query88	5289	2432	2425	2425
query89	425	337	311	311
query90	1765	193	194	193
query91	136	140	113	113
query92	60	58	55	55
query93	2628	883	548	548
query94	688	430	289	289
query95	343	283	275	275
query96	486	630	287	287
query97	3184	3294	3157	3157
query98	228	216	198	198
query99	1315	1410	1339	1339
Total cold run time: 295851 ms
Total hot run time: 198926 ms

@doris-robot
Copy link
Copy Markdown

doris-robot commented Jul 22, 2025

ClickBench: Total hot run time: 28.91 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit b64b014284220742747622a9c23c13d01d48e85f, data reload: false

query1	0.04	0.04	0.02
query2	0.07	0.03	0.03
query3	0.23	0.07	0.06
query4	1.62	0.10	0.11
query5	0.52	0.53	0.50
query6	1.13	0.74	0.73
query7	0.02	0.02	0.02
query8	0.04	0.03	0.03
query9	0.56	0.50	0.51
query10	0.56	0.56	0.60
query11	0.14	0.10	0.10
query12	0.13	0.12	0.11
query13	0.61	0.62	0.60
query14	0.77	0.80	0.79
query15	0.83	0.83	0.82
query16	0.38	0.38	0.38
query17	1.08	1.07	1.08
query18	0.24	0.21	0.22
query19	1.98	1.88	1.90
query20	0.02	0.02	0.02
query21	15.36	0.95	0.56
query22	0.75	0.76	0.62
query23	15.22	1.31	0.55
query24	3.28	1.70	0.92
query25	0.12	0.18	0.08
query26	0.34	0.16	0.13
query27	0.05	0.06	0.04
query28	14.08	1.00	0.43
query29	12.59	3.93	3.30
query30	0.25	0.08	0.07
query31	2.84	0.58	0.38
query32	3.23	0.53	0.45
query33	2.95	3.00	3.04
query34	16.59	5.18	4.50
query35	4.56	4.63	4.56
query36	0.65	0.49	0.49
query37	0.09	0.06	0.06
query38	0.05	0.04	0.03
query39	0.03	0.02	0.02
query40	0.17	0.13	0.12
query41	0.07	0.02	0.02
query42	0.04	0.02	0.03
query43	0.04	0.03	0.03
Total cold run time: 104.32 s
Total hot run time: 28.91 s

@morrySnow morrySnow merged commit 5be1984 into branch-3.1 Jul 22, 2025
21 of 23 checks passed
@github-actions github-actions bot deleted the auto-pick-53611-branch-3.1 branch July 22, 2025 06:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@morrySnow morrySnow morrySnow approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@Thearas @doris-robot @hello-stephen @morrySnow @dataroaring @mymeiyi