Skip to content

[fix](cloud) Fix cloud drop tablet tasks pile up #58131

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
dataroaring merged 1 commit into from
Nov 19, 2025
Merged

[fix](cloud) Fix cloud drop tablet tasks pile up #58131

dataroaring merged 1 commit into from
Nov 19, 2025
+18 −5

Conversation

@deardeng
Copy link
Contributor

@deardeng deardeng commented Nov 18, 2025

What problem does this PR solve?

Previously, since clean tablet task was a lightweight operation on the BE, it was assumed that dropping tablets would not cause task backlog. However, online observations showed that tasks were still backlogged, causing the BE to consume a lot of memory. Therefore, tablet deduplication logic was added to prevent task backlog.

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@hello-stephen
Copy link
Contributor

hello-stephen commented Nov 18, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Nov 18, 2025
@github-actions
Copy link
Contributor

github-actions bot commented Nov 18, 2025

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Contributor

github-actions bot commented Nov 18, 2025

PR approved by anyone and no changes requested.

@deardeng
Copy link
Contributor Author

deardeng commented Nov 18, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Nov 18, 2025

TPC-H: Total hot run time: 33939 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit e1b3b251c85a1c46a038f0795b369335f9d34d84, data reload: false

------ Round 1 ----------------------------------
q1	17639	5050	4906	4906
q2	2023	307	204	204
q3	10287	1263	705	705
q4	10241	972	373	373
q5	7901	2441	2360	2360
q6	186	171	137	137
q7	928	788	641	641
q8	9357	1430	1192	1192
q9	6837	5021	5177	5021
q10	6871	2233	1799	1799
q11	521	313	281	281
q12	342	372	230	230
q13	17785	3693	2996	2996
q14	230	242	209	209
q15	577	502	503	502
q16	1041	1000	933	933
q17	591	869	361	361
q18	7360	7092	7022	7022
q19	1085	953	575	575
q20	368	352	235	235
q21	3920	2551	2281	2281
q22	1047	1008	976	976
Total cold run time: 107137 ms
Total hot run time: 33939 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4955	4958	4942	4942
q2	244	331	230	230
q3	2144	2679	2269	2269
q4	1343	1793	1339	1339
q5	4204	4307	4531	4307
q6	209	170	138	138
q7	2046	2009	1865	1865
q8	2787	2685	2723	2685
q9	7188	7099	7184	7099
q10	3059	3241	2828	2828
q11	584	526	526	526
q12	733	726	671	671
q13	3526	4187	3302	3302
q14	283	292	297	292
q15	539	517	505	505
q16	1080	1087	1061	1061
q17	1232	1625	1448	1448
q18	7780	7620	7600	7600
q19	834	886	898	886
q20	1955	1967	1807	1807
q21	4746	4341	4268	4268
q22	1086	1023	1032	1023
Total cold run time: 52557 ms
Total hot run time: 51091 ms

@doris-robot
Copy link

doris-robot commented Nov 18, 2025

TPC-DS: Total hot run time: 188429 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit e1b3b251c85a1c46a038f0795b369335f9d34d84, data reload: false

query1	1045	407	393	393
query2	6548	1658	1651	1651
query3	6765	239	240	239
query4	26168	23339	22866	22866
query5	4476	710	517	517
query6	358	269	237	237
query7	4651	495	307	307
query8	310	261	255	255
query9	8656	2996	2958	2958
query10	515	359	314	314
query11	15509	15006	15086	15006
query12	197	134	122	122
query13	1704	585	476	476
query14	10894	9164	9020	9020
query15	199	184	170	170
query16	7303	665	492	492
query17	1215	758	606	606
query18	1990	425	334	334
query19	214	194	177	177
query20	140	132	121	121
query21	208	131	114	114
query22	4159	4473	4170	4170
query23	33702	33046	32938	32938
query24	8101	2401	2403	2401
query25	587	514	486	486
query26	1241	279	179	179
query27	2739	504	358	358
query28	4398	2286	2258	2258
query29	870	646	526	526
query30	298	227	206	206
query31	902	838	720	720
query32	95	95	86	86
query33	611	423	371	371
query34	814	840	527	527
query35	807	839	762	762
query36	953	1032	903	903
query37	134	117	98	98
query38	3532	3509	3468	3468
query39	1471	1523	1460	1460
query40	239	146	131	131
query41	68	69	64	64
query42	140	122	123	122
query43	487	472	475	472
query44	1239	850	797	797
query45	188	196	175	175
query46	894	985	653	653
query47	1791	1795	1751	1751
query48	397	439	339	339
query49	767	493	426	426
query50	655	692	406	406
query51	3830	3934	3953	3934
query52	117	115	111	111
query53	243	278	215	215
query54	333	312	295	295
query55	93	97	91	91
query56	350	361	337	337
query57	1158	1205	1130	1130
query58	301	289	292	289
query59	2492	2650	2466	2466
query60	366	378	355	355
query61	166	149	150	149
query62	767	717	635	635
query63	237	200	197	197
query64	4500	1193	885	885
query65	4020	3947	3929	3929
query66	1194	449	379	379
query67	15379	15328	14878	14878
query68	8193	945	645	645
query69	492	341	310	310
query70	1289	1277	1295	1277
query71	450	356	330	330
query72	5969	4823	4875	4823
query73	656	590	366	366
query74	9260	9014	8863	8863
query75	3574	3280	2756	2756
query76	3252	1185	751	751
query77	646	407	340	340
query78	9501	9739	8936	8936
query79	1828	870	595	595
query80	741	602	534	534
query81	489	260	226	226
query82	222	168	137	137
query83	270	269	253	253
query84	262	117	91	91
query85	915	497	443	443
query86	333	308	299	299
query87	3707	3714	3565	3565
query88	2837	2257	2237	2237
query89	380	335	298	298
query90	1784	231	224	224
query91	170	169	136	136
query92	80	76	74	74
query93	1124	997	689	689
query94	688	429	336	336
query95	421	348	334	334
query96	480	616	282	282
query97	2892	2942	2875	2875
query98	236	228	216	216
query99	1313	1394	1285	1285
Total cold run time: 271344 ms
Total hot run time: 188429 ms

@doris-robot
Copy link

doris-robot commented Nov 18, 2025

ClickBench: Total hot run time: 27.75 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit e1b3b251c85a1c46a038f0795b369335f9d34d84, data reload: false

query1	0.05	0.04	0.05
query2	0.11	0.05	0.05
query3	0.25	0.08	0.08
query4	1.61	0.12	0.11
query5	0.27	0.25	0.24
query6	1.16	0.66	0.63
query7	0.04	0.03	0.03
query8	0.06	0.05	0.05
query9	0.58	0.54	0.51
query10	0.58	0.58	0.57
query11	0.16	0.11	0.11
query12	0.15	0.12	0.13
query13	0.61	0.60	0.61
query14	1.00	1.00	1.01
query15	0.85	0.83	0.84
query16	0.38	0.41	0.39
query17	1.02	1.04	0.99
query18	0.22	0.20	0.20
query19	1.93	1.76	1.87
query20	0.02	0.01	0.01
query21	15.44	0.21	0.13
query22	5.04	0.08	0.05
query23	15.66	0.24	0.10
query24	2.85	0.65	1.10
query25	0.09	0.06	0.06
query26	0.15	0.13	0.14
query27	0.06	0.06	0.05
query28	5.38	1.12	0.96
query29	12.70	3.91	3.24
query30	0.27	0.14	0.12
query31	2.82	0.57	0.40
query32	3.23	0.56	0.47
query33	2.99	3.09	3.14
query34	15.74	5.14	4.57
query35	4.60	4.57	4.60
query36	0.66	0.50	0.49
query37	0.10	0.07	0.06
query38	0.07	0.04	0.04
query39	0.04	0.02	0.03
query40	0.17	0.15	0.14
query41	0.08	0.03	0.03
query42	0.04	0.03	0.03
query43	0.05	0.04	0.03
Total cold run time: 99.28 s
Total hot run time: 27.75 s

@hello-stephen
Copy link
Contributor

hello-stephen commented Nov 18, 2025

BE UT Coverage Report

Increment line coverage 0.00% (0/1) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 52.73% (18273/34651)
Line Coverage 38.10% (166027/435717)
Region Coverage 33.03% (129066/390706)
Branch Coverage 33.81% (55386/163807)

@hello-stephen
Copy link
Contributor

hello-stephen commented Nov 18, 2025

BE Regression && UT Coverage Report

Increment line coverage 0.00% (0/1) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 73.06% (24876/34051)
Line Coverage 60.06% (262094/436386)
Region Coverage 55.27% (219039/396282)
Branch Coverage 57.05% (94074/164886)

@deardeng
Copy link
Contributor Author

deardeng commented Nov 19, 2025

run cloud_p0

@deardeng
Copy link
Contributor Author

deardeng commented Nov 19, 2025

run external

@deardeng
Copy link
Contributor Author

deardeng commented Nov 19, 2025

run p0

@hello-stephen
Copy link
Contributor

hello-stephen commented Nov 19, 2025

BE Regression && UT Coverage Report

Increment line coverage 0.00% (0/1) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 73.05% (24874/34051)
Line Coverage 60.04% (262016/436386)
Region Coverage 55.25% (218935/396282)
Branch Coverage 57.03% (94032/164886)

1 similar comment
@hello-stephen
Copy link
Contributor

hello-stephen commented Nov 19, 2025

BE Regression && UT Coverage Report

Increment line coverage 0.00% (0/1) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 73.05% (24874/34051)
Line Coverage 60.04% (262016/436386)
Region Coverage 55.25% (218935/396282)
Branch Coverage 57.03% (94032/164886)

dataroaring
dataroaring approved these changes Nov 19, 2025
View reviewed changes
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@dataroaring dataroaring merged commit 0d1c7c3 into apache:master Nov 19, 2025
28 of 30 checks passed
github-actions bot pushed a commit that referenced this pull request Nov 19, 2025
@deardeng
[fix](cloud) Fix cloud drop tablet tasks pile up (#58131)
000550d 
### What problem does this PR solve?

Previously, since clean tablet task was a lightweight operation on the
BE, it was assumed that dropping tablets would not cause task backlog.
However, online observations showed that tasks were still backlogged,
causing the BE to consume a lot of memory. Therefore, tablet
deduplication logic was added to prevent task backlog.
@github-actions github-actions bot mentioned this pull request Nov 19, 2025
Merged
github-actions bot pushed a commit that referenced this pull request Nov 19, 2025
@deardeng
[fix](cloud) Fix cloud drop tablet tasks pile up (#58131)
63f0fa2 
### What problem does this PR solve?

Previously, since clean tablet task was a lightweight operation on the
BE, it was assumed that dropping tablets would not cause task backlog.
However, online observations showed that tasks were still backlogged,
causing the BE to consume a lot of memory. Therefore, tablet
deduplication logic was added to prevent task backlog.
@github-actions github-actions bot mentioned this pull request Nov 19, 2025
Merged
yiguolei pushed a commit that referenced this pull request Nov 20, 2025
@github-actions @deardeng
branch-4.0: [fix](cloud) Fix cloud drop tablet tasks pile up #58131 (#…
993888e 
…58158)

Cherry-picked from #58131

Co-authored-by: deardeng <dengxin@selectdb.com>
morrySnow pushed a commit that referenced this pull request Nov 25, 2025
@github-actions @deardeng
branch-3.1: [fix](cloud) Fix cloud drop tablet tasks pile up #58131 (#…
d027875 
…58157)

Cherry-picked from #58131

Co-authored-by: deardeng <dengxin@selectdb.com>
@yiguolei yiguolei mentioned this pull request Dec 2, 2025
Open
nagisa-kunhah pushed a commit to nagisa-kunhah/doris that referenced this pull request Dec 14, 2025
@deardeng @nagisa-kunhah
[fix](cloud) Fix cloud drop tablet tasks pile up (apache#58131)
6cc788a 
### What problem does this PR solve?

Previously, since clean tablet task was a lightweight operation on the
BE, it was assumed that dropping tablets would not cause task backlog.
However, online observations showed that tasks were still backlogged,
causing the BE to consume a lot of memory. Therefore, tablet
deduplication logic was added to prevent task backlog.
w41ter pushed a commit to w41ter/incubator-doris that referenced this pull request Dec 26, 2025
@deardeng
[fix](cloud) Fix cloud drop tablet tasks pile up (apache#5793)
23c6e8e 
## Proposed changes
pick from apache#58131

Previously, since clean tablet task was a lightweight operation on the
BE, it was assumed that dropping tablets would not cause task backlog.
However, online observations showed that tasks were still backlogged,
causing the BE to consume a lot of memory. Therefore, tablet
deduplication logic was added to prevent task backlog.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@gavinchou gavinchou gavinchou approved these changes

@dataroaring dataroaring dataroaring approved these changes

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. dev/3.1.4-merged dev/4.0.2-merged reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

7 participants

@deardeng @hello-stephen @doris-robot @gavinchou @dataroaring @yiguolei @morrySnow