Skip to content

[improvement](fe) Add cloud tablet rebalancer metrics - #66576

Open
deardeng wants to merge 1 commit into
apache:masterfrom
deardeng:improve-cloud-tablet-rebalancer-metrics
Open

[improvement](fe) Add cloud tablet rebalancer metrics#66576
deardeng wants to merge 1 commit into
apache:masterfrom
deardeng:improve-cloud-tablet-rebalancer-metrics

Conversation

@deardeng

@deardeng deardeng commented Aug 7, 2026

Copy link
Copy Markdown
Contributor

Problem Summary: Cloud tablet rebalancer memory optimizations can only be inferred from process-wide JVM metrics. Record per-round thread allocation, duration, and tablet-route scan work so allocation changes can be compared per round and per work item. Use ThreadMXBean when supported and expose -1 for the last-round allocation when unavailable.

after Related PR: #66378, #66389, #66451, #66447 fix

image image

Release note

Expose CloudTabletRebalancer round, allocation, duration, and tablet-scan metrics in FE metrics.

Check List (For Author)

  • Test: Unit Test
    • ./run-fe-ut.sh --run org.apache.doris.cloud.catalog.CloudTabletRebalancerMetricsTest
    • ./run-fe-ut.sh --coverage --run org.apache.doris.cloud.catalog.CloudTabletRebalancerMetricsTest,org.apache.doris.metric.MetricsTest
    • mvn checkstyle:check -pl fe-core
  • Behavior changed: Yes (adds FE observability metrics without changing rebalance decisions)
  • Does this need documentation: No

What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason
  • Behavior changed:

    • No.
    • Yes.
  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

### What problem does this PR solve?

Issue Number: None

Related PR: apache#66447

Problem Summary: Cloud tablet rebalancer memory optimizations can only be inferred from process-wide JVM metrics. Record per-round thread allocation, duration, and tablet-route scan work so allocation changes can be compared per round and per work item. Use ThreadMXBean when supported and expose -1 for the last-round allocation when unavailable.

### Release note

Expose CloudTabletRebalancer round, allocation, duration, and tablet-scan metrics in FE metrics.

### Check List (For Author)

- Test: Unit Test
    - ./run-fe-ut.sh --run org.apache.doris.cloud.catalog.CloudTabletRebalancerMetricsTest
    - ./run-fe-ut.sh --coverage --run org.apache.doris.cloud.catalog.CloudTabletRebalancerMetricsTest,org.apache.doris.metric.MetricsTest
    - mvn checkstyle:check -pl fe-core
- Behavior changed: Yes (adds FE observability metrics without changing rebalance decisions)
- Does this need documentation: No
@deardeng
deardeng requested a review from gavinchou as a code owner August 7, 2026 08:12
@hello-stephen

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@deardeng

deardeng commented Aug 7, 2026

Copy link
Copy Markdown
Contributor Author

run buildall

@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-H: Total hot run time: 28903 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 5dd697fafde899b63d71eaa05aa387b57ef6d6d1, data reload: false

------ Round 1 ----------------------------------
============================================
q1	17969	3984	4012	3984
q2	1990	313	193	193
q3	10340	1359	794	794
q4	4680	468	336	336
q5	7497	831	555	555
q6	193	165	135	135
q7	747	823	600	600
q8	9884	1700	1592	1592
q9	5754	4067	3971	3971
q10	6779	1602	1382	1382
q11	504	337	319	319
q12	779	558	445	445
q13	18116	3243	2740	2740
q14	262	260	242	242
q15	q16	730	704	650	650
q17	1081	959	948	948
q18	6598	5563	5551	5551
q19	1760	1293	1111	1111
q20	796	700	576	576
q21	5951	2529	2485	2485
q22	431	355	294	294
Total cold run time: 102841 ms
Total hot run time: 28903 ms

----- Round 2, with runtime_filter_mode=off -----
============================================
q1	4338	4240	4227	4227
q2	272	320	205	205
q3	4566	4879	4352	4352
q4	2169	2247	1410	1410
q5	4231	4094	4076	4076
q6	227	172	128	128
q7	1703	1583	1685	1583
q8	2482	2059	2005	2005
q9	7232	7168	7247	7168
q10	4338	4247	3876	3876
q11	595	423	367	367
q12	749	740	498	498
q13	3174	3626	2806	2806
q14	346	318	272	272
q15	q16	724	725	671	671
q17	1319	1304	1266	1266
q18	12142	11093	11810	11093
q19	1159	1145	1151	1145
q20	2236	2206	1910	1910
q21	5665	4927	4532	4532
q22	505	445	413	413
Total cold run time: 60172 ms
Total hot run time: 54003 ms

@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-DS: Total hot run time: 158230 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 5dd697fafde899b63d71eaa05aa387b57ef6d6d1, data reload: false

query5	4341	585	462	462
query6	460	230	208	208
query7	4931	611	326	326
query8	320	181	165	165
query9	8772	4051	4062	4051
query10	515	393	312	312
query11	5877	2181	1984	1984
query12	149	101	97	97
query13	1255	621	426	426
query14	6068	4263	3976	3976
query14_1	3775	3767	3790	3767
query15	201	196	183	183
query16	1009	542	477	477
query17	1001	708	559	559
query18	2439	464	377	377
query19	200	189	139	139
query20	103	98	104	98
query21	225	155	130	130
query22	13022	13128	12708	12708
query23	15752	14890	14625	14625
query23_1	14697	14557	14585	14557
query24	7903	1676	1202	1202
query24_1	1254	1235	1203	1203
query25	517	427	374	374
query26	1297	343	203	203
query27	2609	610	391	391
query28	4526	2040	2017	2017
query29	1040	587	457	457
query30	343	261	225	225
query31	1159	1114	1036	1036
query32	112	60	58	58
query33	519	302	239	239
query34	1170	1111	646	646
query35	753	738	651	651
query36	765	781	725	725
query37	160	111	93	93
query38	1808	1755	1674	1674
query39	806	817	802	802
query39_1	775	792	789	789
query40	244	165	143	143
query41	63	63	61	61
query42	93	94	92	92
query43	311	314	278	278
query44	1404	769	777	769
query45	183	173	175	173
query46	1091	1211	742	742
query47	1557	1503	1484	1484
query48	408	425	305	305
query49	600	411	292	292
query50	1055	434	335	335
query51	10595	10765	10726	10726
query52	86	86	75	75
query53	259	273	201	201
query54	277	232	218	218
query55	81	71	72	71
query56	309	277	277	277
query57	1036	983	932	932
query58	291	268	244	244
query59	1507	1619	1369	1369
query60	287	268	258	258
query61	145	145	158	145
query62	405	321	266	266
query63	232	200	194	194
query64	2901	1149	985	985
query65	3824	3807	3777	3777
query66	1850	481	381	381
query67	20247	20185	19929	19929
query68	3261	1508	967	967
query69	421	304	284	284
query70	889	789	795	789
query71	385	350	313	313
query72	3129	2767	2213	2213
query73	806	753	450	450
query74	4627	4511	4292	4292
query75	2363	2349	1958	1958
query76	2363	1122	743	743
query77	340	365	262	262
query78	11187	11186	10475	10475
query79	1714	1106	755	755
query80	1244	566	470	470
query81	525	337	286	286
query82	640	175	141	141
query83	372	323	294	294
query84	325	160	130	130
query85	959	604	548	548
query86	392	244	229	229
query87	1973	1971	1827	1827
query88	3723	2791	2735	2735
query89	390	322	286	286
query90	1883	197	200	197
query91	198	188	163	163
query92	64	58	54	54
query93	1755	1527	990	990
query94	695	341	316	316
query95	808	539	558	539
query96	1068	787	380	380
query97	2468	2452	2366	2366
query98	198	183	188	183
query99	767	750	611	611
Total cold run time: 246128 ms
Total hot run time: 158230 ms

@hello-stephen

Copy link
Copy Markdown
Contributor
ClickBench: Total hot run time: 23.61 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 5dd697fafde899b63d71eaa05aa387b57ef6d6d1, data reload: false

query1	0.01	0.01	0.01
query2	0.10	0.05	0.05
query3	0.25	0.14	0.13
query4	1.61	0.14	0.13
query5	0.24	0.22	0.21
query6	1.16	0.80	0.82
query7	0.03	0.01	0.00
query8	0.06	0.04	0.04
query9	0.37	0.30	0.32
query10	0.54	0.54	0.58
query11	0.20	0.13	0.13
query12	0.18	0.14	0.14
query13	0.46	0.46	0.46
query14	0.99	1.00	0.98
query15	0.59	0.59	0.58
query16	0.30	0.32	0.31
query17	1.06	1.07	1.12
query18	0.22	0.19	0.20
query19	2.05	1.90	1.88
query20	0.02	0.01	0.01
query21	15.43	0.21	0.14
query22	4.89	0.06	0.05
query23	16.13	0.30	0.12
query24	3.02	0.42	0.31
query25	0.10	0.06	0.04
query26	0.76	0.21	0.14
query27	0.04	0.03	0.04
query28	3.45	0.79	0.36
query29	12.49	3.97	3.18
query30	0.27	0.14	0.15
query31	2.76	0.58	0.31
query32	3.21	0.60	0.49
query33	3.14	3.11	3.27
query34	15.66	3.92	3.26
query35	3.22	3.23	3.21
query36	0.54	0.42	0.41
query37	0.08	0.06	0.07
query38	0.05	0.04	0.03
query39	0.04	0.03	0.02
query40	0.17	0.16	0.15
query41	0.08	0.03	0.03
query42	0.04	0.03	0.03
query43	0.05	0.03	0.03
Total cold run time: 96.06 s
Total hot run time: 23.61 s

@hello-stephen

Copy link
Copy Markdown
Contributor

FE Regression Coverage Report

Increment line coverage 74.42% (64/86) 🎉
Increment coverage report
Complete coverage report

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants