Skip to content

branch-3.0: [fix](cloud) Fix ConcurrentModificationException in cloud rebalance #52013 #52309

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
dataroaring merged 1 commit into from
Jun 26, 2025
Merged

branch-3.0: [fix](cloud) Fix ConcurrentModificationException in cloud rebalance #52013 #52309

dataroaring merged 1 commit into from
Jun 26, 2025

Conversation

@github-actions
Copy link
Contributor

@github-actions github-actions bot commented Jun 25, 2025

Cherry-picked from #52013

@deardeng
[fix](cloud) Fix ConcurrentModificationException in cloud rebalance (…
6beef18 
…#52013)

```
2025-06-19 15:58:44,191 ERROR (cloud tablet rebalancer|277) [Daemon.run():121] daemon thread got exception. name: cloud tablet rebalancer
java.util.ConcurrentModificationException: null
        at java.util.HashMap$HashIterator.nextNode(HashMap.java:1597) ~[?:?]
        at java.util.HashMap$KeyIterator.next(HashMap.java:1620) ~[?:?]
        at org.apache.doris.cloud.catalog.CloudTabletRebalancer.checkInflghtWarmUpCacheAsync(CloudTabletRebalancer.java:421) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.cloud.catalog.CloudTabletRebalancer.runAfterCatalogReady(CloudTabletRebalancer.java:257) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.common.util.MasterDaemon.runOneCycle(MasterDaemon.java:58) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.common.util.Daemon.run(Daemon.java:119) ~[doris-fe.jar:1.2-SNAPSHOT]
```
@github-actions github-actions bot requested a review from dataroaring as a code owner June 25, 2025 11:38
@Thearas
Copy link
Contributor

Thearas commented Jun 25, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@dataroaring dataroaring reopened this Jun 25, 2025
@Thearas
Copy link
Contributor

Thearas commented Jun 25, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Jun 25, 2025

TPC-H: Total hot run time: 39731 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 6beef187de53280974ff0ec7d60036bd5a0118b3, data reload: false

------ Round 1 ----------------------------------
q1	17586	6900	6635	6635
q2	2073	177	160	160
q3	10732	1094	1190	1094
q4	10475	770	734	734
q5	7723	2907	2885	2885
q6	209	136	131	131
q7	963	629	606	606
q8	9347	1931	2063	1931
q9	6667	6402	6434	6402
q10	7047	2292	2299	2292
q11	464	257	266	257
q12	399	211	213	211
q13	17780	3007	3038	3007
q14	244	207	216	207
q15	511	461	451	451
q16	471	386	370	370
q17	983	523	651	523
q18	7399	6805	6541	6541
q19	1395	1103	1149	1103
q20	492	205	210	205
q21	3908	3134	2993	2993
q22	1108	993	1002	993
Total cold run time: 107976 ms
Total hot run time: 39731 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6653	6592	6786	6592
q2	330	235	234	234
q3	2936	2749	2928	2749
q4	2053	1856	1801	1801
q5	5805	5759	5742	5742
q6	217	130	129	129
q7	2247	1830	1814	1814
q8	3357	3562	3561	3561
q9	8988	8776	8882	8776
q10	3542	3516	3548	3516
q11	595	485	491	485
q12	807	620	606	606
q13	8588	3149	3206	3149
q14	309	261	271	261
q15	528	469	461	461
q16	489	444	452	444
q17	1872	1668	1619	1619
q18	8232	7912	7629	7629
q19	1681	1531	1553	1531
q20	2150	1900	1827	1827
q21	5016	5020	5166	5020
q22	1133	1042	1031	1031
Total cold run time: 67528 ms
Total hot run time: 58977 ms

@doris-robot
Copy link

doris-robot commented Jun 25, 2025

TPC-DS: Total hot run time: 197755 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 6beef187de53280974ff0ec7d60036bd5a0118b3, data reload: false

query1	1284	904	920	904
query2	6327	2054	1943	1943
query3	10823	4632	4318	4318
query4	61176	29912	23613	23613
query5	5250	458	470	458
query6	405	182	180	180
query7	5481	303	308	303
query8	310	220	219	219
query9	8368	2570	2551	2551
query10	463	257	248	248
query11	17612	15462	15730	15462
query12	156	107	105	105
query13	1447	452	438	438
query14	10425	6705	6768	6705
query15	205	184	179	179
query16	7186	499	508	499
query17	1176	594	575	575
query18	1882	339	302	302
query19	195	148	152	148
query20	125	109	102	102
query21	196	98	103	98
query22	4735	4642	4646	4642
query23	34954	34254	34196	34196
query24	6253	2938	2911	2911
query25	496	385	426	385
query26	663	168	166	166
query27	2095	351	349	349
query28	4300	2144	2107	2107
query29	687	454	437	437
query30	232	163	157	157
query31	1002	803	843	803
query32	68	57	55	55
query33	471	307	308	307
query34	930	526	501	501
query35	856	720	708	708
query36	1074	960	967	960
query37	113	66	75	66
query38	4049	3974	3947	3947
query39	1526	1495	1484	1484
query40	208	99	104	99
query41	51	51	49	49
query42	118	102	99	99
query43	537	502	504	502
query44	1191	834	808	808
query45	190	172	172	172
query46	1143	727	727	727
query47	2060	1981	1986	1981
query48	478	392	378	378
query49	751	428	405	405
query50	840	438	418	418
query51	7541	7166	7258	7166
query52	103	96	92	92
query53	262	200	190	190
query54	578	498	468	468
query55	83	77	84	77
query56	274	267	291	267
query57	1294	1200	1237	1200
query58	226	211	225	211
query59	3400	3260	3231	3231
query60	292	263	266	263
query61	112	111	124	111
query62	783	695	671	671
query63	211	186	186	186
query64	2186	707	617	617
query65	3258	3238	3219	3219
query66	813	297	307	297
query67	16251	15730	15807	15730
query68	4191	569	555	555
query69	430	257	258	257
query70	1221	1174	1100	1100
query71	318	260	248	248
query72	6072	4087	4089	4087
query73	753	345	348	345
query74	10453	9205	9019	9019
query75	3376	2654	2647	2647
query76	2115	1126	1067	1067
query77	493	266	276	266
query78	10743	9643	9775	9643
query79	1384	593	594	593
query80	886	436	422	422
query81	495	225	222	222
query82	1162	89	87	87
query83	238	147	167	147
query84	273	80	73	73
query85	871	305	297	297
query86	332	264	306	264
query87	4430	4249	4215	4215
query88	3595	2390	2377	2377
query89	422	293	288	288
query90	1950	186	184	184
query91	179	147	145	145
query92	66	49	47	47
query93	1717	551	548	548
query94	785	295	296	295
query95	362	249	256	249
query96	607	280	288	280
query97	3288	3132	3165	3132
query98	214	202	196	196
query99	1630	1299	1295	1295
Total cold run time: 315281 ms
Total hot run time: 197755 ms

@doris-robot
Copy link

doris-robot commented Jun 25, 2025

ClickBench: Total hot run time: 30.47 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 6beef187de53280974ff0ec7d60036bd5a0118b3, data reload: false

query1	0.04	0.03	0.02
query2	0.07	0.03	0.03
query3	0.23	0.07	0.06
query4	1.63	0.10	0.10
query5	0.51	0.51	0.51
query6	1.14	0.73	0.72
query7	0.02	0.02	0.02
query8	0.04	0.03	0.03
query9	0.57	0.50	0.49
query10	0.55	0.55	0.53
query11	0.13	0.10	0.10
query12	0.14	0.11	0.11
query13	0.61	0.60	0.58
query14	0.79	0.78	0.81
query15	0.83	0.82	0.82
query16	0.38	0.39	0.39
query17	1.07	1.08	0.98
query18	0.23	0.22	0.21
query19	1.90	1.81	1.89
query20	0.01	0.02	0.01
query21	15.39	0.60	0.59
query22	2.22	2.51	1.56
query23	17.05	0.98	0.86
query24	3.33	1.63	2.20
query25	0.30	0.22	0.04
query26	0.43	0.15	0.13
query27	0.05	0.04	0.05
query28	8.58	0.52	0.45
query29	12.61	3.25	3.28
query30	0.24	0.05	0.05
query31	2.88	0.41	0.37
query32	3.25	0.47	0.46
query33	2.97	3.01	3.06
query34	16.93	4.48	4.53
query35	4.56	4.46	4.55
query36	0.66	0.49	0.48
query37	0.08	0.06	0.06
query38	0.05	0.04	0.04
query39	0.03	0.03	0.02
query40	0.16	0.13	0.12
query41	0.08	0.02	0.02
query42	0.03	0.02	0.02
query43	0.03	0.03	0.03
Total cold run time: 102.8 s
Total hot run time: 30.47 s

dataroaring
dataroaring approved these changes Jun 26, 2025
View reviewed changes
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@dataroaring dataroaring merged commit 1c02fdb into branch-3.0 Jun 26, 2025
23 of 25 checks passed
@github-actions github-actions bot deleted the auto-pick-52013-branch-3.0 branch June 26, 2025 02:06
koarz pushed a commit to koarz/doris that referenced this pull request Jul 3, 2025
@github-actions @deardeng @koarz
branch-3.0: [fix](cloud) Fix ConcurrentModificationException in clo…
45cbd48 
…ud rebalance apache#52013 (apache#52309)

Cherry-picked from apache#52013

Co-authored-by: deardeng <dengxin@selectdb.com>
@gavinchou gavinchou mentioned this pull request Aug 15, 2025
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@dataroaring dataroaring dataroaring approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@Thearas @doris-robot @dataroaring @deardeng