[fix](doris catalog) FragmentMgr should not cancel virtual doris cluster query by HonestManXin · Pull Request #62135 · apache/doris

HonestManXin · 2026-04-06T10:27:40Z

RuntimeLogger 2026-03-10 06:52:47,187 INFO (mysql-nio-pool-0|263) [MultiFragmentsPipelineTask.cancelExecute():112] Fragment has already been cancelled. Query 125d099a3c8644db-b25df79355a22d7d backend: Backend [id=1770171528024, host=doris-be-0.doris-be-internal.doris.svc.cluster.local, heartbeatPort=9050, alive=true, lastStartTime=2026-03-10 05:13:37, process epoch=1773119617506, isDecommissioned=false, tags: {location=default}], backendStatus: [lastSuccessReportTabletsTime='2026-03-10 06:52:31', lastStreamLoadTime=-1, isQueryDisabled=false, isLoadDisabled=false, currentFragmentNum=0, lastFragmentUpdateTime=1773125544536]
RuntimeLogger 2026-03-10 06:52:47,187 WARN (mysql-nio-pool-0|263) [StmtExecutor.handleQueryWithRetry():927] retry due to exception errCode = 2, detailMessage = (doris-be-0.doris-be-internal.doris.svc.cluster.local)[CANCELLED]Source frontend is not running or restarted. retried 0 times. is rpc error: false, is user error: true.
RuntimeLogger 2026-03-10 06:52:47,187 WARN (mysql-nio-pool-0|263) [StmtExecutor.execute():550] Nereids execute failed. stmt[144, 125d099a3c8644db-b25df79355a22d7d]
org.apache.doris.common.UserException: errCode = 2, detailMessage = (doris-be-0.doris-be-internal.doris.svc.cluster.local)[CANCELLED]Source frontend is not running or restarted
at org.apache.doris.qe.runtime.QueryProcessor.doGetNext(QueryProcessor.java:166)
at org.apache.doris.qe.runtime.QueryProcessor.getNext(QueryProcessor.java:123)
at org.apache.doris.qe.NereidsCoordinator.getNext(NereidsCoordinator.java:217)
at org.apache.doris.qe.StmtExecutor.executeAndSendResult(StmtExecutor.java:1321)
at org.apache.doris.qe.StmtExecutor.handleCacheStmt(StmtExecutor.java:1183)
at org.apache.doris.qe.StmtExecutor.handleQueryStmt(StmtExecutor.java:1254)
at org.apache.doris.qe.StmtExecutor.handleQueryWithRetry(StmtExecutor.java:914)
at org.apache.doris.qe.StmtExecutor.executeByNereids(StmtExecutor.java:818)
at org.apache.doris.qe.StmtExecutor.execute(StmtExecutor.java:541)
at org.apache.doris.qe.StmtExecutor.queryRetry(StmtExecutor.java:500)
at org.apache.doris.qe.StmtExecutor.execute(StmtExecutor.java:485)
at org.apache.doris.qe.ConnectProcessor.executeQuery(ConnectProcessor.java:311)
at org.apache.doris.qe.ConnectProcessor.handleQuery(ConnectProcessor.java:198)
at org.apache.doris.qe.MysqlConnectProcessor.handleQuery(MysqlConnectProcessor.java:231)
at org.apache.doris.qe.MysqlConnectProcessor.dispatch(MysqlConnectProcessor.java:259)
at org.apache.doris.qe.MysqlConnectProcessor.processOnce(MysqlConnectProcessor.java:403)
at org.apache.doris.mysql.ReadListener.lambda$handleEvent$0(ReadListener.java:52)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:833)

Thearas · 2026-04-06T10:27:48Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

HonestManXin · 2026-04-06T10:34:58Z

run buildall

doris-robot · 2026-04-06T11:39:46Z

TPC-H: Total hot run time: 28650 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 4f091d42f189c583da340293760be9d11a18698f, data reload: false

------ Round 1 ----------------------------------
orders	Doris	NULL	NULL	0	0	0	NULL	0	NULL	NULL	2023-12-26 18:27:23	2023-12-26 18:42:55	NULL	utf-8	NULL	NULL	
============================================
q1	17645	3619	3623	3619
q2	q3	10716	874	600	600
q4	4672	471	362	362
q5	7463	1338	1133	1133
q6	183	161	131	131
q7	916	928	771	771
q8	9302	1425	1279	1279
q9	5556	5337	5294	5294
q10	6302	2028	1753	1753
q11	489	276	269	269
q12	636	400	288	288
q13	18071	2780	2142	2142
q14	285	292	262	262
q15	q16	887	859	784	784
q17	1103	1166	747	747
q18	6443	5679	5545	5545
q19	1138	1319	1048	1048
q20	596	416	289	289
q21	5095	2381	1979	1979
q22	496	426	355	355
Total cold run time: 97994 ms
Total hot run time: 28650 ms

----- Round 2, with runtime_filter_mode=off -----
orders	Doris	NULL	NULL	150000000	42	6422171781	NULL	22778155	NULL	NULL	2023-12-26 18:27:23	2023-12-26 18:42:55	NULL	utf-8	NULL	NULL	
============================================
q1	4475	4334	4304	4304
q2	q3	4621	4735	4203	4203
q4	2002	2073	1322	1322
q5	4859	4955	5250	4955
q6	194	168	128	128
q7	1977	1745	1654	1654
q8	3309	3051	3243	3051
q9	8375	8229	8231	8229
q10	4501	4432	4214	4214
q11	569	402	364	364
q12	668	705	491	491
q13	2805	3063	2444	2444
q14	288	305	294	294
q15	q16	787	777	668	668
q17	1336	1269	1228	1228
q18	7825	6969	6992	6969
q19	1179	1157	1129	1129
q20	2340	2375	1952	1952
q21	5962	5227	4717	4717
q22	548	520	424	424
Total cold run time: 58620 ms
Total hot run time: 52740 ms

doris-robot · 2026-04-06T11:51:01Z

TPC-DS: Total hot run time: 177448 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 4f091d42f189c583da340293760be9d11a18698f, data reload: false

query5	4349	672	514	514
query6	344	223	213	213
query7	4266	603	333	333
query8	334	260	231	231
query9	8741	3873	3880	3873
query10	460	366	316	316
query11	6652	5501	5111	5111
query12	197	129	128	128
query13	1313	602	457	457
query14	5652	5154	4728	4728
query14_1	4122	4119	4077	4077
query15	216	206	183	183
query16	1032	457	363	363
query17	1163	795	660	660
query18	2585	497	385	385
query19	224	209	171	171
query20	147	135	131	131
query21	226	146	123	123
query22	14020	14799	14155	14155
query23	17900	16995	16529	16529
query23_1	17197	16767	16752	16752
query24	8186	1719	1336	1336
query24_1	1322	1368	1343	1343
query25	662	494	420	420
query26	1260	316	184	184
query27	2676	612	372	372
query28	4423	1879	1882	1879
query29	949	672	546	546
query30	297	234	196	196
query31	1095	1028	946	946
query32	96	72	69	69
query33	525	354	284	284
query34	1196	1164	658	658
query35	732	765	660	660
query36	1210	1276	1057	1057
query37	152	100	82	82
query38	3093	3011	2986	2986
query39	921	885	859	859
query39_1	831	843	824	824
query40	234	147	129	129
query41	60	58	57	57
query42	122	108	108	108
query43	305	313	277	277
query44	
query45	204	190	193	190
query46	1112	1229	821	821
query47	2350	2326	2231	2231
query48	420	424	297	297
query49	651	524	422	422
query50	742	303	214	214
query51	4305	4243	4216	4216
query52	107	108	96	96
query53	241	274	201	201
query54	322	267	257	257
query55	100	90	87	87
query56	291	321	321	321
query57	1744	1599	1641	1599
query58	298	266	269	266
query59	2866	2990	2768	2768
query60	336	335	320	320
query61	169	152	156	152
query62	677	623	563	563
query63	242	202	191	191
query64	5323	1338	968	968
query65	
query66	1479	468	360	360
query67	24217	24251	24151	24151
query68	
query69	429	333	311	311
query70	1023	1009	970	970
query71	311	280	284	280
query72	2889	2705	2437	2437
query73	793	782	443	443
query74	9845	9739	9521	9521
query75	2743	2617	2313	2313
query76	2297	1131	766	766
query77	398	403	320	320
query78	11274	11415	10754	10754
query79	1499	1143	823	823
query80	1276	596	524	524
query81	495	280	236	236
query82	1268	159	119	119
query83	336	292	259	259
query84	263	155	120	120
query85	992	517	449	449
query86	431	310	348	310
query87	3283	3188	3098	3098
query88	3552	2693	2680	2680
query89	434	381	352	352
query90	1911	177	176	176
query91	182	165	141	141
query92	76	73	70	70
query93	955	963	588	588
query94	616	332	283	283
query95	662	447	334	334
query96	1039	753	324	324
query97	2692	2675	2581	2581
query98	240	235	228	228
query99	1095	1077	930	930
Total cold run time: 258796 ms
Total hot run time: 177448 ms

hello-stephen · 2026-04-06T12:48:37Z

BE UT Coverage Report

Increment line coverage 0.00% (0/14) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	53.00% (20084/37892)
Line Coverage	36.56% (188597/515840)
Region Coverage	32.80% (146395/446291)
Branch Coverage	33.97% (64128/188796)

hello-stephen · 2026-04-06T14:28:56Z

BE Regression && UT Coverage Report

Increment line coverage 71.43% (10/14) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	71.81% (26647/37108)
Line Coverage	54.72% (281426/514259)
Region Coverage	51.80% (233315/450425)
Branch Coverage	53.27% (100883/189378)

HonestManXin · 2026-04-07T01:51:41Z

run nonConcurrent

HonestManXin · 2026-04-07T03:20:58Z

run buildall

…ter query

HonestManXin · 2026-04-07T06:05:55Z

run buildall

HonestManXin · 2026-04-07T07:11:41Z

run performance

HonestManXin · 2026-04-07T08:23:33Z

run cloud_p0

hello-stephen · 2026-04-07T08:51:16Z

BE Regression && UT Coverage Report

Increment line coverage 78.80% (171/217) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	71.93% (26728/37158)
Line Coverage	54.90% (283035/515527)
Region Coverage	51.79% (233630/451125)
Branch Coverage	53.34% (101240/189788)

hello-stephen · 2026-04-07T09:58:16Z

BE Regression && UT Coverage Report

Increment line coverage 80.34% (188/234) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	71.93% (26728/37158)
Line Coverage	54.90% (283035/515527)
Region Coverage	51.79% (233630/451125)
Branch Coverage	53.34% (101240/189788)

yiguolei · 2026-04-07T10:13:17Z

be/src/runtime/fragment_mgr.cpp

+                        QuerySource actual_query_source = query_source;
+                        if (query_source == QuerySource::INTERNAL_FRONTEND &&
+                            !_exec_env->get_running_frontends().contains(params.coord)) {
+                            actual_query_source = QuerySource::EXTERNAL_FRONTEND;


why not set the query source during send fragment in FE??

github-actions · 2026-04-07T11:03:37Z

PR approved by at least one committer and no changes requested.

github-actions · 2026-04-07T11:03:40Z

PR approved by anyone and no changes requested.

…ter query (#62135) RuntimeLogger 2026-03-10 06:52:47,187 INFO (mysql-nio-pool-0|263) [MultiFragmentsPipelineTask.cancelExecute():112] Fragment has already been cancelled. Query 125d099a3c8644db-b25df79355a22d7d backend: Backend [id=1770171528024, host=[doris-be-0.doris-be-internal.doris.svc.cluster.local](http://doris-be-0.doris-be-internal.doris.svc.cluster.local/), heartbeatPort=9050, alive=true, lastStartTime=2026-03-10 05:13:37, process epoch=1773119617506, isDecommissioned=false, tags: {location=default}], backendStatus: [lastSuccessReportTabletsTime='2026-03-10 06:52:31', lastStreamLoadTime=-1, isQueryDisabled=false, isLoadDisabled=false, currentFragmentNum=0, lastFragmentUpdateTime=1773125544536] RuntimeLogger 2026-03-10 06:52:47,187 WARN (mysql-nio-pool-0|263) [StmtExecutor.handleQueryWithRetry():927] retry due to exception errCode = 2, detailMessage = ([doris-be-0.doris-be-internal.doris.svc.cluster.local](http://doris-be-0.doris-be-internal.doris.svc.cluster.local/))[CANCELLED]Source frontend is not running or restarted. retried 0 times. is rpc error: false, is user error: true. RuntimeLogger 2026-03-10 06:52:47,187 WARN (mysql-nio-pool-0|263) [StmtExecutor.execute():550] Nereids execute failed. stmt[144, 125d099a3c8644db-b25df79355a22d7d] org.apache.doris.common.UserException: errCode = 2, detailMessage = ([doris-be-0.doris-be-internal.doris.svc.cluster.local](http://doris-be-0.doris-be-internal.doris.svc.cluster.local/))[CANCELLED]Source frontend is not running or restarted at org.apache.doris.qe.runtime.QueryProcessor.doGetNext(QueryProcessor.java:166) at org.apache.doris.qe.runtime.QueryProcessor.getNext(QueryProcessor.java:123) at org.apache.doris.qe.NereidsCoordinator.getNext(NereidsCoordinator.java:217) at org.apache.doris.qe.StmtExecutor.executeAndSendResult(StmtExecutor.java:1321) at org.apache.doris.qe.StmtExecutor.handleCacheStmt(StmtExecutor.java:1183) at org.apache.doris.qe.StmtExecutor.handleQueryStmt(StmtExecutor.java:1254) at org.apache.doris.qe.StmtExecutor.handleQueryWithRetry(StmtExecutor.java:914) at org.apache.doris.qe.StmtExecutor.executeByNereids(StmtExecutor.java:818) at org.apache.doris.qe.StmtExecutor.execute(StmtExecutor.java:541) at org.apache.doris.qe.StmtExecutor.queryRetry(StmtExecutor.java:500) at org.apache.doris.qe.StmtExecutor.execute(StmtExecutor.java:485) at org.apache.doris.qe.ConnectProcessor.executeQuery(ConnectProcessor.java:311) at org.apache.doris.qe.ConnectProcessor.handleQuery(ConnectProcessor.java:198) at org.apache.doris.qe.MysqlConnectProcessor.handleQuery(MysqlConnectProcessor.java:231) at org.apache.doris.qe.MysqlConnectProcessor.dispatch(MysqlConnectProcessor.java:259) at org.apache.doris.qe.MysqlConnectProcessor.processOnce(MysqlConnectProcessor.java:403) at org.apache.doris.mysql.ReadListener.lambda$handleEvent$0(ReadListener.java:52) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) at java.base/java.lang.Thread.run(Thread.java:833)

…l doris cluster query #62135 (#62213) Cherry-picked from #62135 Co-authored-by: HonestManXin <HonestManXin@gmail.com>

…ter query (apache#62135) RuntimeLogger 2026-03-10 06:52:47,187 INFO (mysql-nio-pool-0|263) [MultiFragmentsPipelineTask.cancelExecute():112] Fragment has already been cancelled. Query 125d099a3c8644db-b25df79355a22d7d backend: Backend [id=1770171528024, host=[doris-be-0.doris-be-internal.doris.svc.cluster.local](http://doris-be-0.doris-be-internal.doris.svc.cluster.local/), heartbeatPort=9050, alive=true, lastStartTime=2026-03-10 05:13:37, process epoch=1773119617506, isDecommissioned=false, tags: {location=default}], backendStatus: [lastSuccessReportTabletsTime='2026-03-10 06:52:31', lastStreamLoadTime=-1, isQueryDisabled=false, isLoadDisabled=false, currentFragmentNum=0, lastFragmentUpdateTime=1773125544536] RuntimeLogger 2026-03-10 06:52:47,187 WARN (mysql-nio-pool-0|263) [StmtExecutor.handleQueryWithRetry():927] retry due to exception errCode = 2, detailMessage = ([doris-be-0.doris-be-internal.doris.svc.cluster.local](http://doris-be-0.doris-be-internal.doris.svc.cluster.local/))[CANCELLED]Source frontend is not running or restarted. retried 0 times. is rpc error: false, is user error: true. RuntimeLogger 2026-03-10 06:52:47,187 WARN (mysql-nio-pool-0|263) [StmtExecutor.execute():550] Nereids execute failed. stmt[144, 125d099a3c8644db-b25df79355a22d7d] org.apache.doris.common.UserException: errCode = 2, detailMessage = ([doris-be-0.doris-be-internal.doris.svc.cluster.local](http://doris-be-0.doris-be-internal.doris.svc.cluster.local/))[CANCELLED]Source frontend is not running or restarted at org.apache.doris.qe.runtime.QueryProcessor.doGetNext(QueryProcessor.java:166) at org.apache.doris.qe.runtime.QueryProcessor.getNext(QueryProcessor.java:123) at org.apache.doris.qe.NereidsCoordinator.getNext(NereidsCoordinator.java:217) at org.apache.doris.qe.StmtExecutor.executeAndSendResult(StmtExecutor.java:1321) at org.apache.doris.qe.StmtExecutor.handleCacheStmt(StmtExecutor.java:1183) at org.apache.doris.qe.StmtExecutor.handleQueryStmt(StmtExecutor.java:1254) at org.apache.doris.qe.StmtExecutor.handleQueryWithRetry(StmtExecutor.java:914) at org.apache.doris.qe.StmtExecutor.executeByNereids(StmtExecutor.java:818) at org.apache.doris.qe.StmtExecutor.execute(StmtExecutor.java:541) at org.apache.doris.qe.StmtExecutor.queryRetry(StmtExecutor.java:500) at org.apache.doris.qe.StmtExecutor.execute(StmtExecutor.java:485) at org.apache.doris.qe.ConnectProcessor.executeQuery(ConnectProcessor.java:311) at org.apache.doris.qe.ConnectProcessor.handleQuery(ConnectProcessor.java:198) at org.apache.doris.qe.MysqlConnectProcessor.handleQuery(MysqlConnectProcessor.java:231) at org.apache.doris.qe.MysqlConnectProcessor.dispatch(MysqlConnectProcessor.java:259) at org.apache.doris.qe.MysqlConnectProcessor.processOnce(MysqlConnectProcessor.java:403) at org.apache.doris.mysql.ReadListener.lambda$handleEvent$0(ReadListener.java:52) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) at java.base/java.lang.Thread.run(Thread.java:833)

…ter query (#62135) (#62281) Cherry-picked from #62135

HonestManXin force-pushed the fix_cross_cluster_cancel_query branch from 8a1bd92 to 4f091d4 Compare April 6, 2026 10:32

HonestManXin force-pushed the fix_cross_cluster_cancel_query branch 2 times, most recently from b0e139b to 43b1e6b Compare April 7, 2026 03:00

[fix](doris catalog) FragmentMgr should not cancel virtual doris clus…

fc90d91

…ter query

HonestManXin force-pushed the fix_cross_cluster_cancel_query branch from 43b1e6b to fc90d91 Compare April 7, 2026 05:59

yiguolei added dev/4.0.x dev/4.1.x labels Apr 7, 2026

yiguolei reviewed Apr 7, 2026

View reviewed changes

yiguolei approved these changes Apr 7, 2026

View reviewed changes

github-actions bot added approved Indicates a PR has been approved by one committer. reviewed labels Apr 7, 2026

morningman approved these changes Apr 8, 2026

View reviewed changes

yiguolei merged commit ebf08f2 into apache:master Apr 8, 2026
32 of 35 checks passed

github-actions bot added the dev/4.0.x-conflict label Apr 8, 2026

github-actions bot mentioned this pull request Apr 8, 2026

branch-4.1: [fix](doris catalog) FragmentMgr should not cancel virtual doris cluster query #62135 #62213

Merged

yiguolei pushed a commit that referenced this pull request Apr 9, 2026

branch-4.1: [fix](doris catalog) FragmentMgr should not cancel virtua…

089c365

…l doris cluster query #62135 (#62213) Cherry-picked from #62135 Co-authored-by: HonestManXin <HonestManXin@gmail.com>

yiguolei added dev/4.1.0-merged and removed dev/4.1.x labels Apr 9, 2026

HonestManXin mentioned this pull request Apr 9, 2026

[fix](doris catalog) FragmentMgr should not cancel virtual doris cluster query (#62135) #62281

Merged

yiguolei pushed a commit that referenced this pull request Apr 10, 2026

[fix](doris catalog) FragmentMgr should not cancel virtual doris clus…

09827b8

…ter query (#62135) (#62281) Cherry-picked from #62135

yiguolei added dev/4.0.5-merged and removed dev/4.0.x dev/4.0.x-conflict labels Apr 10, 2026

Conversation

HonestManXin commented Apr 6, 2026

Uh oh!

Thearas commented Apr 6, 2026

Uh oh!

HonestManXin commented Apr 6, 2026

Uh oh!

doris-robot commented Apr 6, 2026

Uh oh!

doris-robot commented Apr 6, 2026

Uh oh!

hello-stephen commented Apr 6, 2026

BE UT Coverage Report

Uh oh!

hello-stephen commented Apr 6, 2026

BE Regression && UT Coverage Report

Uh oh!

HonestManXin commented Apr 7, 2026

Uh oh!

HonestManXin commented Apr 7, 2026

Uh oh!

HonestManXin commented Apr 7, 2026

Uh oh!

HonestManXin commented Apr 7, 2026

Uh oh!

HonestManXin commented Apr 7, 2026

Uh oh!

hello-stephen commented Apr 7, 2026

BE Regression && UT Coverage Report

Uh oh!

hello-stephen commented Apr 7, 2026

BE Regression && UT Coverage Report

Uh oh!

yiguolei Apr 7, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Apr 7, 2026

Uh oh!

github-actions bot commented Apr 7, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants