Skip to content

[fix](cloud) fix correct pipeline task number calculation in cloud mode #57129

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
yiguolei merged 3 commits into from
Oct 23, 2025
Merged

[fix](cloud) fix correct pipeline task number calculation in cloud mode #57129

yiguolei merged 3 commits into from
Oct 23, 2025

Conversation

@liaoxin01
Copy link
Contributor

@liaoxin01 liaoxin01 commented Oct 19, 2025 •
edited
Loading

What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:
This fixes incorrect resource calculation that could lead to improper pipeline task scheduling in cloud deployments.
Fix pipeline task calculation to use current compute group backends instead of all repository backends in cloud mode. Use getBackendsByCurrentCluster() instead of getAllBackendsByAllCluster().

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@hello-stephen
Copy link
Contributor

hello-stephen commented Oct 19, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@liaoxin01
Copy link
Contributor Author

liaoxin01 commented Oct 19, 2025

run buildall

@liaoxin01 liaoxin01 force-pushed the fix_pipeline_task branch 2 times, most recently from 0e3e6d6 to 22fafdd Compare October 19, 2025 09:29
@liaoxin01
Copy link
Contributor Author

liaoxin01 commented Oct 19, 2025

run buildall

@hello-stephen
Copy link
Contributor

hello-stephen commented Oct 19, 2025

FE Regression Coverage Report

Increment line coverage 80.00% (12/15) 🎉
Increment coverage report
Complete coverage report

dataroaring
dataroaring previously approved these changes Oct 20, 2025
View reviewed changes
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Oct 20, 2025
@github-actions
Copy link
Contributor

github-actions bot commented Oct 20, 2025

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Contributor

github-actions bot commented Oct 20, 2025

PR approved by anyone and no changes requested.

@liaoxin01
Copy link
Contributor Author

liaoxin01 commented Oct 20, 2025

run buildall

@github-actions github-actions bot removed the approved Indicates a PR has been approved by one committer. label Oct 20, 2025
@liaoxin01
Copy link
Contributor Author

liaoxin01 commented Oct 20, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Oct 20, 2025

ClickBench: Total hot run time: 30.99 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 6d4ac379a6de5aba7081b948c8b0efc7db165bb2, data reload: false

query1	0.06	0.05	0.05
query2	0.09	0.06	0.05
query3	0.25	0.08	0.08
query4	1.61	0.12	0.12
query5	0.28	0.27	0.25
query6	1.19	0.67	0.66
query7	0.03	0.03	0.03
query8	0.06	0.04	0.04
query9	0.64	0.53	0.54
query10	0.60	0.60	0.59
query11	0.17	0.11	0.12
query12	0.16	0.13	0.12
query13	0.64	0.62	0.62
query14	1.04	1.05	1.03
query15	0.90	0.87	0.87
query16	0.42	0.40	0.42
query17	1.07	1.08	1.08
query18	0.22	0.21	0.21
query19	2.02	1.84	1.86
query20	0.02	0.02	0.01
query21	15.41	1.00	0.60
query22	0.78	1.26	0.74
query23	14.76	1.48	0.66
query24	7.70	1.24	0.83
query25	0.50	0.31	0.12
query26	0.66	0.16	0.13
query27	0.08	0.06	0.06
query28	9.51	1.37	0.95
query29	12.58	4.06	3.36
query30	0.31	0.14	0.12
query31	2.83	0.61	0.40
query32	3.25	0.57	0.48
query33	3.21	3.21	3.18
query34	16.13	5.48	4.84
query35	5.04	4.96	4.92
query36	0.69	0.53	0.51
query37	0.11	0.08	0.08
query38	0.07	0.05	0.05
query39	0.04	0.03	0.02
query40	0.18	0.16	0.15
query41	0.09	0.04	0.03
query42	0.04	0.03	0.03
query43	0.04	0.04	0.04
Total cold run time: 105.48 s
Total hot run time: 30.99 s

@hello-stephen
Copy link
Contributor

hello-stephen commented Oct 20, 2025

FE Regression Coverage Report

Increment line coverage 69.23% (9/13) 🎉
Increment coverage report
Complete coverage report

@liaoxin01
Copy link
Contributor Author

liaoxin01 commented Oct 21, 2025

run buildall

@liaoxin01
Copy link
Contributor Author

liaoxin01 commented Oct 21, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Oct 21, 2025

TPC-DS: Total hot run time: 190418 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 6bc05b0351a35c9bf07e59ef7cc90ee23881db8b, data reload: false

query1	1050	428	420	420
query2	6580	1700	1752	1700
query3	6751	233	227	227
query4	26645	23554	23585	23554
query5	5053	653	474	474
query6	336	246	234	234
query7	4654	493	298	298
query8	319	272	276	272
query9	8728	2602	2628	2602
query10	521	373	285	285
query11	15966	15269	14786	14786
query12	182	115	113	113
query13	1665	547	447	447
query14	12582	9288	9162	9162
query15	216	193	214	193
query16	7684	687	504	504
query17	1606	787	670	670
query18	2248	451	363	363
query19	242	224	199	199
query20	145	134	130	130
query21	228	144	128	128
query22	4715	4698	4619	4619
query23	35153	33585	34274	33585
query24	8239	2521	2543	2521
query25	650	581	530	530
query26	1387	288	183	183
query27	4038	523	375	375
query28	4533	2297	2227	2227
query29	855	689	536	536
query30	312	250	226	226
query31	975	905	897	897
query32	87	70	70	70
query33	579	407	375	375
query34	844	910	565	565
query35	854	864	771	771
query36	976	995	901	901
query37	124	108	83	83
query38	3556	3607	3544	3544
query39	1472	1423	1417	1417
query40	218	122	114	114
query41	70	69	60	60
query42	119	109	108	108
query43	493	491	469	469
query44	1208	729	735	729
query45	186	181	170	170
query46	878	988	629	629
query47	1746	1762	1701	1701
query48	416	432	319	319
query49	770	509	428	428
query50	652	688	403	403
query51	3930	3897	3816	3816
query52	117	115	99	99
query53	242	269	196	196
query54	631	593	521	521
query55	94	83	84	83
query56	322	327	318	318
query57	1156	1215	1125	1125
query58	280	279	279	279
query59	2592	2656	2554	2554
query60	359	349	331	331
query61	159	181	153	153
query62	799	744	661	661
query63	243	194	195	194
query64	4388	1152	862	862
query65	4032	3955	3993	3955
query66	1086	434	337	337
query67	15406	15238	14999	14999
query68	9706	935	602	602
query69	504	321	296	296
query70	1412	1222	1222	1222
query71	508	332	321	321
query72	5553	4999	4861	4861
query73	718	573	357	357
query74	8997	9144	8905	8905
query75	4441	3360	2820	2820
query76	3860	1177	741	741
query77	1017	387	326	326
query78	9507	9874	8903	8903
query79	2078	829	599	599
query80	696	591	513	513
query81	480	269	230	230
query82	459	167	143	143
query83	303	272	252	252
query84	304	113	96	96
query85	911	472	433	433
query86	343	321	297	297
query87	3671	3775	3737	3737
query88	3195	2276	2262	2262
query89	409	313	295	295
query90	2010	208	228	208
query91	162	167	137	137
query92	82	68	64	64
query93	1179	973	646	646
query94	689	453	343	343
query95	406	316	309	309
query96	496	579	280	280
query97	2941	3003	2892	2892
query98	237	223	217	217
query99	1450	1466	1312	1312
Total cold run time: 284066 ms
Total hot run time: 190418 ms

@doris-robot
Copy link

doris-robot commented Oct 21, 2025

ClickBench: Total hot run time: 28.54 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 6bc05b0351a35c9bf07e59ef7cc90ee23881db8b, data reload: false

query1	0.06	0.05	0.05
query2	0.10	0.04	0.06
query3	0.25	0.08	0.09
query4	1.61	0.12	0.12
query5	0.27	0.26	0.25
query6	1.15	0.63	0.65
query7	0.03	0.02	0.03
query8	0.05	0.05	0.04
query9	0.63	0.53	0.52
query10	0.57	0.58	0.57
query11	0.16	0.11	0.11
query12	0.16	0.12	0.12
query13	0.63	0.63	0.65
query14	1.04	1.04	1.04
query15	0.88	0.85	0.86
query16	0.39	0.40	0.39
query17	1.03	1.11	1.04
query18	0.21	0.20	0.20
query19	1.91	1.89	1.83
query20	0.02	0.01	0.02
query21	15.43	0.18	0.13
query22	5.01	0.06	0.05
query23	15.74	0.27	0.11
query24	2.43	1.00	0.38
query25	0.08	0.06	0.06
query26	0.14	0.14	0.13
query27	0.07	0.06	0.06
query28	4.18	1.18	0.94
query29	12.57	3.96	3.28
query30	0.28	0.14	0.13
query31	2.83	0.61	0.39
query32	3.24	0.56	0.48
query33	3.11	3.18	3.16
query34	16.13	5.46	4.88
query35	4.87	4.92	4.93
query36	0.71	0.51	0.51
query37	0.10	0.07	0.07
query38	0.06	0.04	0.04
query39	0.04	0.04	0.03
query40	0.18	0.17	0.15
query41	0.09	0.03	0.03
query42	0.04	0.03	0.03
query43	0.04	0.04	0.04
Total cold run time: 98.52 s
Total hot run time: 28.54 s

@hello-stephen
Copy link
Contributor

hello-stephen commented Oct 21, 2025

FE Regression Coverage Report

Increment line coverage 69.23% (9/13) 🎉
Increment coverage report
Complete coverage report

1 similar comment
@hello-stephen
Copy link
Contributor

hello-stephen commented Oct 21, 2025

FE Regression Coverage Report

Increment line coverage 69.23% (9/13) 🎉
Increment coverage report
Complete coverage report

dataroaring
dataroaring approved these changes Oct 23, 2025
View reviewed changes
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Oct 23, 2025
@github-actions
Copy link
Contributor

github-actions bot commented Oct 23, 2025

PR approved by at least one committer and no changes requested.

@hello-stephen
Copy link
Contributor

hello-stephen commented Oct 23, 2025

FE Regression Coverage Report

Increment line coverage 69.23% (9/13) 🎉
Increment coverage report
Complete coverage report

deardeng
deardeng approved these changes Oct 23, 2025
View reviewed changes
Copy link
Contributor

@deardeng deardeng left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@yiguolei yiguolei merged commit e1884e4 into apache:master Oct 23, 2025
30 of 31 checks passed
github-actions bot pushed a commit that referenced this pull request Oct 23, 2025
@liaoxin01
[fix](cloud) fix correct pipeline task number calculation in cloud mo…
91a6ae6 
…de (#57129)

### What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:
This fixes incorrect resource calculation that could lead to improper
pipeline task scheduling in cloud deployments.
Fix pipeline task calculation to use current compute group backends
instead of all repository backends in cloud mode. Use
getBackendsByCurrentCluster() instead of getAllBackendsByAllCluster().


### Release note

None

### Check List (For Author)

- Test <!-- At least one of them must be included. -->
    - [ ] Regression test
    - [x] Unit Test
    - [ ] Manual test (add detailed scripts or steps below)
    - [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
        - [ ] Previous test can cover this change.
        - [ ] No code files have been changed.
        - [ ] Other reason <!-- Add your reason?  -->

- Behavior changed:
    - [x] No.
    - [ ] Yes. <!-- Explain the behavior change -->

- Does this need documentation?
    - [x] No.
- [ ] Yes. <!-- Add document PR link here. eg:
apache/doris-website#1214 -->

### Check List (For Reviewer who merge this PR)

- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR
should merge into -->
@github-actions github-actions bot mentioned this pull request Oct 23, 2025
Merged
github-actions bot pushed a commit that referenced this pull request Oct 23, 2025
@liaoxin01
[fix](cloud) fix correct pipeline task number calculation in cloud mo…
623d739 
…de (#57129)

### What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:
This fixes incorrect resource calculation that could lead to improper
pipeline task scheduling in cloud deployments.
Fix pipeline task calculation to use current compute group backends
instead of all repository backends in cloud mode. Use
getBackendsByCurrentCluster() instead of getAllBackendsByAllCluster().


### Release note

None

### Check List (For Author)

- Test <!-- At least one of them must be included. -->
    - [ ] Regression test
    - [x] Unit Test
    - [ ] Manual test (add detailed scripts or steps below)
    - [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
        - [ ] Previous test can cover this change.
        - [ ] No code files have been changed.
        - [ ] Other reason <!-- Add your reason?  -->

- Behavior changed:
    - [x] No.
    - [ ] Yes. <!-- Explain the behavior change -->

- Does this need documentation?
    - [x] No.
- [ ] Yes. <!-- Add document PR link here. eg:
apache/doris-website#1214 -->

### Check List (For Reviewer who merge this PR)

- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR
should merge into -->
@github-actions github-actions bot mentioned this pull request Oct 23, 2025
Merged
yiguolei pushed a commit that referenced this pull request Oct 23, 2025
@github-actions @liaoxin01
branch-4.0: [fix](cloud) fix correct pipeline task number calculation…
c31bf41 
… in cloud mode #57129 (#57262)

Cherry-picked from #57129

Co-authored-by: Xin Liao <liaoxin@selectdb.com>
dwdwqfwe pushed a commit to dwdwqfwe/doris that referenced this pull request Oct 24, 2025
@liaoxin01 @dwdwqfwe
[fix](cloud) fix correct pipeline task number calculation in cloud mo…
2cd64e2 
…de (apache#57129)

### What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:
This fixes incorrect resource calculation that could lead to improper
pipeline task scheduling in cloud deployments.
Fix pipeline task calculation to use current compute group backends
instead of all repository backends in cloud mode. Use
getBackendsByCurrentCluster() instead of getAllBackendsByAllCluster().


### Release note

None

### Check List (For Author)

- Test <!-- At least one of them must be included. -->
    - [ ] Regression test
    - [x] Unit Test
    - [ ] Manual test (add detailed scripts or steps below)
    - [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
        - [ ] Previous test can cover this change.
        - [ ] No code files have been changed.
        - [ ] Other reason <!-- Add your reason?  -->

- Behavior changed:
    - [x] No.
    - [ ] Yes. <!-- Explain the behavior change -->

- Does this need documentation?
    - [x] No.
- [ ] Yes. <!-- Add document PR link here. eg:
apache/doris-website#1214 -->

### Check List (For Reviewer who merge this PR)

- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR
should merge into -->
morrySnow pushed a commit that referenced this pull request Oct 30, 2025
@github-actions @liaoxin01
branch-3.1: [fix](cloud) fix correct pipeline task number calculation…
ad3cf0f 
… in cloud mode #57129 (#57261)

Cherry-picked from #57129

Co-authored-by: Xin Liao <liaoxin@selectdb.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@dataroaring dataroaring dataroaring approved these changes

@gavinchou gavinchou Awaiting requested review from gavinchou gavinchou is a code owner

@w41ter w41ter Awaiting requested review from w41ter w41ter is a code owner

+1 more reviewer

@deardeng deardeng deardeng approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. dev/3.1.3-merged dev/4.0.1-merged reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

7 participants

@liaoxin01 @hello-stephen @doris-robot @deardeng @dataroaring @yiguolei @morrySnow