[fix](cloud) Exclude decommissioning BE from BE selection#65221
Open
deardeng wants to merge 4 commits into
Open
[fix](cloud) Exclude decommissioning BE from BE selection#65221deardeng wants to merge 4 commits into
deardeng wants to merge 4 commits into
Conversation
### What problem does this PR solve?
Issue Number: None
Related PR: None
Problem Summary: Cloud decommissioning backends can still be alive before they reach the final decommissioned state. These backends could still be selected by CloudReplica routing, routine load assignment, and Kafka proxy backend selection. This change treats decommissioning backends the same as decommissioned backends in those selection paths, while preserving current master behavior such as direct multi-replica hashing and routine load blacklist stale-backend cleanup.
### Release note
Cloud backend selection avoids routing new replica, routine load, and Kafka proxy work to decommissioning backends.
### Check List (For Author)
- Test: Unit Test
- ./run-fe-ut.sh --run org.apache.doris.cloud.catalog.CloudReplicaTest,org.apache.doris.cloud.load.CloudRoutineLoadManagerTest,org.apache.doris.datasource.kafka.KafkaUtilTest,org.apache.doris.load.routineload.RoutineLoadTaskSchedulerTest
- mvn checkstyle:check -pl fe-core
- git diff --cached --check
- Behavior changed: Yes. Cloud BE selection now excludes decommissioning/decommissioned backends in the affected routing and load paths.
- Does this need documentation: No
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
Collaborator
Author
|
/review |
Collaborator
Author
|
run buildall |
Contributor
|
Codex automated review failed and did not complete. Error: Your access token could not be refreshed because your refresh token was already used. Please log out and sign in again. Please inspect the workflow logs and rerun the review after the underlying issue is resolved. |
Contributor
TPC-H: Total hot run time: 29334 ms |
Contributor
TPC-DS: Total hot run time: 174858 ms |
Contributor
ClickBench: Total hot run time: 25.04 s |
Contributor
FE Regression Coverage ReportIncrement line coverage |
Collaborator
Author
|
/review |
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Review summary for PR 65221
I found no substantiated issue requiring inline comments.
Critical checkpoint conclusions:
- Goal and behavior: the patch consistently excludes decommissioning/decommissioned cloud backends from the changed CloudReplica routing, cloud routine-load backend assignment, routine-load submit guard, and Kafka proxy backend selection paths while preserving the existing fallback behavior when only decommissioned backends remain.
- Scope: the implementation is focused on backend selection predicates plus targeted unit coverage.
- Concurrency/lifecycle: no new shared lifecycle state is introduced; the changed code reads existing Backend state during selection and submit-time validation.
- Config/session propagation: no new config items or session variables are added.
- Compatibility/persistence/protocol: no storage format, edit-log, Thrift, or rolling-upgrade protocol change was introduced.
- Parallel paths: checked colocate, primary/secondary cached replica selection, multi-replica hashing, routine-load allocation/submit, and Kafka blacklist fallback paths. I also checked Kinesis metadata selection as a nearby parallel helper; Kinesis routine load rejects cloud mode, so it is not a missed cloud decommissioning path for this PR.
- Tests: the added/updated unit tests cover the changed decommissioning filters for CloudReplica, CloudRoutineLoadManager, KafkaUtil, and RoutineLoadTaskScheduler. I did not run the FE unit tests because this runner is missing thirdparty/installed/bin/protoc, which the FE AGENTS instructions require before build/test execution.
- Observability/performance: the added debug logging is guarded by isDebugEnabled, and the new filters do not add a concerning cost beyond existing backend-list scans.
User focus: no additional user-provided review focus was present.
Subagent conclusions:
- optimizer-rewrite: no candidates in the initial pass; convergence round C1 returned NO_NEW_VALUABLE_FINDINGS for the final empty inline comment set.
- tests-session-config: no candidates in the initial pass; convergence round C1 returned NO_NEW_VALUABLE_FINDINGS for the final empty inline comment set.
- No subagent candidates were accepted, dismissed, or merged as duplicates because none were appended to the shared ledger.
Validation performed:
- Loaded the review prompt, repo code-review skill, required AGENTS files, existing review thread/comment context, review focus, changed-file list, and shared ledger before diff inspection.
- Used GitHub PR metadata and patch data as the authoritative scope for the shallow checkout.
- Ran git diff --check FETCH_HEAD HEAD over the eight PR files; it passed with no output.
### What problem does this PR solve?
Issue Number: None
Related PR: None
Problem Summary: Routine load backend selection treated decommissioning backends as unavailable in all deploy modes. In shared-nothing mode this could prevent routine load tasks or Kafka meta lookup from using an otherwise load-available backend. This change only filters decommissioning backends in cloud mode and keeps the Kafka meta blacklist fallback available while still removing stale blacklist entries.
### Release note
Routine load backend selection no longer excludes decommissioning backends outside cloud mode.
### Check List (For Author)
- Test: Unit Test
- ./run-fe-ut.sh --run org.apache.doris.datasource.kafka.KafkaUtilTest,org.apache.doris.load.routineload.RoutineLoadTaskSchedulerTest
- cd fe && mvn checkstyle:check -pl fe-core
- Behavior changed: Yes. Decommissioning backends remain eligible for routine load backend selection outside cloud mode.
- Does this need documentation: No
### What problem does this PR solve?
Issue Number: None
Related PR: None
Problem Summary: Cloud routine load filtered candidate backends by Backend.isDecommissioning() unconditionally. The decommissioning state should only affect cloud mode, matching other load backend selection paths. This change guards the decommissioning check with Config.isCloudMode() so non-cloud paths do not exclude backends based on this cloud-only state while cloud mode behavior is unchanged.
### Release note
None
### Check List (For Author)
- Test: Manual test
- git diff --check
- cd fe && mvn checkstyle:check -pl fe-core
- Behavior changed: Yes. Non-cloud mode no longer filters routine load candidate backends by the cloud-only decommissioning state.
- Does this need documentation: No
Collaborator
Author
|
run buildall |
### What problem does this PR solve?
Issue Number: None
Related PR: None
Problem Summary: The branch included routine load and Kafka backend selection changes in CloudRoutineLoadManager, KafkaUtil, and RoutineLoadTaskScheduler, along with their unit tests. These changes are now covered by another PR, so this commit removes them from this branch and leaves the CloudReplica change as the remaining scope.
### Release note
None
### Check List (For Author)
- Test: Unit Test / Manual test
- git diff --check
- cd fe && mvn checkstyle:check -pl fe-core
- ./run-fe-ut.sh --run org.apache.doris.cloud.catalog.CloudReplicaTest
- Behavior changed: No. This removes duplicate changes from this PR branch.
- Does this need documentation: No
Collaborator
Author
|
run buildall |
Contributor
TPC-H: Total hot run time: 29551 ms |
Contributor
TPC-DS: Total hot run time: 173994 ms |
Contributor
ClickBench: Total hot run time: 25.19 s |
Contributor
FE Regression Coverage ReportIncrement line coverage |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem Summary: Cloud decommissioning backends can still be alive before they reach the final decommissioned state. These backends could still be selected by CloudReplica routing, routine load assignment, and Kafka proxy backend selection. This change treats decommissioning backends the same as decommissioned backends in those selection paths, while preserving current master behavior such as direct multi-replica hashing and routine load blacklist stale-backend cleanup.
Release note
Cloud backend selection avoids routing new replica, routine load, and Kafka proxy work to decommissioning backends.
Check List (For Author)
What problem does this PR solve?
Issue Number: close #xxx
Related PR: #xxx
Problem Summary:
Release note
None
Check List (For Author)
Test
Behavior changed:
Does this need documentation?
Check List (For Reviewer who merge this PR)