[chore](distribute plan) add Javadoc to UnassignedJob.computeAssignedJobs and related methods#63743
[chore](distribute plan) add Javadoc to UnassignedJob.computeAssignedJobs and related methods#63743morrySnow wants to merge 1 commit into
Conversation
…and related methods Add comprehensive Javadoc comments to the worker job abstraction layer in the Nereids distributed plan optimizer. The comments document: - UnassignedJob interface and its computeAssignedJobs contract - Two-phase parallelization strategy in AbstractUnassignedScanJob (cross-machine replica selection + intra-machine instance split) - Compute logic for each concrete subclass (shuffle, gather, scan, etc.) - Protected helper methods governing parallelism decisions and fallback Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
|
/review |
There was a problem hiding this comment.
Reviewed the documentation-only PR that adds Javadocs for Nereids distributed worker jobs. I found two accuracy issues where the new comments describe behavior that the implementation does not actually provide.
Critical checkpoint conclusions:
- Goal/test: The goal appears to be documenting UnassignedJob.computeAssignedJobs and related methods. No behavior changed and no tests are required for runtime behavior, but the documentation needs to match the code.
- Scope: The PR is focused and small in behavior terms, but some descriptions overstate or misstate existing control flow.
- Concurrency/lifecycle/config/compatibility/persistence/data writes: No code behavior changes in these areas.
- Parallel paths: I checked the related scan, gather, shuffle, and specified-instance paths; the reported issues are distinct documentation inaccuracies in their respective methods.
- Test coverage/results: No tests or result files were changed; acceptable for comment-only changes once the comments are accurate.
- Observability/performance: No runtime changes.
- User focus: No additional user-provided review focus was supplied.
Please fix the inaccurate Javadocs before merging.
|
|
||
| /** | ||
| * If no workers could be selected for the metadata scan (e.g. all backends are | ||
| * unavailable), create a single empty instance on a random available worker |
There was a problem hiding this comment.
This example is misleading. fillUpAssignedJobs() only reaches fillUpSingleEmptyInstance() when the selected scan ranges produced no assigned jobs; it does not recover from all BEs being unavailable. The fallback still calls workerManager.randomAvailableWorker(catalogId), which throws a NereidsException when no scan backend is available. Please describe this as the empty-scan-range/pruned case instead of an all-backends-unavailable fallback that prevents query failure.
| * fragment. When the expected instance count is lower than the child count | ||
| * (e.g. due to session variable limits or query cache constraints), workers | ||
| * are shuffled to spread instances across different backends for load balancing. | ||
| * When more instances are needed, worker assignment follows the child layout. |
There was a problem hiding this comment.
This sentence does not match the implementation. When expectInstanceNum is greater than or equal to the largest child fragment size, the else branch still calls buildInstances(biggestParallelChildFragment.size(), ...), so this method never creates "more instances" than the child layout. It only preserves the child instance count and worker placement. Please reword this to avoid implying that higher requested exchange parallelism increases the instance count here.
|
PR approved by at least one committer and no changes requested. |
Summary
UnassignedJob.computeAssignedJobs()and all its overrides across 10 concrete subclassesAbstractUnassignedScanJob(multipleMachinesParallelization+insideMachineParallelization)degreeOfParallelism,assignLocalShuffleJobs,assignedDefaultJobs,useLocalShuffleToAddParallel,fillUpAssignedJobs,fillUpSingleEmptyInstanceTest plan
./build.sh --fe🤖 Generated with Claude Code