[autoscaler] Improve V2 scheduler pre-filter precision for multi-resource demands - #65171
Open
Jade07-1 wants to merge 3 commits into
Open
[autoscaler] Improve V2 scheduler pre-filter precision for multi-resource demands#65171Jade07-1 wants to merge 3 commits into
Jade07-1 wants to merge 3 commits into
Conversation
…urce demands The quick-reject pre-filter in `_try_schedule` previously used OR logic across resource dimensions: a node was kept in the scheduling loop if ANY single dimension met the minimum demand. This caused the optimization to be ineffective when one dimension (e.g., CPU) was exhausted but another (e.g., memory) had plenty of headroom. Change the pre-filter to use AND logic within each resource shape and OR logic across shapes: a node is only kept if it can satisfy ALL dimensions of at least one pending request shape simultaneously. Benchmark (scaling to 3000 nodes, 15000 tasks × 0.2 CPU + 30MB memory): - Before: pre-filter ineffective, ~2h+ to reach 3000 nodes - After: pre-filter correctly skips CPU-exhausted nodes, ~1081s Signed-off-by: wangjia23 <wangjia23@xiaomi.com>
… add mixed-shape tests - Rename _compute_min_resource_demand -> _collect_unique_resource_shapes - Rename parameter/variable min_resource_demand -> resource_shapes - Update call-site comment to clarify AND-within-shape, OR-across-shapes - Add test_quick_reject_mixed_shapes_or_across_shapes: verifies nodes are kept when at least one shape fits (OR across shapes) - Add test_quick_reject_mixed_shapes_all_exhausted: verifies nodes are rejected when no shape fits Signed-off-by: wangjia23 <wangjia23@xiaomi.com> Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> Signed-off-by: wangjia23 <wangjia23@xiaomi.com>
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Code Review
This pull request refactors the scheduler's quick feasibility pre-check by replacing the minimum resource demand calculation with a check based on unique resource shapes. This ensures that all resource dimensions of a request shape are satisfied simultaneously (AND logic) rather than independently. New unit tests are added to cover various rejection scenarios. The reviewer identified a critical issue in _can_fit_any_request where implicit resources default to 0.0 instead of 1.0, which could cause valid nodes to be incorrectly skipped, and provided a code suggestion to resolve this.
Comment on lines
+168
to
173
| if not resource_shapes: | ||
| return True | ||
| for k, min_v in min_resource_demand.items(): | ||
| if available.get(k, 0.0) >= min_v: | ||
| for shape in resource_shapes: | ||
| if all(available.get(k, 0.0) >= v for k, v in shape.items()): | ||
| return True | ||
| return False |
Contributor
There was a problem hiding this comment.
The current implementation of _can_fit_any_request uses available.get(k, 0.0) which defaults to 0.0 for missing resource keys. However, for implicit resources (which start with ray._raylet.IMPLICIT_RESOURCE_PREFIX), the default value on a node is 1.0 if not explicitly present (as implemented in _fits).\n\nWith the new all() check (AND logic), if a request shape contains both a standard resource (like CPU) and an implicit resource, and the node does not explicitly list the implicit resource, available.get(implicit_resource, 0.0) >= v will evaluate to False (since 0.0 < v). This causes the entire shape check to fail, and the node is incorrectly skipped (pre-filtered) even though it actually has enough resources.\n\nTo fix this, we should default the available resource value to 1.0 for implicit resources, matching the behavior in _fits.
if not resource_shapes:\n return True\n import ray\n implicit_prefix = ray._raylet.IMPLICIT_RESOURCE_PREFIX\n for shape in resource_shapes:\n if all(\n available.get(k, 1.0 if k.startswith(implicit_prefix) else 0.0) >= v\n for k, v in shape.items()\n ):\n return True\n return False![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.
Reviewed by Cursor Bugbot for commit 069a835. Configure here.
ray-gardener
Bot
added
core
Contributor
|
Hi @Jade07-1, there is a test failure: |
- Use IMPLICIT_RESOURCE_PREFIX-aware default (1.0) in _can_fit_any_request to match _fits() behavior, preventing false rejection of nodes with implicit resources not listed in available_resources. - Fix test_quick_reject_mixed_shapes_all_exhausted: set type_gpu CPU=1 so it cannot absorb shape A (CPU=2), making the assertion correct. Signed-off-by: wangjia23 <wangjia23@xiaomi.com>
Contributor
Author
|
5 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why are these changes needed?
Since #64175, I've been continuing to optimize the V2 autoscaler scheduling path for large clusters — including demand vector truncation,
SerializeToStringhotspot elimination, and homogeneous-batch fast-path. During benchmarking across different workload configurations, I noticed the quick-reject pre-filter from #64175 becomes ineffective when task resource demands are multi-dimensional but only one dimension saturates the node.Root cause: The pre-filter uses OR logic across resource dimensions — a node is kept in the scheduling loop if any single dimension meets the minimum demand. For workloads like
@ray.remote(num_cpus=0.2, memory=30MB)on nodes with 1 CPU + 1 GB memory, after 5 tasks fill the CPU, the node still has ~850 MB memory available. The OR check seesmemory >= 30MB→ returnsTrue→ the node is NOT skipped → expensivedeepcopy + try_scheduleruns on every reconcile round for thousands of fully-loaded nodes.Changes
Change the pre-filter to use AND logic within each resource shape and OR logic across shapes: a node is only kept if it can satisfy ALL dimensions of at least one pending request shape simultaneously.
_compute_min_resource_demand()→_collect_unique_resource_shapes(): returns a deduplicated list of complete resource bundles instead of collapsing all dimensions into per-key minimums._can_fit_any_request(): checks each shape holistically (all()within a shape) instead of checking dimensions independently.Complexity:
O(S × D)where S = number of unique shapes (typically 1–10), D = dimensions per shape (typically 2–4). For the common single-shape workload, S=1 and cost is identical to the previous O(D).Benchmark
Scaling to 3000 nodes (15000 tasks × 0.2 CPU + 30 MB memory, 1 CPU per worker):
The OR pre-filter fails to skip nodes when CPU is exhausted but memory has headroom — the common case for small-memory tasks on large-memory nodes. With AND logic, these nodes are correctly skipped:
Related issue number
Follow-up to #64175. This does NOT address the
UnschedulableRequestCache.contains()bottleneck noted there (separate follow-up).Checks
scripts/format.shto lint the changes in this PR.