Fix high CPU usage in object manager due to O(n^2) iteration over active pulls list #13724
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
cc @wuisawesome we should probably pick this into 1.2 since it causes workloads with lots of objects to grind to a halt |
wuisawesome
approved these changes
Jan 27, 2021
|
Sounds good, can one of you add it to the tracker and shepherd it? |
stephanie-wang
approved these changes
Jan 27, 2021
rkooo567
approved these changes
Jan 27, 2021
wuisawesome
added
the
@author-action-required
|
@ericl Lint failures are there. |
|
Lint :( |
wuisawesome
pushed a commit
that referenced
this pull request
Jan 28, 2021
fishbone
pushed a commit
to fishbone/ray
that referenced
this pull request
Feb 16, 2021
fishbone
added a commit
to fishbone/ray
that referenced
this pull request
Feb 16, 2021
…over active pulls list (ray-project#13724)" This reverts commit baee1e4.
fishbone
added a commit
to fishbone/ray
that referenced
this pull request
Feb 16, 2021
…over active pulls list (ray-project#13724)" This reverts commit baee1e4.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Why are these changes needed?
We discovered while testing object spilling that the current pull manager iterates over the entire list of active pull requests for each new pull request. This leads to O(n^2) CPU usage with respect to the number of object dependencies in Ray, causing the cluster to grind to a halt.