Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[core][object spillin] Fix bugs in admission control #13781

Merged

Conversation

stephanie-wang
Copy link
Contributor

Why are these changes needed?

This fixes two bugs in admission control:

  1. Object sizes were not reported to the directory for spilled object locations. This caused bugs in admission control because the node would pull more objects than it had room for.
  2. Underflow bug in computing the number of bytes in use in the object store. This was caused by not properly computing when bytes were no longer in use.

Checks

  • I've run scripts/format.sh to lint the changes in this PR.
  • I've included any doc changes needed for https://docs.ray.io/en/master/.
  • I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
  • Testing Strategy
    • Unit tests
    • Release tests
    • This PR is not tested :(

These bugs are hard to test for because they require integration testing and the corner cases are subtle.

Copy link
Contributor

@rkooo567 rkooo567 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are you making a PR for not sending IPC to raylet separately?

RAY_LOG(DEBUG) << "Updated size of object " << object_id << " to " << object_size
<< ", num bytes being pulled is now " << num_bytes_being_pulled_;
if (it->second.object_size == 0) {
RAY_LOG(WARNING) << "Size of object " << object_id
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why don't we just RAY_CHECK here? Can you write a comment when's the case the size is 0 in normal condition?

@ericl ericl added the @author-action-required The PR author is responsible for the next step. Remove tag to send back to the reviewer. label Jan 29, 2021
@ericl ericl merged commit 754bee9 into ray-project:master Feb 1, 2021
fishbone added a commit to fishbone/ray that referenced this pull request Feb 16, 2021
fishbone added a commit to fishbone/ray that referenced this pull request Feb 16, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
@author-action-required The PR author is responsible for the next step. Remove tag to send back to the reviewer.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants