Enable distributed ref counting by default

[stephanie-wang]

Why are these changes needed?

Turn on distributed ref counting by default. This can be turned off with:

ray.init(_internal_config=json.dumps({
    "distributed_ref_counting_enabled": 0,
  }))

This also turns on eager eviction for objects by default. This means that all copies of an object will be proactively evicted once the ID is no longer in scope on any workers. This can be turned off with:

ray.init(_internal_config=json.dumps({
    "free_objects_period_milliseconds,": -1,
  }))

Related issue number

Checks

• I've run scripts/format.sh to lint the changes in this PR.
• I've included any doc changes needed for https://ray.readthedocs.io/en/latest/.
• I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failure rates at https://ray-travis-tracker.herokuapp.com/.
• Testing Strategy
  • Unit tests
  • Release tests
  • This PR is not tested (please justify below)

[ericl]

Can we also enable eager eviction at the same time?

[AmplabJenkins]

Can one of the admins verify this patch?

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/23245/
Test PASSed.

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/23246/
Test PASSed.

[stephanie-wang]

Note that I had to add a DrainAndShutdown method to the ReferenceCounter, similar to the one for TaskManager. This is to make sure that if the worker is doing a clean exit and still has some objects that it owns, that it will wait for their refs to go out of scope before exiting. This became necessary because eager eviction is now enabled in this PR.

[AmplabJenkins]

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/23254/
Test FAILed.

[AmplabJenkins]

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/23251/
Test FAILed.

[zhijunfu]

@stephanie-wang Sounds great. Just a few questions on this:

• Are there any scenarios that are not yet supported with this? Or are there any limitations so far?
• Are there any noticeable impacts with this enabled beside eager eviction? e.g. are there performance regressions .etc?

[AmplabJenkins]

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/23257/
Test FAILed.

[edoakes]

@zhijunfu

One limitation is that workloads that previously relied on LRU to function (e.g., workloads like IPython that keep references around forever even if they aren't used) will not work (and same goes for the previous two releases). However, they should be able to revert to the old LRU behavior by turning off the object_pinning_enabled flag.

We have not seen substantial performance regressions, but will be keeping an eye out for this while running the release tests (cc @simon-mo).

[edoakes]

@stephanie-wang the diff is showing as +1314/-813 LOC, not sure why.

[AmplabJenkins]

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/23286/
Test FAILed.

[AmplabJenkins]

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/23297/
Test FAILed.

[AmplabJenkins]

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/23304/
Test FAILed.

[stephanie-wang]

@stephanie-wang Sounds great. Just a few questions on this:

* Are there any scenarios that are not yet supported with this? Or are there any limitations so far?

Please see the design doc for this.

* Are there any noticeable impacts with this enabled beside eager eviction? e.g. are there performance regressions .etc?

On my laptop, the PR for eager eviction (#7220) seemed to incur ~10% overhead for workloads with many plasma objects. The goal is to have this in for the next ray release so we can run the performance benchmarks from the release testing.

[AmplabJenkins]

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/23338/
Test FAILed.

[AmplabJenkins]

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/23340/
Test FAILed.

[AmplabJenkins]

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/23355/
Test FAILed.

[AmplabJenkins]

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/23363/
Test FAILed.