[core] (FreeObjects 2/n) Added GCS passive owner death object freeing

[aaronscalene]

Description

This PR covers the second part of the FreeObjects change. This second part amends the functionality of the current owner dead pubsub callback to be moved from an eager pubsub death callback to a more passive listener callback. Originally, the design was that the owner is thought to be dead when the pubsub connection fails / ends for some unexpected reason, which then tells all the primary copy raylet to free and queue up FreeObjects.

Currently, the GCS already publishes WorkerFailure and NodeRemoved announcements to all raylets. With this new design, the raylets will now just have additional functionality to clear any objects that had their owner on the failed worker / node.

Considerations

Pass Over All Objects

This new design introduces a pass over all objects within the current node every announcement. However, failed nodes and worker announcements should not be too common and if the object is not owned by the failed worker / node it would be a no-op; thus this should not lead to any latency issues.

Owner is Less Eager

This new design will be a less eager free, especially for the primary copy of the node. This is definitely an important part to consider, however since it is similar a pubsub for node information, the latency should not be that big of a deal. Furthermore, the pubsub interface already keeps track of network failures etc. to some extent, and if there are further failures / GCS failures there are probably worse problems to deal with.

LocalObjectManager local_objects_ vs. ObjectManager local_objects_

An important distinction is needed between both classes' local objects, where we need to aggregate and free any objects within both classes. LocalObjectManager in this case keeps track of primary copies as well as spilled copies. ObjectManager keeps track of anything in plasma such as primary copies / secondary copies. Thus, we need to get the union of both objects to get the objects that we want to free.

Testing

Ran python tests and C++ tests adjacent to these changes to ensure no regressions. Future e2e testing in #63181

[gemini-code-assist]

Code Review

This pull request introduces a new RPC mechanism, FreeLocalObjects, to allow the CoreWorker to explicitly instruct NodeManager instances to free local objects. This replaces the previous eviction-pubsub callback approach. My review identified several areas for improvement: the CoreWorker lambda capture should use shared_from_this for lifetime safety, the RPC implementation should be batched to avoid performance degradation in large clusters, and the now-unused objects_pending_deletion_ logic in LocalObjectManager should be cleaned up.

[gemini-code-assist]

There are two potential improvements for this block:

1. Lifetime Safety: Since CoreWorker inherits from std::enable_shared_from_this, it is safer to capture a shared_ptr (e.g., self = shared_from_this()) in the lambda. This ensures the CoreWorker instance remains alive while the task is pending in the io_service_.
2. Efficiency: Checking if locations is empty before posting to the io_service_ avoids unnecessary task scheduling overhead.

  if (locations.empty()) {
    return;
  }
  io_service_.post(
      [self = shared_from_this(), object_id, locations]() {

[gemini-code-assist]

With the removal of the on_objects_freed_ callback, objects_pending_deletion_ appears to be dead code as the objects are now deleted directly in NodeManager::FreeLocalObjects. As noted in the TODO on line 334 of the header, this set and the logic in FlushFreeObjects should be removed to simplify the code and avoid confusion.

[cursor]

Cursor Bugbot has reviewed your changes and found 2 potential issues.

^{Reviewed by Cursor Bugbot for commit a8c096f. Configure here.}

[cursor]

Proto file modification requires fault-tolerance review

Low Severity

⚠️ This PR modifies one or more .proto files.
Please review the RPC fault-tolerance & idempotency standards guide here:
https://github.com/ray-project/ray/tree/master/doc/source/ray-core/internals/rpc-fault-tolerance.rst

Additional Locations (1)

• src/ray/protobuf/node_manager.proto#L594-L597

 

^{Triggered by project rule: Bugbot Rules}

^{Reviewed by Cursor Bugbot for commit a8c096f. Configure here.}

[cursor]

Dead code: on_objects_freed_ member never invoked

Medium Severity

The on_objects_freed_ member variable (still accepted as a constructor parameter and stored) is never invoked anywhere after the FlushFreeObjects refactor removed its only call site. In production this stored a lambda calling object_manager->FreeObjects(object_ids, /*local_only=*/false), which also performed buffer_pool_.AbortCreate to cancel in-flight object transfers. That abort logic is now silently lost—neither NodeManager::FreeLocalObjects nor the new RPC path calls it.

Additional Locations (1)

• src/ray/raylet/local_object_manager.h#L247-L248

 

^{Reviewed by Cursor Bugbot for commit a8c096f. Configure here.}