[core] fix placement groups with label domain being stuck on the infeasible queue

[aaronscalene]

Description

Let us say we have a placement group that has nodes scheduled on some domain. Part of the nodes go down, which makes the scheduler try to schedule the bundles back on the same domain assignment, which fails and the placement group is placed into the infeasible queue.

Now that the placement group is infeasible, let us say all of the bundles' nodes go down and the placement group is all unplaced. There is no code currently to wake up the scheduling for this placement group from the infeasible queue again. Instead, the placement group will be forever stuck there until OnNodeAdd clears the queue. These fixes are to help with this.

Testing

If you copy the test and run it without any changes, this should currently fail with timeout on pg2.ready() line.
Now, run the test again with these changes and it should now succeed.

Command to run:
python -m pytest python/ray/tests/test_bundle_label_selector.py::test_scheduling_feasible_after_rack_kill

[gemini-code-assist]

Code Review

This pull request improves label-locality-aware rescheduling for placement groups in Ray by ensuring that if all bundles become unplaced due to a total domain failure, the placement group is moved back to the pending queue to allow for rescheduling on a new domain. The changes include adding a helper method to check for all unplaced bundles and updating the GCS placement group manager and scheduler logic. I have provided feedback to improve the readability of a lambda capture and to correct typos in a code comment.

[Sparks0219]

nice find! 🚢