[Ray core] Stopped job leaks worker

[codingl2k1]

What happened + What you expected to happen

No idle worker exists for the stopped job.

Versions / Dependencies

2.10.0

Reproduction script

No reliable script to reproduce this issue.

Issue Severity

High

[codingl2k1]

The issue is that the leaked idle workers are not in the idle_of_all_languages_ of the worker pool. However, in the dashboard, it shows "IDLE".

[codingl2k1]

Leaked again. From the state dump, I can see that there are two Python workers not idle, but they are shown as IDLE on the dashboard. The two registered jobs have already been stopped. No job running now.

[state-dump] WorkerPool:
[state-dump] - registered jobs: 2
[state-dump] - process_failed_job_config_missing: 0
[state-dump] - process_failed_rate_limited: 0
[state-dump] - process_failed_pending_registration: 0
[state-dump] - process_failed_runtime_env_setup_failed: 0
[state-dump] - num CPP workers: 0
[state-dump] - num CPP drivers: 0
[state-dump] - num object spill callbacks queued: 0
[state-dump] - num object restore queued: 0
[state-dump] - num util functions queued: 0
[state-dump] - num PYTHON workers: 2
[state-dump] - num PYTHON drivers: 0
[state-dump] - num object spill callbacks queued: 0
[state-dump] - num object restore queued: 0
[state-dump] - num util functions queued: 0
[state-dump] - num idle workers: 0

[codingl2k1]

Both of the leaked workers have received the force kill request. They have released all reference counts but are waiting for the flight tasks.

task_manager.cc:387: This worker is still managing 90 in flight tasks, waiting for them to finish before shutting down.

I am not sure why 908 + 90 != 1002, but there is no Completing task after the received the Exit signal and all the Add pending task with 0 retries and 0 oom retries.

grep -c "Adding pending task" python-core-worker-a758bca1c0a1cf40c2c29449d44810d8a3b371082c9830891d1c28ec_380790.log
1002
grep -c "Completing task" python-core-worker-a758bca1c0a1cf40c2c29449d44810d8a3b371082c9830891d1c28ec_380790.log
908

[codingl2k1]

From the source code: https://github.com/ray-project/ray/blob/master/src/ray/core_worker/core_worker.cc#L4236
Even the core worker has received a force exit, it still use the Exit instead of ForceExit. I think here should call the ForceExit in HandleExit if the force_exit is true. https://github.com/ray-project/ray/blob/master/src/ray/core_worker/core_worker.cc#L925

[jjyao]

@codingl2k1 you are absolutely right and I have a PR to fix it #44214