[Core] Ray workers are not killed by SIGTERM

[rkooo567]

What happened + What you expected to happen

It looks like Ray workers are not killed by SIGTERM. We have several code that uses SIGTERM to terminate worker processes, and this means that those workers are always ungracefully terminated, which means it will not run critical destructor (it is important for ML workloads). I.e., cleaning up child processes.

Versions / Dependencies

master

Reproduction script

send sigterm to a ray worker and see what happens.

Issue Severity

None

[rkooo567]

the original issue due to this issue; #40189

[YQ-Wang]

Just want to add my observation. When I run a ray job in the ray cluster, the driver code in the ray head is not able to catch the exception after I terminate the worker node ungracefully.

[rkooo567]

Hmm I think it may take some time until it can detect ungraceful failures. For example, detecting a ungraceful node failure would take 30 seconds ~ 1 minute.

[YQ-Wang]

Hmm I think it may take some time until it can detect ungraceful failures. For example, detecting a ungraceful node failure would take 30 seconds ~ 1 minute.

Yeah, it cannot detect the ungraceful failure like forever. It always stuck in running status even though the worker node is down.

[rkooo567]

Hmm intersting. That seems an orthogonal issue from this particular issue (probably related to keepalive). Is it possible to create a new issue with a reproducible script? We can start converstaion from there