[Feature] Restart worker pod when raylet exited

[chenk008]

Search before asking

• I had searched in the issues and found no similar feature requirement.

Description

For now, ray-operator generate worker pod command

ulimit -n 65536; ray start  --node-ip-address=$MY_POD_IP  --redis-password=LetMeInRay  --address=raycluster-heterogeneous-head-svc:6379  && sleep infinity

If the raylet in worker pod exited, the raylet don't restart, so the pod will be useless without raylet.

Use case

If the raylet in worker pod exited, the raylet should restart.

One option is starting ray with --block and delete the sleep infinity.
Another option is restarting the pod, so the raylet will start. Maybe we can add some default liveness probe.

Related issues

No response

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

[Jeffwan]

Agree. this should be improved to handle failures