[Feature] Another possible way to detect readiness failures in fault tolerance after release 0.5.0

[Yicheng-Lu-llll]

Search before asking

• I had searched in the issues and found no similar feature requirement.

Description

In the current implementation of fault tolerance, the kuberay operator watches the events to detect readiness probe failures. If detected, the pod is marked as unhealthy and will be deleted, and start a new one in later Reconciliation.

As @kevin85421 suggests #601 (comment), watching readiness probe fail event may have downside:

we finally need to stop watching events because operator operations should be idempotent and stateless. However, events are time-sensitive and deleting Pods based on events are not idempotent.

A potential solution to the problem is to examine the ContainerStatus.Ready property to determine if a readiness probe has failed (refer to https://pkg.go.dev/k8s.io/api/core/v1#ContainerStatus for more information).

// ContainerStatus contains details for the current status of this container.
type ContainerStatus struct {
	...
	// Specifies whether the container has passed its readiness probe.
	Ready bool `json:"ready" protobuf:"varint,4,opt,name=ready"`
        ...
	
}

And kuberay operator can watch the ContainerStatus.Ready (not sure watch status of an object is in this case is a good approach or not). Alternatively, Kuberay operator can handle readiness probe failures during the reconciliations that are triggered by other object changes.

Use case

No response

Related issues

No response

Are you willing to submit a PR?

• Yes I am willing to submit a PR!