ComputeNode status is not precisely sync with Pod status

[mlycore]

In the snapshot below, the ComputeNode readyInstances remains 2 even if the 2 Pods are terminating and non of them is ready (0/1) actually.

Expected behavior is the ready instances are 0 in this situation.

[yikuaibro]

It seems to be affected by network fluctuations? Sometimes I also find that pods in the no ready(0/1) and terminating state are also regarded as readyInstances. When the cluster network is smooth, this happens for a short time.

[yikuaibro]

The Ready field in the ContainerStatus struct in the Kubernetes library is set based on the result of the readiness probe defined for the container in the Pod's specification. If the readiness probe succeeds, the Ready field is set to true, and if it fails, it is set to false.

If the Ready field is true, it means that the container has passed its readiness probe and is ready to serve requests. However, this does not necessarily mean that the container is currently ready. If the container has just started or is still initializing, it may take some time before it becomes ready. During this time, the Ready field may be true, but the READY column in the kubectl get pods command result may show a value of (0/1).

Therefore, the READY column in the kubectl get pods command result provides a more real-time view of the readiness of the containers in the Pod, while the Ready field in the ContainerStatus struct provides information about the overall readiness of the container.

[yikuaibro]

I don't know if I understand right. HAHA

[mlycore]

In this function getReadyInstances(), we just align with the ready status true. And it will only be updated in next reconcliation.

shardingsphere-on-cloud/shardingsphere-operator/pkg/controllers/compute_node_controller.go

Lines 321 to 337 in c500401

	func getReadyInstances(podlist v1.PodList) int32 {
	var cnt int32
	for _, p := range podlist.Items {
	if p.Status.Phase == v1.PodRunning {
	for _, c := range p.Status.Conditions {
	if c.Type == v1.PodReady && c.Status == v1.ConditionTrue {
	for _, con := range p.Status.ContainerStatuses {
	if con.Name == "shardingsphere-proxy" && con.Ready {
	cnt++
	}
	}
	}
	}
	}
	}
	return cnt
	}

We define readyInstances to imply how many proxy instances are ready to accept connections. So it should be real-time. And ReadinessProbe are executed periodically, new conditions will be updated every time it executes.

Perhaps we need to check the kubectl get pod and see if we need to update our getReadyInstances logic. @yikuaibro

[yikuaibro]

Test environment:
k8s v1.20
operator v lasted
Test case:
Simulate kubectl get pod inside the code to get the latest pod status. In order to be as consistent as possible with the podList time obtained by ComputeNodeReconciler.List, eliminate the intermediate pod changes and synchronize them.
Add the source code of the func printpod part in k8s to compare with the current func getReadyInstances results, and get three kinds of data:

• execPod: command result
• testPod: partial results of the source code
• realPod: the original logic result

Testing:

Conclusion:
The error between execPod and testPod is almost within 1 second, and the content of testPod is consistent with the original logic. It is not a problem of getReadyInstances. It may be caused by the delay when the return value is rendered to computenode, or the cache problem of apiserver.

[mlycore]

Thanks for your great efforts! This procedure is pretty clear.

[mlycore]

The readiness status of Pod is controlled by the Kubelet, you can check the Pod.Spec to see if it has setup a readinessProbe. If not, maybe look up the default behaviour in the Kubelet source code.

[yikuaibro]

Got it.