Health Checks Failing - Readiness & Liveness For Production Azure AKS Cluster

[zohebs341]

Hi All,

We are using Azure AKS Version 1.20.7 with nearly 10 Nodes running on it. Vault is Up, Database is Up, Istio is up and running.

But still health checks(Liveness & Readiness) are failing.

context deadline exceeded (Client.Timeout exceeded while awaiting headers)
Readiness probe failed: HTTP probe failed with statuscode: 500

Any suggestions or guess?

Is it because of load at API server?? As we have Istio & vault as an init container to application pods.

[k8s-ci-robot]

@zohebs341: This issue is currently awaiting triage.

If a SIG or subproject determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[zohebs341]

In general, if DB connectivity issues or if vault having issues. Health check will fail, that is expected.

But in my case, vault, istio& DB all are up. But still liveness & readiness failing

[zohebs341]

@swetharepakula Please check if you find time.

Adding more info to it.

I am using Horizontal pod autoscaler CPU based, continuously it's creating and deleting pods based on CPU load. I've noticed "failed to update endpoints error" in events.

[yxxhero]

/sig node

[ehashman]

Kubernetes does not use issues on this repo for support requests. If you have a question on how to use Kubernetes or to debug a specific issue, please visit our forums.

/kind support
/close

[k8s-ci-robot]

@ehashman: Closing this issue.

In response to this:

Kubernetes does not use issues on this repo for support requests. If you have a question on how to use Kubernetes or to debug a specific issue, please visit our forums.

/kind support
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.