k8s readiness check failing

[pocesar]

Question

When running on k8s, I'm now seeing [Health Check] Readiness failed: Dial error: handshake failed (connection name = "xxx:xxx:xxx"): context canceled only and it never recovers.
I'm using the config from here
https://github.com/GoogleCloudPlatform/cloud-sql-proxy/blob/main/examples/k8s-health-check/proxy_with_http_health_check.yaml
Also, the pod is showing as healthy, but it's not, since the application isn't being able to connect to it.
Nothing changed in the configuration and started failing this weekend on GCP

Code

No response

Additional Details

No response

[jackwotherspoon]

Hi @pocesar! Thanks for opening an issue on the Cloud SQL Proxy.

Couple quick questions for you to hopefully understand your scenario a bit better:

1. Was your deployment working fine and then all of a sudden started seeing the above error?
2. Are you using Public or Private IP Cloud SQL connection?

We are currently updating our Health Check sample and this work is being tracked here #1950

[hessjcg]

Hi @pocesar,

If your application can't connect, and also the proxy readiness check fails, then there may be a network configuration problem preventing the proxy from establishing a connection to the Cloud SQL Instance.

About readiness checks:

We discourage the use of the readiness check in most applications. If you are running an application like a website or API service, you usually want the application container to provide a readiness check that includes connecting through the proxy to the database. (see Readiness Check Configuration).

The example readiness check configuration is tuned for batch jobs. In this configuration, the readiness check will attempt to connect to the instance for a full 60 seconds, failing 6 times before marking the pod as "not ready".

[pocesar]

@jackwotherspoon

1. Yes, it was working fine and now having those issues
2. I'm using the private IP of the SQL instance

@hessjcg I see, so the recommended manifest should leave the readiness block out?

[enocom]

Yes -- readiness is a tricky check to get right.

@pocesar Can your app reach your database otherwise?

[hessjcg]

Closing due to inactivity.