Implementing readiness probe for Kubernetes

[nept]

Hi,

We're considering replacing Puma with Falcon in our Rails app running on top of Kubernetes, but we're struggling to implement effective readiness probes. Unlike Puma, where thread availability is easily checked, Falcon's use of fibers makes it more challenging.

We're considering using the availability of ActiveRecord connections as a metric for Falcon's readiness probe. However, we're uncertain about its efficiency and reliability. Do you have some best practices for implementing it?

Additionally, we're interested in learning how other users typically handle resource allocation (CPU/memory) and ActiveRecord pool management in their production environments.

Thanks

[ioquatix]

We already have a readiness interface as part of async-container which by default supports systemd and a few other mechanisms:

https://github.com/socketry/async-container/blob/main/lib/async/container/notify/client.rb

Wondering what would be the best way to expose this to Kubernetes? Maybe a command?

Regarding memory usage and pool size, that will be highly application specific. In general, a Rails app will be around ~500MiB per server instance and the number of database connections will depend on the number of clients you are hoping to serve simultaneously.

The new activerecord code path with Rails 7.2+ reduces contention on the connection pool. If you can measure then P95 time of your application, and give me a rough idea of how many clients you want to serve, we can figure out the appropriate pool size.

[nept]

Thank you for your input @ioquatix.

Based on your feedback, we've considered two options for exposing the connection pool as a readiness to Kubernetes.
Each options must expose an http endpoint and check if some connections are still available in the pool. We could either develop a command calling this endpoint or just let Kubernetes call it directly.

Option A: Leveraging async-container

While async-container offers a readiness interface, it's unclear how we could directly expose the connection pool through it. Could you please elaborate on how we might achieve this?

Option B: Custom Health Check Endpoint

Given our experience with Puma, a custom health check endpoint seems like a straightforward approach. We could extend our existing Puma health check endpoint to include connection pool checks for Falcon. This would provide a consistent and well-understood mechanism.

Questions:

Do you see any significant drawbacks to using a custom health check endpoint?
We're eager to hear your thoughts on these options and any additional recommendations you may have.

[ioquatix]

I think option B makes more sense if you want control over exactly what kind of things you are checking as healthy. The async-container readiness protocol indicates when Falcon has bound to the socket and will accept incoming connections, but that's different from "the server will meaningfully respond to those requests". Having an HTTP endpoint for this would be the easiest option.

Can you elaborate on exactly what things you want to check the health of? e.g. ActiveRecord?

[nept]

Awesome. Option B is the one we'll like to focus on.

Can you elaborate on exactly what things you want to check the health of? e.g. ActiveRecord?

We want to return OK/KO based on the ActiveRecord stats.
If there is 10 connections in the pool and 10 are in use we return KO and start balancing the trafic to the pods in ready state.

This check will be done for the readiness probe only*

[ioquatix]

We recently implemented support for readiness probes: https://socketry.github.io/async-container/releases/index#add-support-for-notify_log-for-kubernetes-readiness-probes.

Regarding the health check, we now have our own internal health check: https://socketry.github.io/async-container/releases/index#add-health_check_timeout-for-detecting-hung-processes.

In principle, this should be sufficient, but we may also expose a health check interface via the supervisor, but actually issuing a query to your application is probably best for liveliness checks.