updated liveness and readiness probe descriptions

[tsloughter]

I still need to add a paragraph about the startup probe. I'm not planning on using it in service_discovery.

[github-actions]

🚀 Deployed on https://6517036c7f43a60f917a5c3b--inspiring-shannon-000ae0.netlify.app

[tsloughter]

I'll also add a paragraph at the end about the values used for readiness.

Main thing with startup probe will be why we don't use it.

[ferd]

Eeeeh, if you don't remove yourself from the load balancer pool, some gateways (like ALBs or ELBs) may still try to send traffic to a process that's halfway through shutting down, and that will cause interrupted connections that are forwarded to the client. This may however be stack dependent because not all stacks clean up as nicely and orderly as an OTP tree.

[tsloughter]

Removal should already be handled by k8s. There is always delay, even in the case of the readinessProbe being setup to return 503 on shutdown since the check is only done every period seconds, but its handled by k8s and not the application itself is my point. Maybe it needs rewording?

[ferd]

Yeah that could make it clearer. It's possible this was just my misunderstanding and I didn't realize that being in a shutting down state also automatically took you out.

[tsloughter]

Actually, I should probably mention the use of prep_stop. I haven't implemented this in service_discovery because it is a little tricky in its case since it has 2 applications, one for http and one for grpc, that both have to wait until active requests are handled before shutting down.

I suppose I should also mention the need to have a proper boot order and supervision tree so that stuff like the database pool aren't shutdown before the http app can pause to complete open requests.

I could just add a "see Graceful Shutdown section (to come)" for more...

[tsloughter]

Eh, maybe it isn't tricky, was overthinking it. It is the same whether they are both in the release or not. Each can wait for their active connections one after the other and the SIGKIll takes care of it if they take too long.

[tsloughter]

Wait, yea, this is just k8s specific. I will just add a callout to another section and with saying in k8s you don't need to do anything special, but see this section for how to handle the shutdown in your Erlang application.

[ferd]

might be interesting to eventually aim for /health/<type> but we're good to go here.