Enhance healthy endpoint to detect unhealthy upstreams

[luizberti]

The healthcheck probe (/-/healthy) seems to be useless at the moment, and doing nothing more than what the readiness probe (/-/ready) should do.

In my current situation, the Promtail integration was working correctly and pushing to Loki, but the Prometheus one was botched. The healthcheck probe should not state that the agent is healthy when the upstream is botched, the correct behaviour would be to raise an issue if any upstream is faulty, unless the user hasn't configured any upstream to forward metrics to (such as cases where they wanna scrape the Agent's /metrics themselves).

On that same note, the accompanying readiness probe should only report a 200 if config parsed correctly + all internal components have managed to start successfully + I can see things on /metrics on the agent (even if remote_write is failing, or some SD configuration is failing, as those should be reported on the healthcheck instead). I assume this is what happens already but haven't validated, so I'm just mentioning this here for reassurance.

In my case, the remote_write URL was botched, but I've had this happen when I applied a manifest with botched credentials and upstream was returning a 403, and also experienced the same problem with Promtail (not the in-agent one). The URL 404 error of the last occurrence was the one below:

ts=2020-09-23T20:23:27.251090321Z caller=dedupe.go:112 agent=prometheus instance=811f01a1f9d2e40bc66826c12931e4c8 component=remote level=error remote_name=811f01-83fd73 url=http://prometheus/api/prom/push msg="non-recoverable error" count=3 err="server returned HTTP status 404 Not Found: Not Found"

As it stands, even when doing a gradual rollout on Kubernetes, there is nothing that can stop the rollout if I'm pushing botched configuration and killing all of the cluster's monitoring unless if I were to make a disgusting log-parsing hack, but I am not fond of that idea 😅

[rfratto]

The healthcheck probe (/-/healthy) seems to be useless at the moment, and doing nothing more than what the readiness probe (/-/ready) should do.

You're right, and we admittedly just copied the healthcheck from Prometheus where it also blindly returns 200 as long as the server is running and the config loaded successfully.

Handling this intelligently is harder than it looks, and it's easy to implement this in a way that makes the experience worse. Using remote_write errors to determine healthiness is especially error-prone. If the config works one day but a server-side change temporarily breaks things and starts returning 404s, the pod should still be considered healthy (IMO).

There's a few things you can do here to handle this outside of parsing the logs manually:

• Increasing minReadySeconds on the deployment/replicaset to allow more buffer for stopping a failed rollout.
• Rolling out changes to a staging environment first.
• Add alerting to catch a sudden loss of metrics. e.g., absent(up{job="<some-agent-job-label>"}) to catch an invalid config after it's deployed.
• HA pairs of Agents that scrape the same targets but configs are applied to them separately, allowing failover to the "working" Agent config.

This isn't an exhaustive list and you might find something better than everything I've mentioned here.

A future alternative would be to use Loki alerts to alert on the logs, which we'll have something for in the coming months. Then you could alert on something like count_over_time({job="<some-agent-job-label>"} |= "non-recoverable error"[1m]) > 0

there is nothing that can stop the rollout if I'm pushing botched configuration and killing all of the cluster's monitoring

Is there some reason that kubectl rollout undo doesn't work for you?

[luizberti]

If the config works one day but a server-side change temporarily breaks things and starts returning 404s, the pod should still be considered healthy (IMO)

I see your point, but I don't think this would be the way to go. I think there are two big use cases at play here, and going all in on either of them would be bad. Say you have an API backed by a DB...

If your DB breaks, your app is definitely not healthy, and I think that's how most people would code that healthcheck, because if the DB is borked, the API itself is just paperweight taking up cluster resources. Same reasoning goes for the daemon's backend being down: the daemon's single purpose in life is to forward the stuff to the backend, if the backend is botched for whatever reason, the core functionality that made me deploy it to the cluster in the fist place is critically impaired, and without it the daemon is as useful as a brick. In this case, the daemon is definitely not healthy either.

However, there are hybrid cases where the daemon is used both for forwarding and exporting, for example, if I'm forwarding logs to loki, but scrape the node_exporter integration from an in-cluster Prometheus instance. In this case, the probe's ideal behavior is not as obvious: should a flakey loki backend impact my node_exporters?

I am actually under both of these use cases simultaneously (I briefly described my setup in this other ticket I have open)

I think the right way to go here would be:

• Exposing one probe per internal component, on top of a global one: this allows people to code their own health-checking behavior however they like. It might also be interesting to allow for some queryability on the global endpoint, for example, hitting the endpoint with /-/healthy?q=prometheus,promtail would report "unhealthy" if the probes for any of the selected components are failing. This would integrate very nicely with Kube's own healthchecking, and wouldn't require the user to do any custom probing script that juggles the endpoints they care about;
  • Having the global probe print a status report of the internal subsystems would also be very nice for debugging;
  • Situations like the one described in ticket subsystems can crash "undetected" #207, where a single subsystem is individually failing, make this a doubly good idea IMO
• Making it configurable: allow users to set unhealthy_when_backends_fail in the configuration to control the probe's behaviour, for each integration individually (possibly per configured backend instead? that might make more sense depending on the situation)

Is there some reason that kubectl rollout undo doesn't work for you?

I don't think this is a real solution to the problem, as it defaults to reactiveness, rather than defensiveness. I think reducing the blast radius of these kinds of things is exactly what makes these probes awesome. Consider the following scenarios at play:

1. I kubectl apply some botched stuff; the healthcheck probe is unfazed, and thus Kubernetes will roll it out to all nodes in my cluster; I have now essentially killed monitoring on the entire cluster; all pagers go off, because we all of the sudden went blind; have to investigate what happened while blind which makes things more difficult (it might not be obvious, and the person getting paged might not be the same one that triggered the change); roll things back and everything goes back to normal.
2. I kubectl apply some botched stuff; the healthcheck probe flares up, and thus Kubernetes's existing rollout automation mechanisms will stop the rollout at maxUnavailable nodes; monitoring is not dead, so I'll see the number of pending rollouts in my dashboard; eventually Kube will give up on this rollout, and I'll see a spike in the number of failed rollouts (and probably get alerted for it); meanwhile Kube automatically reverted things; I'll immediately see what failed and when, and can easily replicate it to debug; I fix the manifests and push working changes this time, and everything fine.

The way the healthcheck probes currently work make the existing rollout automation that's meant to catch these things completely ineffective, and I think what we should be aiming at is scenario number 2, as it seems to be a much more pleasant situation to be in

[rfratto]

To clarify, we should be talking just about the behavior of the readiness checks (i.e., readinessProbe) and not health checks (i.e., livenessProbe). Health checks won't stop a rollout, and will rather just signal to Kubernetes that the unhealthy pod should be restarted, which is unhelpful in the scenario we've been discussing. In general, we are unlikely to support liveness probes altogether, which can be dangerous.

I have a few points to make, but I want to reiterate that I agree with the use case that asks for more advanced logic around determining when an Agent is fully ready to signal that a rollout can continue. My argument that this is harder than it seems is unchanged. I can see my teammates having some opinions about this, so we'll need to agree on a list of problems that should cause a readiness check to not succeed.

This likely won't be prioritized very highly internally since there are acceptable workarounds for preventing a misconfigured set of Agents from taking down all of monitoring. A third party PR is more than welcome, but if anyone decides to work on this, I would strongly suggest proposing a solution first that we can reach consensus on. I don't want to waste anyone's time with a PR that won't ever be accepted.

If your DB breaks, your app is definitely not healthy, and I think that's how most people would code that healthcheck, because if the DB is borked, the API itself is just paperweight taking up cluster resources

Sure, if the database backing an application breaks, your application is left in a partially (or completely) inoperable state. However, taking any action on the application isn't going to solve anything, and taking it down may just make things worse. The application being non-functional is a symptom, not the cause.

A broken API likely consumes very little resources beyond its resource requests. I'm not sure I understand the argument that this is a problem, since the idea of needing the resources the broken API is requesting implies underprovisioning, where a functional API may not be able to be scheduled at all once the underlying DB problem is resolved.

should a flakey loki backend impact my node_exporters?

If you are concerned about subsystems interacting with one another, I would recommend splitting up your Agent deployments and have one per subsystem.

monitoring is not dead, so I'll see the number of pending rollouts in my dashboard; eventually Kube will give up on this rollout, and I'll see a spike in the number of failed rollouts (and probably get alerted for it)

Off-topic, but I do want to point out that you can't rely on an alert for this. One of the pods that failed may be the pod responsible for collecting metrics from the Kubernetes API. If this isn't true for your specific deployment, it may be true for other users. If you only have one set of Agents, the only way to reliably alert on Agent availability problems is to either alert on the absence of an expected metric or alerting on logs.

This is still true for Prometheus; you can't alert on the availability for something that is monitoring itself. Rather, an HA setup where config changes are applied asynchronously is ideal.

[stale]

This issue has been automatically marked as stale because it has not had any activity in the past 30 days. It will be closed in 7 days if no further activity occurs. Thank you for your contributions.