Show big and fat warning on web status page upon certain conditions.

[beorn7]

Like dirty storage, quarantining of series, too high persist pressure, too many memory chunks...

Ideally with a little explanation what's happening and what can be done.

[beorn7]

@fabxc fallout from our discussion

[fabxc]

Sounds good.

[RichiH]

Arguably, certain machine-level metrics like available disk space, system load, heavy swap/full memory, high IO wait, and possibly others fall into the same category of "the user should really handle this themselves, but this is really bad and we need to yell about it".

[beorn7]

@RichiH Those are machine level metrics you want to monitor anyway, no matter if Prometheus runs on your machine or not. Also, those metrics are not even accessible for the Prometheus server binary.

The conditions this issue refers to are Prometheus specific. You should still monitor it in you meta-monitoring, too. But putting on the status page if something is fundamentally wrong within Prometheus sounds like a good idea.

There is no intent, though, to turn the status page into a system-health monitor.

[dominikschulz]

Maybe a little off-topic, but it fits this discussion: Is that status page suitable as a health check URL (e.g. for K8s Liveness Probes)?

[juliusv]

@dominikschulz This would just be part of the normal web UI. It wouldn't return a non-200 if it visually reports some errors there. So not really.

Generally, you can use the availability of the web UI in a limited way as a startup health probe, as the web UI is started as the last thing on Prometheus startup. However, it won't tell you much about whether Prometheus is generally healthy, only that it has finished starting up.

[brian-brazil]

See also #1468

[gouthamve]

Is this still relevant given 2.0?

[brian-brazil]

The storage less so, but there's still things like rule groups taking longer than their interval.

[RichiH]

It probably changes a bit. For example isolation might want to let the users know if there's a really old write hanging somewhere or, as Brian beat me to writing, recording rules taking longer than they should.

[RichiH]

Though @gouthamve 's implicit question is valid; there should be a new list of things to handle.

[roidelapluie]

Easy ones:

• Prometheus failed to reload its configuration
• WAL corruptions