pg_up accuracy issues

[Tom-Fawcett]

postgres_exporter v0.4.1

I experienced an incident on a Postgres database, which was picked up by a blackbox_exporter TCP probe.

The probe failed between 11:43:47-11:55:47 (approx).

During that time period pg_up remained equal to 1, apart from a few minutes of being absent.

pg_exporter_last_scrape_error was reasonably accurate in tracking the incident.

The postgres_exporter logs mainly contained "connection refused" for the duration of the incident.

Could the accuracy of pg_up be improved, so that I longer need to rely on the blackbox_exporter probe?

---

Graphs/logs

pg_up graph

pg_exporter_last_scrape_error graph

postgres_exporter_redacted.log

[karora]

I shut my database off for 20 minutes to test my alerting, and pg_up never blinked at all!

In the meantime I am using 1 - pg_exporter_last_scrape_error but this seems a fairly severe issue.

[wrouesnel]

So originally I excluded a pg_up metric for exactly this reason - its meaning is ambiguous. The exporter can lose connection to postgres without actually determining if postgres is "up" which actually checks if its processing queries.

There's not really a good way to represent an indeterminate state like that - I.e. should we emit NaN to Prometheus when we just don't know, or - probably as I'll do - force pg_up to 0 if we get a connection error before we can send it.

[karora]

I think it is far better to show a false negative than a false positive in this case. If pg_exporter is having trouble connecting then something, somewhere is wrong, and the issue merits investigation.

I'd also be OK if it simply didn't report the metric when it couldn't connect. We can see that pg_exporter is still running with it's own metric, so I can act on "null or 0" and that could be useful information if it were documented.

The situation where I can connect to PostgreSQL but can't run queries is one I've never encountered. Is it a common case?

[wrouesnel]

Looked at this today, looks like it really is just an oversight (the connect failed trips error, doesn't trip pgUp properly). Fixed and pushing a new release.

[Tom-Fawcett]

Thank you for the fast fix

[Tom-Fawcett]

@wrouesnel following #162 can pg_up ever be 1 now?

[wrouesnel]

Hmm I think you're right there, woops. I'll kill release 0.4.3 and fix that in 0.4.4, as well as add a test for it.

Don't patch when the cat wakes you up early I guess :)