Improve health detection of /-/healthy endpoint

[jeremyeder]

We're trying to find the best way to monitor prometheus itself. We were evaluating using the /healthy and /ready endpoints. But we have a case where prometheus was failed, yet the endpoints still reported back as healthy.

What did you do?
Used Prometheus 2.0, ran out of disk space.

What did you expect to see?
I expected to see prometheus marked unhealthy when querying the /-/healthy endpoint.

What did you see instead? Under which circumstances?
Prometheus responded "Prometheus is Healthy."

Environment

• RHEL/OpenShift

• System information:
  Linux 3.10.0-693.11.6.el7.x86_64 x86_64

• Prometheus version:
  prometheus:v2.0.0

• Logs:

level=error ts=2018-02-05T21:42:18.480373214Z caller=wal.go:713 component=tsdb msg="sync failed" err="flush buffer: write data/wal/000001: file already closed"
level=warn ts=2018-02-05T21:42:21.495046779Z caller=scrape.go:673 component="target manager" scrape_pool=kubernetes-service-endpoints target=https://10.131.12.9:8443/metrics msg="append failed" err="WAL log samples: log series: write data/wal/000001: file already closed"

[krasi-georgiev]

I think tsdb is out of scope here so removed the local storage label.