Add more useful grafana dashboards to the prombench tests

[krasi-georgiev]

http://prombench.prometheus.io/grafana
Currently we use the https://grafana.com/dashboards/6725 Prometheus Benchmark - 2.3.x by Robust Perception.

The most useful metrics I see here are RAM allocations and CPU utilisation.

Looking to add more so any input from people that use Prometheus in production and know where it brakes will be very useful.

Current deployment file: https://github.com/prometheus/prombench/blob/f144b52bb8502c86e8d6c61371019babfb648c99/components/prombench/manifests/results/grafana_dashboard.yaml

cc @sipian

[tariq1890]

@krasi-georgiev I am interested in taking this up. Did you happen to have any other useful dashboards in mind?

[krasi-georgiev]

we haven't been using it long enough to decide what other dashboards would be useful.

One dashboard that is currently broken and it would be useful to have is the Disk Space Used Basic

I would advise to ping @sipian on the prometheus-dev channel for some more info on what needs doing there.

[sipian]

@tariq1890
This is the issue prometheus/prombench#159 being referred to.
Currently the Disk I/O Utilisation, Disk Space Used Basic, Filesystem Fullness graphs
are not showing the metrics of the SSD.
The relevant queries need to be updated in grafana_dashboard_prow_bench.yaml

[tariq1890]

@sipian Thanks for the info :). Will start looking into this

[krasi-georgiev]

I am closing this one as it is too broad so will continue to track in prometheus/prombench#159