Fix Grafana dashboards for Prometheus Overview

[poornima-krishnasamy]

Background

There are few dashboards which comes out of the box from kube-prometheus-stack.
https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack/templates/grafana/dashboards-1.14

But some of the dashboard doesnt show in Grafana. e.g. Prometheus Overview
https://grafana.live.cloud-platform.service.justice.gov.uk/d/cd596f85-461f-4236-b93a-3574d6f9f2ed/prometheus-overview?orgId=1&refresh=60s

The dashboard is essential to get useful insights about prometheus and debugging purposes.

If the default chart doesnt have a fix, find a alternative grafana dashboard available in open community, import the dashboard for us to use it

Investigate why this is not showing up. Is it due to the angular_js support issue.
prometheus-community/helm-charts#4398

Which part of the user docs does this impact

Communicate changes

• post for #cloud-platform-update
• Weeknotes item
• Show the Thing/P&A All Hands/User CoP
• Announcements channel

Questions / Assumptions

Definition of done

• Prometheus Overview dashboard fixed
• readme has been updated
• user docs have been updated
• another team member has reviewed
• smoke tests are green
• prepare demo for the team

Reference

How to write good user stories

[sj-williams]

The no data issue we are experiencing is due to a change applied to the Prometheus Overview default dashboard config, which now includes a cluster label for the dashboard queries. See this open issue for info.

At this time overriding / disabling the default dashboards via community chart is not available, so for now at least we can apply our own fixed version of the Prometheus Overview dashboard as a custom CM in monitoring repo.

[sj-williams]

Overview dashboard now working
https://grafana.live.cloud-platform.service.justice.gov.uk/d/BNLlmx9nk/prometheus-overview-working?orgId=1&refresh=60s