Improve Grafana queries by adding multi-cluster support to distinguish between installed environments

[totallyGreg]

Gloo Edge Product

Enterprise

Gloo Edge Version

v1.15.7

Is your feature request related to a problem? Please describe.

Customer has installed edge in multiple environments and is using a single external Grafana server. Metrics from each of these servers are aggregated from each of these environments but since there is no way to distinguish where the metrics came from they are all displayed together.

In order to view these separate it is necessary to hand edit the configmaps glooe-grafana-custom-dashboards-v2 and gloo-observability-configby addingcluster=cmedev`

e.g. changing:

"expr": "glooe_solo_io_xds_insync{gloo=\"gloo\", namespace=\"$namespace\", resource=\"type.googleapis.com/envoy.config.listener.v3.Listener\"}"

to:

"expr": "glooe_solo_io_xds_insync{cluster=\"cmedev\", gloo=\"gloo\", namespace=\"$namespace\", resource=\"type.googleapis.com/envoy.config.listener.v3.Listener\"}"

This is tedious and causes problems on upgrade.

{"level":"warn","ts":"2023-12-15T21:59:14.055Z","logger":"observability.v1.event_loop.observability","caller":"syncer/default_dashboards.go:33","msg":"failed to save default dashboard to grafana: envoy: 412 The dashboard has been changed by someone else","version":"1.15.7"}

Describe the solution you'd like

Support multi-cluster in the generated dashboards.

Describe alternatives you've considered

No response

Additional Context

No response

[soloio-bot]

Zendesk ticket #3040 has been linked to this issue.

[davidjumani]

The new feature introduces a new observability option extraMetricQueryParameters : Specifies extra query parameters to add to all metric query definitions in the Grafana dashboards generated by gloo

gloo:
  settings:
    observabilityOptions:
      grafanaIntegration:
        extraMetricQueryParameters: gateway_proxy_id="rand\"

---

This will be in v1.17.0-beta1, v1.16.1 and v1.15.12 when released