mgr: adding filters for prometheus endpoint configuration

[ehuzii]

Is this a bug report or feature request?

• Feature Request

What should the feature do:
When we started using the new dashboard, we had to specify the prometheus server using the parameter: prometheusEndpoint, from which mgr takes metrics. but since we collect metrics for several ceph clusters in one place, we are faced with the problem of displaying the data correctly.

Could you consider adding a parameter where you could specify query filtering labels when accessing Prometheus?
for example, at the moment I see the following requests to prometheus: sum(rate(ceph_pool_rd[1m]))
it would be super to change it to: sum(rate{label="value"}(ceph_pool_rd[1m]))

What is use case behind this feature:

Us parameter un cluster values.yaml:

cephClusterSpec:
  dashboard:
    prometheusLabelFilter:
      label1: value1
      label2: value2
      ...

Environment:

Kubernetes version (use kubectl version): v1.28.6
Kubernetes cluster type (e.g. Tectonic, GKE, OpenShift): vanilla kubeadm

[travisn]

@ehuzii Have you tried using Kustomize to modify the prometheus rules as described in the monitoring guide? Other efforts to customize alerts in the past have not worked out.

[rkachach]

@ehuzii thank you for opening this feature request. As I understand it, basically you have several clusters which are being monitored by the same Prometheus instance and you'd like to have some mechanism to separate/filter the incoming metrics because at this moment there's no way to distinguish metrics coming from different clusters. Please, correct me if my understanding is not correct.

The thing is that latest Ceph upcoming release (squid) will come with a new multi-cluster management feature which add a similar support for what you are asking for. Basically, the user will be able to manage/monitor multiple clusters from a single dashboard. For that, a new label cluster (with the fsid) will be included in the metrics. This way, the dashboard by default will be filtering only the metrics that correspond to the current selected cluster (the actual cluster if only one). I think that should be enough to solve your issue if I'm understanding your feature-request correctly.

This functionality should be coming soon and Rook will also add support for squid in the next releases. So if that covers your needs I'll just recommend to stay tuned for future updates both on Ceph and Rook projects.

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in a week if no further activity occurs. Thank you for your contributions.