Change grafana configuration to match hosted che configuration

[tomgeorge]

What does this PR do?

• Removes the dashboard provider yaml from grafana-dashboards.yaml
• Changes the name of grafana-dashboards ConfigMap to che-grafana-dashboards
• Create a grafana-dashboard-provider ConfigMap, that has a JSON spec (not yaml) defining the grafana dashboard provider, which looks like how hosted Che does it
• Change che-monitoring.yaml to mount the grafana-dashboard-provider ConfigMap at /etc/grafana/provisioning/dashboards, and to mount the che-grafana-dashboards ConfigMap at /grafana-dashboard-definitions/0

What issues does this PR fix or reference?

#14652

Release Notes

To deploy:

oc apply -f grafana-dashboard-provider.yaml -f grafana-dashboards.yaml -f grafana-datasources.yaml -f prometheus-config.yaml
oc process -f che-monitoring.yaml | oc apply -f -

Docs PR

eclipse-che/che-docs#835

EDIT: Added doc PR link

[che-bot]

Can one of the admins verify this patch?

[che-bot]

Can one of the admins verify this patch?

[tomgeorge]

cc @ibuziuk

[skabashnyuk]

configs look good. Can you also add ${OC_BINARY} apply -f ${BASE_DIR}/templates/monitoring/grafana-dashboard-provider.yaml to deploy_che.sh method deployMetrics(){

[ibuziuk]

is /grafana-dashboard-definitions/0 expected mountPath ?

[tomgeorge]

I chose that mount path because that is what downstream uses, grafana docs default is in /etc/grafana/provisioning/dashboards, but that is where we mount the grafana-dashboard-provider

[skabashnyuk]

kind of strange. Can you explain why we need that in upstream?

[tomgeorge]

The convention of Hosted Che is to place the dashboard models in /grafana-dashboard-definitions. /etc/grafana/provisioning/dashboards is taken as a mount point by the dashboard provider configuration. I put this change in upstream to make upstream Che look more like Hosted Che.

Alternatively, I could change the mount path, or place everything in /etc/grafana/provisioning/dashboards using subPath, and tell the provider configuration to look in the subPath for the dashboard models.

[skabashnyuk]

I think in the case of che-monitoring.yaml we should be close to what vanilla Grafana documentation recommends. @ibuziuk wdyt?

[tomgeorge]

@ibuziuk is on PTO for the next two weeks. We could put the dashboard in e.g. /etc/grafana/provisioning/dashboards/che, but that would be against the goal of being able to run the same configmaps upstream and downstream without changing them.

The mount path of the dashboards IMO is an implementation detail that users shouldn't really have to care about. They should just know that when they deploy the monitoring stack, they will see the dashboards that they are supposed to see.

[skabashnyuk]

From what we discussed with @ibuziuk the goal was to make grafana-dashboards.yaml as close to downstream as possible. Other configuration should be close to Grafana upstream in my opinion.

[tomgeorge]

That makes sense. I made a change to che-monitoring.yaml to reflect this.

[tomgeorge]

I think I need to make a docs PR to account for https://www.eclipse.org/che/docs/che-7/monitoring-che/#configuring-and-deploying-grafana

[skabashnyuk]

Change che-monitoring.yaml to mount the grafana-dashboard-provider ConfigMap at /etc/grafana/provisioning/dashboards, and to mount the che-grafana-dashboards ConfigMap at /grafana-dashboard-definitions/0

@tomgeorge can you give more details about the reasoning/motivation of that change?

[tomgeorge]

@skabashnyuk I thought that putting the dashboards in /etc/grafana/provisioning/dashboards/che would be in keeping with the upstream grafana convention of placing provisioning configuration in /etc/grafana/dashboards/provisioning

[skabashnyuk]

@tomgeorge do you want to add anything else or we can merge this pr?

[tomgeorge]

@skabashnyuk I think this is OK. I added eclipse-che/che-docs#835

[skabashnyuk]

waiting for eclipse-che/che-docs#835 approval

[skabashnyuk]

ci-build