cephadm: persist the grafana.db file

[pcuzner]

This patch persists the grafana.db file, so the user can
create there own dashboard referencing ceph and node
metrics and not lose them on grafana restart! This also
ensures changes to users/passwords within grafana are
persisted.

Fixes: https://tracker.ceph.com/issues/49954

Signed-off-by: Paul Cuzner pcuzner@redhat.com

[sebastian-philipp]

I think we have to do more than just mounting a local path.

Imagine we have two grafana daemons deployed on host1 and host2. How do we make sure those grafana.dbs are equal? How do we make sure a newly deployed third daemon will have the same users?

If we don't properly manage the database file on the host, we're going to expose an inconsistent state across multiple daemons of the same service. I know this is also true for Prometheus to some degree, but in contrast to Grafana, Prometheus instances are supposed to be somewhat independent.

Also, we can't store the gafana.db in RADOS, as this would make the monitoring unavailable in case of a cluster error.

[pcuzner]

@sebastian-philipp - this is a regression from how a ceph-ansible deployment of grafana worked. This patch addresses this regression.

Your points about HA of the grafana and prometheus are valid but HA of these 3rd party services will require orch and dashboard changes which is perhaps better targeted for Quincy. My goal here is to fix the regression for the backport to Pacific.

[liewegas]

@sebastian-philipp - this is a regression from how a ceph-ansible deployment of grafana worked. This patch addresses this regression.

Your points about HA of the grafana and prometheus are valid but HA of these 3rd party services will require orch and dashboard changes which is perhaps better targeted for Quincy. My goal here is to fix the regression for the backport to Pacific.

Sounds reasonable to me. This is the same durability model that we have for prometheus.