Config change after deployment does not update Grafana dashboard

[cbartz]

Bug Description

Changing the configuration (e.g. grafana_dashboards) does not update the Grafana dashboard. This seems to be due to the fact that the dashboard is only updated during an install or upgrade event:

cos-configuration-k8s-operator/lib/charms/grafana_k8s/v0/grafana_dashboard.py

Lines 1051 to 1052 in 051e146

	self.framework.observe(self._charm.on.leader_elected, self._update_all_dashboards_from_dir)
	self.framework.observe(self._charm.on.upgrade_charm, self._update_all_dashboards_from_dir)

As a user, I would expect the dashboard data to be updated when the configuration is changed.

To Reproduce

1. juju deploy --trust grafana-k8s grafana
2. juju deploy cos-configuration-k8s --config git_repo=https://github.com/canonical/github-runner-operator --config git_branch=main
3. juju config cos-configuration-k8s grafana_dashboards_path=src/grafana_dashboard_metrics
4. juju relate cos-configuration-k8s:grafana-dashboards grafana:grafana-dashboard
5. juju show-unit grafana/0 --format json | jq '."grafana/0"."relation-info"'

The last command shows an empty dashboard in the integration data. It would not be empty if the configuration value grafana_dashboards_path=src/grafana_dashboard_metric had been passed at deployment time.

Environment

Juju (tested with 3.1 and 2.9) within multipass using microk8s.

Relevant log output

nit-cos-configuration-k8s-0: 14:14:25 INFO unit.cos-configuration-k8s/0.juju-log git-sync: I1012 12:14:24.881140     135 main.go:473] "level"=0 "msg"="starting up" "pid"=135 "args"=["/git-sync","--repo","https://github.com/canonical/github-runner-operator","--branch","main","--rev","HEAD","--depth","1","--root","/git","--dest","repo","--one-time"]
unit-cos-configuration-k8s-0: 14:14:25 INFO juju.worker.uniter.operation ran "config-changed" hook (via hook dispatching script: dispatch)
unit-cos-configuration-k8s-0: 14:14:40 INFO juju.worker.uniter.operation ran "grafana-dashboards-relation-created" hook (via hook dispatching script: dispatch)
unit-grafana-0: 14:14:42 INFO unit.grafana/0.juju-log grafana-dashboard:39: Restarted grafana-k8s
unit-grafana-0: 14:14:43 INFO juju.worker.uniter.operation ran "grafana-dashboard-relation-changed" hook (via hook dispatching script: dispatch)
unit-grafana-0: 14:14:43 INFO juju.worker.uniter.operation ran "grafana-dashboard-relation-joined" hook (via hook dispatching script: dispatch)
unit-grafana-0: 14:14:44 INFO juju.worker.uniter.operation ran "grafana-relation-changed" hook (via hook dispatching script: dispatch)
unit-grafana-0: 14:14:45 INFO unit.grafana/0.juju-log grafana-dashboard:39: HTTP Request: GET https://10.152.183.1/apis/apps/v1/namespaces/observability/statefulsets/grafana "HTTP/1.1 200 OK"
unit-grafana-0: 14:14:45 INFO unit.grafana/0.juju-log grafana-dashboard:39: HTTP Request: GET https://10.152.183.1/api/v1/namespaces/observability/pods/grafana-0 "HTTP/1.1 200 OK"
unit-grafana-0: 14:14:45 INFO unit.grafana/0.juju-log grafana-dashboard:39: reqs=ResourceRequirements(claims=None, limits={}, requests={'cpu': '0.25', 'memory': '200Mi'}), templated=ResourceRequirements(claims=None, limits=None, requests={'cpu': '250m', 'memory': '200Mi'}), actual=ResourceRequirements(claims=None, limits=None, requests={'cpu': '250m', 'memory': '200Mi'})
unit-grafana-0: 14:14:45 INFO unit.grafana/0.juju-log grafana-dashboard:39: HTTP Request: GET https://10.152.183.1/apis/apps/v1/namespaces/observability/statefulsets/grafana "HTTP/1.1 200 OK"
unit-grafana-0: 14:14:45 INFO unit.grafana/0.juju-log grafana-dashboard:39: HTTP Request: GET https://10.152.183.1/api/v1/namespaces/observability/pods/grafana-0 "HTTP/1.1 200 OK"
unit-grafana-0: 14:14:45 INFO unit.grafana/0.juju-log grafana-dashboard:39: Initializing dashboard provisioning path
unit-grafana-0: 14:14:46 INFO unit.grafana/0.juju-log grafana-dashboard:39: Restarted grafana-k8s
unit-grafana-0: 14:14:46 INFO juju.worker.uniter.operation ran "grafana-dashboard-relation-changed" hook (via hook dispatching script: dispatch)
unit-grafana-0: 14:14:47 INFO juju.worker.uniter.operation ran "grafana-relation-changed" hook (via hook dispatching script: dispatch)

Additional context

No response

[PietroPasotti]

I'm not sure that that is the issue, because while the relation provider class is in fact only doing that 'automatically' on install and upgrade, the charm itself is doing that in its common exit hook (https://github.com/canonical/cos-configuration-k8s-operator/blob/051e1461430edb2d342f8b2088a79d363fd02140/src/charm.py#L162C22-L162C22)

which is triggered, among other things, on config-changed.

So there must be something else going on. Perhaps a bug in the hashing?

[cbartz]

Thanks @PietroPasotti for the quick reply. It seems that the update of the boards is not triggered if the git hash does not change, which is indeed true in this case because the config grafana_dashboards is independent of the hash.

cos-configuration-k8s-operator/src/charm.py

Line 363 in 051e146

# Use the contents of the hash file as an indication for a change in the repo.

[dstathis]

We should probably just always update on config-changed.

[IbraAoad]

The issue isn't reproducible when following the original steps and the updated dashboards path in the github-runner-operator (grafana_dashboards_path=src/grafana_dashboards).

However, the issue arises if step 4 is performed before step 3 in the reproduction steps (i.e., relating to Grafana before setting the grafana_dashboards_path config). The fix for this issue is addressed in canonical/grafana-k8s-operator#335

[IbraAoad]

should be fixed by #98