Grafana frequent "Bad Gateway" and "Templating Failed to upgrade legacy queries Datasource was not found" error messages

[Barteus]

Bug Description

When using Grafana frequently there are messages of error shown and also the filters on top are removed (emptied) for Kubeflow Dashboards (notebooks etc.)

To Reproduce

1. Bootstrap juju using MAAS VMs (HA)
2. Create a VM for COS in MAAS (Ubuntu 22.04)
3. Install microk8s on COS VM
4. Add microk8s to juju controller
5. Deploy COS on microk8s
6. Deploy Charmed K8s
7. Use the same controller to add Charmed K8s as a cloud
8. Deploy Kubeflow and add relations

#prometheus
$ juju add-relation grafana-agent-k8s:send-remote-write admin/cos.prometheus-receive-remote-write


#grafana
$ juju add-relation jupyter-controller:grafana-dashboard admin/cos.grafana-dashboards
$ juju add-relation seldon-controller-manager:grafana-dashboard admin/cos.grafana-dashboards
$ juju add-relation argo-controller:grafana-dashboard admin/cos.grafana-dashboards
$ juju add-relation dex-auth:grafana-dashboard admin/cos.grafana-dashboards
$ juju add-relation katib-controller:grafana-dashboard admin/cos.grafana-dashboards
$ juju add-relation kfp-api:grafana-dashboard admin/cos.grafana-dashboards
$ juju add-relation metacontroller-operator:grafana-dashboard admin/cos.grafana-dashboards
$ juju add-relation minio:grafana-dashboard admin/cos.grafana-dashboards
$ juju add-relation training-operator:grafana-dashboard admin/cos.grafana-dashboards
$ juju add-relation mlflow-server:grafana-dashboard admin/cos.grafana-dashboards

Environment

COS - latest/stable
microk8s - 1.24/stable
Charmed K8s - 1.24/stable
juju - 2.9.42

Relevant log output

https://pastebin.canonical.com/p/K5Ks2nwPqy/

Additional context

No response

[rbarry82]

@Barteus is this specific to any particular dashboard(s)?

[simskij]

Closing due to no response and a lot of fixes going into the charm since. If this is still a thing, feel free to open a new issue. 👍🏼