[Bug] Grafanafolders permissions not synced

[oleksii-kalinin]

Describe the bug
Permissions are set via spec.permissions for the GrafanaFolder CR. Permissions are not synced with the folders after 10m.

Version
v5.0.2

To Reproduce
Steps to reproduce the behavior:

1. Apply permissions to the folder

 permissions: |
    {
      "items": [
        {
          "role": "Admin",
          "permission": 4
        },
        {
          "role": "Editor",
          "permission": 4
        }
      ]
    }

2. Verify folder permissions, only Admin or default set.

Expected behavior
Permissions are set to the folder right after sync.

Suspect component/Location where the bug might be occurring
GrafanaFolder

Screenshots
Wrong permission:

Default permissions are applied even if spec.permissions set

Runtime (please complete the following information):

• OS: Linux
• Grafana Operator Version: v5.0.2
• Environment: EKS
• Deployment type: deployed kustomize
• Other: [Other variables/things that might be relevant to this bug, versions of other services e.g. operator-sdk]

Additional context
Logs

❯ kl -n prometheus grafana-operator-controller-manager-5469d4c465-2w7bh --tail 100 | grep 'GrafanaFolderReconciler'
1.687943994094615e+09	INFO	GrafanaFolderReconciler	found matching Grafana instances for folder	{"controller": "grafanafolder", "controllerGroup": "grafana.integreatly.org", "controllerKind": "GrafanaFolder", "GrafanaFolder": {"name":"appdashboards","namespace":"prometheus"}, "namespace": "prometheus", "name": "appdashboards", "reconcileID": "c2818a6d-8dee-4af9-93b0-c766649f5fe9", "count": 1}
1.6879439951945112e+09	INFO	GrafanaFolderReconciler	found matching Grafana instances for folder	{"controller": "grafanafolder", "controllerGroup": "grafana.integreatly.org", "controllerKind": "GrafanaFolder", "GrafanaFolder": {"name":"kubernetesdashboards","namespace":"prometheus"}, "namespace": "prometheus", "name": "kubernetesdashboards", "reconcileID": "b24aac02-ce2c-4c65-bd5b-1745e8dbe344", "count": 1}
1.687943996149822e+09	INFO	GrafanaFolderReconciler	found matching Grafana instances for folder	{"controller": "grafanafolder", "controllerGroup": "grafana.integreatly.org", "controllerKind": "GrafanaFolder", "GrafanaFolder": {"name":"monitoringdashboards","namespace":"prometheus"}, "namespace": "prometheus", "name": "monitoringdashboards", "reconcileID": "e422b617-d864-48e3-a065-3e3580de7d36", "count": 1}
1.687944294118677e+09	INFO	GrafanaFolderReconciler	found matching Grafana instances for folder	{"controller": "grafanafolder", "controllerGroup": "grafana.integreatly.org", "controllerKind": "GrafanaFolder", "GrafanaFolder": {"name":"appdashboards","namespace":"prometheus"}, "namespace": "prometheus", "name": "appdashboards", "reconcileID": "076360be-c3f9-4936-b8b7-4cbaad315193", "count": 1}
1.6879442952216134e+09	INFO	GrafanaFolderReconciler	found matching Grafana instances for folder	{"controller": "grafanafolder", "controllerGroup": "grafana.integreatly.org", "controllerKind": "GrafanaFolder", "GrafanaFolder": {"name":"kubernetesdashboards","namespace":"prometheus"}, "namespace": "prometheus", "name": "kubernetesdashboards", "reconcileID": "7905bacd-3d66-46d4-a81f-2e81b468b7f7", "count": 1}
1.6879442961729028e+09	INFO	GrafanaFolderReconciler	found matching Grafana instances for folder	{"controller": "grafanafolder", "controllerGroup": "grafana.integreatly.org", "controllerKind": "GrafanaFolder", "GrafanaFolder": {"name":"monitoringdashboards","namespace":"prometheus"}, "namespace": "prometheus", "name": "monitoringdashboards", "reconcileID": "d39418b0-85f1-434b-964a-f1903816e537", "count": 1}
1.6879445941458797e+09	INFO	GrafanaFolderReconciler	found matching Grafana instances for folder	{"controller": "grafanafolder", "controllerGroup": "grafana.integreatly.org", "controllerKind": "GrafanaFolder", "GrafanaFolder": {"name":"appdashboards","namespace":"prometheus"}, "namespace": "prometheus", "name": "appdashboards", "reconcileID": "fa9701a9-197e-4cda-96ed-be893cbf3e1f", "count": 1}
1.6879445952469494e+09	INFO	GrafanaFolderReconciler	found matching Grafana instances for folder	{"controller": "grafanafolder", "controllerGroup": "grafana.integreatly.org", "controllerKind": "GrafanaFolder", "GrafanaFolder": {"name":"kubernetesdashboards","namespace":"prometheus"}, "namespace": "prometheus", "name": "kubernetesdashboards", "reconcileID": "b82da700-a4c4-4526-b181-feb64f1b74e5", "count": 1}
1.6879445961975048e+09	INFO	GrafanaFolderReconciler	found matching Grafana instances for folder	{"controller": "grafanafolder", "controllerGroup": "grafana.integreatly.org", "controllerKind": "GrafanaFolder", "GrafanaFolder": {"name":"monitoringdashboards","namespace":"prometheus"}, "namespace": "prometheus", "name": "monitoringdashboards", "reconcileID": "d45ce8b0-5f93-4bb2-8329-f68cdff5b5f7", "count": 1}

Full grafanafolder manifest

---
apiVersion: grafana.integreatly.org/v1beta1
kind: GrafanaFolder
metadata:
  name: appdashboards
spec:
  instanceSelector:
    matchLabels:
      dashboards: "grafana"
  # If title is not defined, the value will be taken from metadata.name
  title: App
  resyncPeriod: 30s # set for the testing purpose
  permissions: |
    {
      "items": [
        {
          "role": "Admin",
          "permission": 4
        },
        {
          "role": "Editor",
          "permission": 4
        }
      ]
    }

If I completely delete NS where operator and grafana deployed and redeploy - everything becomes ok.

[oleksii-kalinin]

Update:
After the initial deployment, a new RS is created. The only difference is in the added plugin grafana-opensearch-datasource 2.5.1 So in the first RS everything was fine (dashboards, datasources, and folders created with permissions). In the new RS, folders are created w/o permissions.
So the managing plugins in the dashboards/datasources are a mess and they should be moved to the grafana CR.

❯ diff old-rs.yaml new-rs.yaml
7,9c7,9
<     deployment.kubernetes.io/revision: "1"
<   creationTimestamp: "2023-06-30T08:30:04Z"
<   generation: 2
---
>     deployment.kubernetes.io/revision: "2"
>   creationTimestamp: "2023-06-30T08:31:39Z"
>   generation: 1
12,13c12,13
<     pod-template-hash: 749c444d97
<   name: grafana-deployment-749c444d97
---
>     pod-template-hash: 77b98747f8
>   name: grafana-deployment-77b98747f8
22,23c22,23
<   resourceVersion: "29828225"
<   uid: 557a577c-053a-441c-bf29-0d6e13e828e7
---
>   resourceVersion: "29828210"
>   uid: ea105a9f-20dd-4482-8819-922f1dfea191
25c25
<   replicas: 0
---
>   replicas: 1
29c29
<       pod-template-hash: 749c444d97
---
>       pod-template-hash: 77b98747f8
35c35
<         pod-template-hash: 749c444d97
---
>         pod-template-hash: 77b98747f8
50a51
>           value: grafana-opensearch-datasource 2.5.1
125,126c126,130
<   observedGeneration: 2
<   replicas: 0
---
>   availableReplicas: 1
>   fullyLabeledReplicas: 1
>   observedGeneration: 1
>   readyReplicas: 1
>   replicas: 1

So it looks like permissions won't be synced for the new grafana pod.

[NissesSenap]

@oleksii-kalinin could you share your grafana setup?
Also, can you share the exact folder CR that you have created?

We don't have any resyncPeriod in the folder. So depending on if you have a database or something like that, it could cause issues.

We need to troubleshoot this a bit more before being able to merge your PR.

[oleksii-kalinin]

Hi. resyncPeriod described in the API docs https://grafana-operator.github.io/grafana-operator/docs/api/#grafanafolderspec

Manifests:
Grafana CR

---
apiVersion: grafana.integreatly.org/v1beta1
kind: Grafana
metadata:
  name: grafana
  labels:
    dashboards: "grafana"
spec:
  config:
    log:
      mode: "console"
    auth:
      disable_login_form: "true"
      disable_signout_menu: "true"
    auth.anonymous:
      enabled: "false"
    auth.proxy:
      enabled: "true"
      header_name: "X-Webauth-Email"
      header_property: "username"
      auto_sign_up: "true"
      sync_ttl: "60"
      headers: "Email:X-Webauth-Email, Role:X-Webauth-Role"
    security:
      admin_user: admin
  deployment:
    spec:
      template:
        spec:
          containers:
            - name: grafana
              image: grafana/grafana:9.3.6
              env:
                - name: GF_INSTALL_PLUGINS
                  value: "grafana-opensearch-datasource 2.5.1"
              securityContext:
                runAsUser: 472
                runAsNonRoot: true
                runAsGroup: 472
              resources:
                limits:
                  cpu: 500m
                  memory: 2500Mi
                requests:
                  cpu: 100m
                  memory: 100Mi

Grafana Folder CR

---
apiVersion: grafana.integreatly.org/v1beta1
kind: GrafanaFolder
metadata:
  name: appdashboards
spec:
  instanceSelector:
    matchLabels:
      dashboards: "grafana"
  # If title is not defined, the value will be taken from metadata.name
  title: App
  resyncPeriod: 30s
  permissions: |
    {
      "items": [
        {
          "role": "Admin",
          "permission": 4
        },
        {
          "role": "Editor",
          "permission": 4
        }
      ]
    }

Grafana is fully stateless. The possible reason I've found: the operator can and will sync permissions to the first created grafana pod. The operator won't sync permissions to the new pod.
Here operator checks if permissions need to be synced.
https://github.com/grafana-operator/grafana-operator/blob/e65eb53f587abb382d8d808904dd043a9d393f5b/controllers/grafanafolder_controller.go#L332-L344
Here operator calculates hashes for the Title and Permissions, but they're unchanged since nothing new deployed, just a new pod come up.
https://github.com/grafana-operator/grafana-operator/blob/e65eb53f587abb382d8d808904dd043a9d393f5b/api/v1beta1/grafanafolder_types.go#L96-L104