[Bug]RBAC error - list namespace cause dashboard not insatlled

[wenerme]

Describe the bug
A clear and concise description of what the bug is.

E0309 15:03:18.721231       1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.20.2/tools/cache/reflector.go:167: Failed to watch *v1.Namespace: failed to list *v1.Namespace: namespaces is forbidden: User "system:serviceaccount:monitoring-system:controller-manager" cannot list resource "namespaces" in API group "" at the cluster scope

dashboard not installed

Version
Full semver version of the operator being used e.g. v3.10.1, v4.0.0

v4.2.0

To Reproduce
Steps to reproduce the behavior:

1. deploy grafana withdashboardNamespaceSelector: { }
2. add a dumy dashboard

spec:
  # ...
  dashboardLabelSelector:
  - matchExpressions:
    - { key: category, operator: In, values: [ plugin, base ] }
  - matchExpressions:
    - { key: instance, operator: In, values: [ incmon ] }
  dashboardNamespaceSelector: { }

apiVersion: integreatly.org/v1alpha1
kind: GrafanaDashboard
metadata:
  name: dumy
  labels:
    category: plugin
spec:
  json: "{}"
  customFolderName: System
  # https://grafana.com/grafana/plugins/
  plugins:
  - name: "grafana-piechart-panel"
    version: "1.6.2"
  - name: "grafana-clock-panel"
    version: "1.3.0"
  - name: "grafana-polystat-panel"
    version: "1.2.8"

Expected behavior

dashboard installed without error

[pb82]

@wenerme how did you install the Operator? When using OLM or kubebuilder you should end up with all the permissions provided, including namespace list.

[BapRx]

Hi, I have the same issue. I deployed grafana-operator using the Bitnami chart: https://github.com/bitnami/charts/tree/master/bitnami/grafana-operator
(versions tested: 2.2.10, 2.2.11)

The I have the following error:

E0318 15:50:28.009266       1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.20.2/tools/cache/reflector.go:167: Failed to watch *v1.Namespace: failed to list *v1.Namespace: namespaces is forbidden: User "system:serviceaccount:monitoring:grafana-operator" cannot list resource "namespaces" in API group "" at the cluster scope

The dashboard installation went through after commenting out the following, but that's not ideal.

  # dashboardLabelSelectors:
  #   - matchLabels:
  #       app.kubernetes.io/instance: "{{ .Release.Name }}"

[pb82]

@BapRx are you sure that you commented out the dashboardLabelSelectors? The operator would not import any dashboards if that was missing. Was it dashboardNamespaceSelectors by any chance?

[pb82]

I think we set up the permissions for the Operator to only run in namespaced mode. The documentation mentions that you need to apply cluster roles when using namspace label selectors (or the --namespaces flag): https://github.com/grafana-operator/grafana-operator/blob/7754cd15386ff6da1e3e7b820f8baf53e6dd9356/documentation/dashboards.md#discovering-dashboards-in-other-namespaces

However it looks like we no longer add the list permission for namespaces even when installing via kustomize. We don't want to add that permission to OLM installations, so how about we provide an example cluster role / binding and instructions for how to apply it? We can add it to those: https://github.com/grafana-operator/grafana-operator/tree/master/deploy/cluster_roles

what do you think?

[BapRx]

Hi thank your for your comment, you're right, I commented dashboardNamespaceSelector

  # dashboardNamespaceSelector:
  #   matchLabels:
  #     app.kubernetes.io/name: monitoring

The helm chart does create the cluster issuer:
❯ k get clusterroles grafana-operator -oyaml

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"rbac.authorization.k8s.io/v1","kind":"ClusterRole","metadata":{"annotations":{},"creationTimestamp":null,"name":"grafana-operator"},"rules":[{"apiGroups":[""],"resources":["events"],"verbs":["get","list","watch","create","delete","update","patch"]},{"apiGroups":["integreatly.org"],"resources":["grafanadashboards","grafanadatasources","grafanadatasources/status"],"verbs":["get","list","create","update","delete","deletecollection","watch"]}]}
    meta.helm.sh/release-name: grafana-operator
    meta.helm.sh/release-namespace: monitoring
  creationTimestamp: "2022-03-23T16:22:40Z"
  labels:
    app.kubernetes.io/instance: grafana-operator
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: grafana-operator
    env: prod
    helm.sh/chart: grafana-operator-2.2.10
    k8s-app: grafana
  name: grafana-operator
  resourceVersion: "184098300"
  uid: 1581d720-d761-4a80-8064-ab88512356a2
rules:
- apiGroups:
  - ""
  resources:
  - configmaps
  - persistentvolumeclaims
  - secrets
  - serviceaccounts
  - services
  verbs:
  - create
  - delete
  - get
  - list
  - patch
  - update
  - watch
- apiGroups:
  - ""
  resources:
  - events
  verbs:
  - create
  - get
  - list
  - watch
  - patch
- apiGroups:
  - ""
  resources:
  - pods
  verbs:
  - get
  - list
  - watch
- apiGroups:
  - apps
  - extensions
  resources:
  - deployments
  - deployments/finalizers
  verbs:
  - create
  - delete
  - get
  - list
  - patch
  - update
  - watch
- apiGroups:
  - integreatly.org
  resources:
  - grafanadashboards
  verbs:
  - create
  - delete
  - get
  - list
  - patch
  - update
  - watch
- apiGroups:
  - integreatly.org
  resources:
  - grafanadashboards/status
  verbs:
  - get
  - patch
  - update
- apiGroups:
  - integreatly.org
  resources:
  - grafanadatasources
  verbs:
  - create
  - delete
  - get
  - list
  - patch
  - update
  - watch
- apiGroups:
  - integreatly.org
  resources:
  - grafanadatasources/status
  verbs:
  - get
  - patch
  - update
- apiGroups:
  - integreatly.org
  resources:
  - grafananotificationchannels
  verbs:
  - create
  - delete
  - get
  - list
  - patch
  - update
  - watch
- apiGroups:
  - integreatly.org
  resources:
  - grafananotificationchannels/status
  verbs:
  - get
  - patch
  - update
- apiGroups:
  - integreatly.org
  resources:
  - grafanas
  - grafanas/finalizers
  verbs:
  - create
  - delete
  - get
  - list
  - patch
  - update
  - watch
- apiGroups:
  - integreatly.org
  resources:
  - grafanas/status
  verbs:
  - get
  - patch
  - update
- apiGroups:
  - networking.k8s.io
  resources:
  - ingresses
  verbs:
  - create
  - delete
  - get
  - list
  - patch
  - update
  - watch
- apiGroups:
  - route.openshift.io
  resources:
  - routes
  verbs:
  - create
  - delete
  - get
  - list
  - patch
  - update
  - watch
- nonResourceURLs:
  - /metrics
  verbs:
  - get

I tried installing directly the manifest from the grafana-operator repo, without success. https://github.com/grafana-operator/grafana-operator/tree/master/deploy/cluster_roles

The only way to make it work is to disable the dashboardNamespaceSelector

[pb82]

@BapRx the cluster role created from the helm chart also doesn't have the namespace list permission. We'll likely add those permissions back to the example cluster role, but you'd need to apply them yourself.

[BapRx]

@pb82 ok, I'll update the helm chart with the new cluster roles once you update the examples. Thank you!