How can I create botkube without giving "clusterrole" #227

daegeun-ha · 2019-12-04T01:04:12Z

Is your feature request related to a problem? Please describe.
In our company, kubernetes cluster is big, so not every developers has authorization to make 'clusterrole' resource. And if I use just 'role' than 'clusetrrole', lots of error msgs are appear like below.
(pkg/mod/k8s.io/client-go@v0.0.0-20190918160344-1fbdaa4c8d90/tools/cache/reflector.go:96: Failed to list *v1.StatefulSet: statefulsets.apps is forbidden: User "system:serviceaccount:test-rbac:botkube-sa" cannot list resource "statefulsets" in API group "apps" at the cluster scope)

Describe the solution you'd like
Can I use different k8s library for getting events from k8s api server?

Describe alternatives you've considered
Can I use different k8s library for getting events from k8s api server? ( I don't have a exact solution...)

Additional context

PrasadG193 · 2019-12-09T07:03:46Z

@daegeun-ha if we use clusterrole, then BotKube won't be able to monitor cluster-scoped resources like - PV, Namespace, Node, etc

We should have a field in BotKube config (something like isNamespaced) to restrict the scope of BotKube within a Namespace

daegeun-ha · 2019-12-16T01:56:21Z

We also know about that problem (won't be able to monitor cluster-scoped resources)
But if I give just normal 'role' & 'rolebinding' to botkube, errors are occured like below.

Can I resolve this problem?

(E1216 01:55:23.030699 1 reflector.go:123] pkg/mod/k8s.io/client-go@v0.0.0-20190918160344-1fbdaa4c8d90/tools/cache/reflector.go:96: Failed to list *v1.StatefulSet: statefulsets.apps is forbidden: User "system:serviceaccount:monitoring:botkube-sa" cannot list resource "statefulsets" in API group "apps" at the cluster scope
E1216 01:55:23.032901 1 reflector.go:123] pkg/mod/k8s.io/client-go@v0.0.0-20190918160344-1fbdaa4c8d90/tools/cache/reflector.go:96: Failed to list *v1.RoleBinding: rolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:monitoring:botkube-sa" cannot list resource "rolebindings" in API group "rbac.authorization.k8s.io" at the cluster scope
E1216 01:55:23.032944 1 reflector.go:123] pkg/mod/k8s.io/client-go@v0.0.0-20190918160344-1fbdaa4c8d90/tools/cache/reflector.go:96: Failed to list *v1.ClusterRole: clusterroles.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:monitoring:botkube-sa" cannot list resource "clusterroles" in API group "rbac.authorization.k8s.io" at the cluster scope
E1216 01:55:23.033483 1 reflector.go:123] pkg/mod/k8s.io/client-go@v0.0.0-20190918160344-1fbdaa4c8d90/tools/cache/reflector.go:96: Failed to list *v1.DaemonSet: daemonsets.apps is forbidden: User "system:serviceaccount:monitoring:botkube-sa" cannot list resource "daemonsets" in API group "apps" at the cluster scope
E1216 01:55:23.033600 1 reflector.go:123] pkg/mod/k8s.io/client-go@v0.0.0-20190918160344-1fbdaa4c8d90/tools/cache/reflector.go:96: Failed to list *v1.Namespace: namespaces is forbidden: User "system:serviceaccount:monitoring:botkube-sa" cannot list resource "namespaces" in API group "" at the cluster scope
E1216 01:55:23.033607 1 reflector.go:123] pkg/mod/k8s.io/client-go@v0.0.0-20190918160344-1fbdaa4c8d90/tools/cache/reflector.go:96: Failed to list *v1.ReplicaSet: replicasets.apps is forbidden: User "system:serviceaccount:monitoring:botkube-sa" cannot list resource "replicasets" in API group "apps" at the cluster scope
E1216 01:55:23.033717 1 reflector.go:123] pkg/mod/k8s.io/client-go@v0.0.0-20190918160344-1fbdaa4c8d90/tools/cache/reflector.go:96: Failed to list *v1.PersistentVolume: persistentvolumes is forbidden: User "system:serviceaccount:monitoring:botkube-sa" cannot list resource "persistentvolumes" in API group "" at the cluster scope
E1216 01:55:23.037779 1 reflector.go:123] pkg/mod/k8s.io/client-go@v0.0.0-20190918160344-1fbdaa4c8d90/tools/cache/reflector.go:96: Failed to list *v1.Event: events is forbidden: User "system:serviceaccount:monitoring:botkube-sa" cannot list resource "events" in API group "" at the cluster scope
E1216 01:55:23.129221 1 reflector.go:123] pkg/mod/k8s.io/client-go@v0.0.0-20190918160344-1fbdaa4c8d90/tools/cache/reflector.go:96: Failed to list *v1.Service: services is forbidden: User "system:serviceaccount:monitoring:botkube-sa" cannot list resource "services" in API group "" at the cluster scope)

daegeun-ha · 2019-12-26T06:53:03Z

I hope this problem is solved. If the problem is solved, our team can use botkube to improve work efficiency.

PrasadG193 · 2019-12-26T06:58:56Z

Hey @daegeun-ha,
Sorry, I missed to reply to your earlier comment. Could you please let us know what role bindings you are using? We are planning to use dynamic informers which will hopefully solve this issue (#200), but that will take some time. Let's try to find another solution till then.

daegeun-ha · 2019-12-31T02:06:15Z

@PrasadG193 Thanks for reply!
our role, service account, role binding are like below:

[role]
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
creationTimestamp: 2019-12-16T01:55:14Z
name: botkube-role
namespace: monitoring
resourceVersion: "274459817"
selfLink: /apis/rbac.authorization.k8s.io/v1/namespaces/monitoring/roles/botkube-role
uid: 1a3c793d-1fa7-11ea-bbbc-fa163eaf6fe5
rules:

apiGroups:
- rbac.authorization.k8s.io
- extensions
- apps
- batch
  resources:
- pods
- nodes
- services
- namespaces
- secrets
- configmaps
- deployments
- daemonsets
- replicasets
- statefulsets
- ingresses
- jobs
- cronjobs
- roles
- rolebindings
  verbs:
- get
- list
- watch
- create
- update
- patch
- delete
- deletecollection

[serviceAccount]
apiVersion: v1
kind: ServiceAccount
metadata:
creationTimestamp: 2019-12-16T01:55:14Z
name: botkube-sa
namespace: monitoring
resourceVersion: "274459820"
selfLink: /api/v1/namespaces/monitoring/serviceaccounts/botkube-sa
uid: 1a3b1fa7-1fa7-11ea-bbbc-fa163eaf6fe5
secrets:

name: ***********************(hidden)

[roleBinding]
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
creationTimestamp: 2019-12-16T01:55:14Z
name: botkube-rb
namespace: monitoring
resourceVersion: "274459819"
selfLink: /apis/rbac.authorization.k8s.io/v1/namespaces/monitoring/rolebindings/botkube-rb
uid: 1a3dbdf6-1fa7-11ea-bbbc-fa163eaf6fe5
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: botkube-role
subjects:

kind: ServiceAccount
name: botkube-sa

PrasadG193 · 2019-12-31T02:07:35Z

Also, can you send the BotKube resource configuration ?

daegeun-ha · 2019-12-31T02:08:04Z

And.. I think it's the problem of getting k8s event from api server without clusterrole. Could a dynamic informer solve this problem..?

daegeun-ha · 2019-12-31T02:10:03Z

Below is botkube resource configuration ! (Currently, for experiment, I removed other resources from configuration)

config:
resources:
- name: pod # Name of the resources e.g pod, deployment, ingress, etc. (Resource name must be in singular form)
namespaces:
include:
- all
ignore: # List of namespaces to be ignored (omitempty), used only with include: all
- # example : include [all], ignore [x,y,z]
events: # List of lifecycle events you want to receive, e.g create, update, delete, error OR all
- create
- delete
- error
- name: service
namespaces:
include:
- all
ignore:
-
events:
- create
- delete
- error
- name: deployment
namespaces:
include:
- all
ignore:
-
events:
- create
- update
- delete
- error

PrasadG193 · 2019-12-31T02:20:35Z

If you set “default” service account in deployment, you don’t need to create RBAC resources - clusterrrole, clusterrolebindings and serviceaccount. We should support changing service account through helm chart

PrasadG193 · 2019-12-31T02:24:22Z

If you use role and rolebindings or “default” namespace, setting “all” namespace in resource config won’t work. You will be able to watch only namespace in which BotKube is deployed. Can you try replacing “all” namespaces with appropriate namespace?

daegeun-ha · 2019-12-31T05:52:59Z

@PrasadG193 OK, I'll try it the day after tomorrow. Happy new year! :>

nadilas · 2020-02-13T04:26:59Z

I have set include namespaces to one particular namespace and using the default service account, and only watching namespace relevant elements, still I get the message that the account doesn't have cluster scope.

| E0213 04:22:50.567085 1 reflector.go:123] pkg/mod/k8s.io/client-go@v0.0.0-20190918160344-1fbdaa4c8d90/tools/cache/reflector.go:96: Failed to list *v1.Node: nodes is forbidden: User "system:serviceaccount:namespace:default" cannot list nodes at the cluster scope: no RBAC policy matched

I presume the issue lies in these two being initialized:

INFO[2020-02-13T04:22:50Z] Registering the filter NamespaceChecker
--
| INFO[2020-02-13T04:22:50Z] Registering the filter NodeEventsChecker

Although I don't understand this line:
E0213 04:22:50.566979 1 reflector.go:123] pkg/mod/k8s.io/client-go@v0.0.0-20190918160344-1fbdaa4c8d90/tools/cache/reflector.go:96: Failed to list *v1.ReplicaSet: replicasets.apps is forbidden: User "system:serviceaccount:namespace:default" cannot list replicasets.apps at the cluster scope: no RBAC policy matched

PrasadG193 · 2020-02-26T08:14:43Z

@nadilas How are you deploying BotKube? I am assuming you are using deploy-all-in-one.yaml. If you are deploying BotKube in other namespace than botkube, then you need to update ClusteRoleBinding accordingly. Please make sure you are setting subject in ClusterRoleBinding correctly: https://github.com/infracloudio/botkube/blob/master/deploy-all-in-one.yaml#L260

nadilas · 2020-02-26T09:50:27Z

@PrasadG193 hi, I’m deploying namespace in a dedicated xy namespace, which I want to monitor. It should be namespaced and only look at resources within that namespace. I do not have cluster level access, hence cannot create a ClusterRoleBinding. I want to restrict it to just the one.

PrasadG193 · 2020-02-26T10:38:12Z

Understood. By default BotKube register informers for all the resources. There should be a check to skip cluster level resources as proposed in this issue.
Can you update your resource settings and set namespace xy for all the resources?

nadilas · 2020-03-29T09:22:39Z

@PrasadG193 Should be no issue, although I think I had that initially.. let me try it first thing on Monday.

PrasadG193 · 2020-04-24T05:34:12Z

Now it is possible to change default rbac rule while installing the BotKube via helm. Please have a look at default values: https://github.com/infracloudio/botkube/blob/develop/helm/botkube/values.yaml#L311

nadilas · 2020-05-24T11:04:27Z

Understood. By default BotKube register informers for all the resources. There should be a check to skip cluster level resources as proposed in this issue.
Can you update your resource settings and set namespace xy for all the resources?

Sorry @PrasadG193 ... I eventually got around to test this... it doesn't seem to work at all. I have set the included namespace for every single resource, like:

- name: pod
        namespaces:
          include:
            - namespace
        events:
          - all

But on startup it still runs into the same issue:

  | E0524 11:01:29.849217       1 reflector.go:156] pkg/mod/k8s.io/client-go@v0.17.0/tools/cache/reflector.go:108: Failed to list *v1.Service: services is forbidden: User "system:serviceaccount:namespace:default" cannot list services at the cluster scope: no RBAC policy matched
  | E0524 11:01:30.049081       1 reflector.go:156] pkg/mod/k8s.io/client-go@v0.17.0/tools/cache/reflector.go:108: Failed to list *v1.Node: nodes is forbidden: User "system:serviceaccount:namespace:default" cannot list nodes at the cluster scope: no RBAC policy matched
  | E0524 11:01:30.249372       1 reflector.go:156] pkg/mod/k8s.io/client-go@v0.17.0/tools/cache/reflector.go:108: Failed to list *v1.Pod: pods is forbidden: User "system:serviceaccount:namespace:default" cannot list pods at the cluster scope: no RBAC policy matched

PrasadG193 · 2020-09-26T12:27:01Z

Fixed with #253

kelvinwijaya · 2020-10-02T16:40:56Z

Hi @PrasadG193

I did a test and it seem the clusterrole is still needed, and have been using the latest image with tag v0.11.0 as per your merged for fix #253 but no luck

E1002 16:36:21.845948       1 reflector.go:156] pkg/mod/k8s.io/client-go@v0.17.0/tools/cache/reflector.go:108:
Failed to list *unstructured.Unstructured: events is forbidden: User "system:serviceaccount:mynamepsace:botkube-sa" cannot list resource "events" in API group "" at the cluster scope

I have follow the same instructions above as well to list all the resources to include namespace

namespace: 
  include:
    - mynamespace

PrasadG193 · 2020-10-02T16:44:54Z

Thanks @kelvinwijaya for reporting. I will take a look

akankshakumari393 · 2020-10-27T13:24:21Z

hey @kelvinwijaya Can you please share the role, rolebinding, serviceaccount you used to test.

akankshakumari393 · 2020-10-29T09:36:29Z

@PrasadG193 we are using DynamicInformer which by default takes all namespace https://github.com/infracloudio/botkube/blob/develop/pkg/utils/utils.go#L141 Also, we have ResourceInformer Map https://github.com/infracloudio/botkube/blob/develop/pkg/utils/utils.go#L144 which stores Infomer per resource for all namespace by default. I think we can modify the map and store the informer for a particular namespace for that resource. Let me know your take on it.

LaurentDumont · 2021-07-16T16:23:34Z

Is this still current?

I am trying to deploy botkube in a cluster where we cannot leverage the ClusterRole as low-privileged tenants.

I am able to list ressources within my own namespace and have scoped the config to do just that. But I still see botkube trying to list ressources in that namespace at the Cluster scope.

E0716 16:22:17.633792       1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.20.5/tools/cache/reflector.go:167: Failed to watch *unstructured.Unstructured: failed to list *unstructured.Unstructured: events is forbidden: User "system:serviceaccount:NAMESPACE_HERE:botkube" cannot list resource "events" in API group "" at the cluster scope
E0716 16:22:20.327095       1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.20.5/tools/cache/reflector.go:167: Failed to watch *unstructured.Unstructured: failed to list *unstructured.Unstructured: pods is forbidden: User "system:serviceaccount:NAMESPACE_HERE:botkube" cannot list resource "pods" in API group "" at the cluster scope

Our service-account manifest

---
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: kubebot
  namespace: NAMESPACE_HERE
rules:
- apiGroups: [""]
  resources: ["daemonsets", "deployments", "pods", "replicationcontrollers"]
  verbs: ["get", "watch", "list"]
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: kubebot
  namespace: NAMESPACE_HERE
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: RoleBinding
metadata:
  name: kubebot
  namespace: NAMESPACE_HERE
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: kubebot
subjects:
  - kind: ServiceAccount
    name: kubebot
    namespace: NAMESPACE_HERE

As an example, kubewatch is able to restrict itself to a namespace, without requiring Cluster wide permissions.

https://github.com/bitnami-labs/kubewatch

pkosiec · 2023-06-01T07:57:07Z

Hi,
Please check out the latest Botkube which comes with RBAC configuration: https://docs.botkube.io/configuration/rbac
If there's anything missing, please create a new issue. Cheers!

daegeun-ha added the enhancement New feature or request label Dec 4, 2019

PrasadG193 added the help wanted Extra attention is needed label Dec 9, 2019

PrasadG193 closed this as completed Sep 26, 2020

PrasadG193 reopened this Oct 5, 2020

PrasadG193 added this to To do in v0.12.0 via automation Oct 5, 2020

PrasadG193 removed this from To do in v0.12.0 Feb 2, 2021

PrasadG193 mentioned this issue Nov 29, 2021

Is it possible to run Botkube without ClusterRole and ClusterRoleBinding? #530

Closed

PrasadG193 added the good first issue Good for newcomers label Nov 29, 2021

PrasadG193 added this to To do in v0.12.4 Nov 29, 2021

brampling mentioned this issue Aug 4, 2022

Role-based Access Control #675

Closed

pkosiec closed this as completed Jun 1, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

How can I create botkube without giving "clusterrole" #227

How can I create botkube without giving "clusterrole" #227

daegeun-ha commented Dec 4, 2019

PrasadG193 commented Dec 9, 2019 •

edited

daegeun-ha commented Dec 16, 2019

daegeun-ha commented Dec 26, 2019

PrasadG193 commented Dec 26, 2019 •

edited

daegeun-ha commented Dec 31, 2019

PrasadG193 commented Dec 31, 2019

daegeun-ha commented Dec 31, 2019 •

edited

daegeun-ha commented Dec 31, 2019 •

edited

PrasadG193 commented Dec 31, 2019

PrasadG193 commented Dec 31, 2019

daegeun-ha commented Dec 31, 2019

nadilas commented Feb 13, 2020

PrasadG193 commented Feb 26, 2020 •

edited

nadilas commented Feb 26, 2020

PrasadG193 commented Feb 26, 2020

nadilas commented Mar 29, 2020

PrasadG193 commented Apr 24, 2020

nadilas commented May 24, 2020 •

edited

PrasadG193 commented Sep 26, 2020

kelvinwijaya commented Oct 2, 2020 •

edited

PrasadG193 commented Oct 2, 2020

akankshakumari393 commented Oct 27, 2020

akankshakumari393 commented Oct 29, 2020

LaurentDumont commented Jul 16, 2021 •

edited

pkosiec commented Jun 1, 2023

How can I create botkube without giving "clusterrole" #227

How can I create botkube without giving "clusterrole" #227

Comments

daegeun-ha commented Dec 4, 2019

PrasadG193 commented Dec 9, 2019 • edited

daegeun-ha commented Dec 16, 2019

daegeun-ha commented Dec 26, 2019

PrasadG193 commented Dec 26, 2019 • edited

daegeun-ha commented Dec 31, 2019

PrasadG193 commented Dec 31, 2019

daegeun-ha commented Dec 31, 2019 • edited

daegeun-ha commented Dec 31, 2019 • edited

PrasadG193 commented Dec 31, 2019

PrasadG193 commented Dec 31, 2019

daegeun-ha commented Dec 31, 2019

nadilas commented Feb 13, 2020

PrasadG193 commented Feb 26, 2020 • edited

nadilas commented Feb 26, 2020

PrasadG193 commented Feb 26, 2020

nadilas commented Mar 29, 2020

PrasadG193 commented Apr 24, 2020

nadilas commented May 24, 2020 • edited

PrasadG193 commented Sep 26, 2020

kelvinwijaya commented Oct 2, 2020 • edited

PrasadG193 commented Oct 2, 2020

akankshakumari393 commented Oct 27, 2020

akankshakumari393 commented Oct 29, 2020

LaurentDumont commented Jul 16, 2021 • edited

pkosiec commented Jun 1, 2023

PrasadG193 commented Dec 9, 2019 •

edited

PrasadG193 commented Dec 26, 2019 •

edited

daegeun-ha commented Dec 31, 2019 •

edited

daegeun-ha commented Dec 31, 2019 •

edited

PrasadG193 commented Feb 26, 2020 •

edited

nadilas commented May 24, 2020 •

edited

kelvinwijaya commented Oct 2, 2020 •

edited

LaurentDumont commented Jul 16, 2021 •

edited