Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

RBAC broken in 7.2 if using ClusterRole #1163

Closed
dmegyesi opened this issue Jun 26, 2018 · 3 comments

Comments

@dmegyesi
Copy link

commented Jun 26, 2018

In the following file: https://github.com/appscode/voyager/blob/release-7.2/hack/deploy/rbac-list.yaml

There are 2 role definitions at the beginning:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: voyager-operator
  namespace: ${VOYAGER_NAMESPACE}
  labels:
    app: voyager

and

apiVersion: rbac.authorization.k8s.io/v1
kind: ${VOYAGER_ROLE_TYPE}
metadata:
  name: voyager-operator
  namespace: ${VOYAGER_NAMESPACE}
  labels:
    app: voyager

If you use --restrict-to-namespace=false, for example, I'm only deploying it in kube-system, then the above mentioned parts will overwrite each other, because of this: https://github.com/appscode/voyager/blob/release-7.2/hack/deploy/voyager.sh#L99

Eventually it will render two ClusterRole objects, both at the same name of voyager-operator. So first it will apply one definition, then later overwrite it with the second one, therefore cancelling the definitions of the first file completely.

Because of this, all of the following permissions are missing:

- apiGroups:
  - apiextensions.k8s.io
  resources:
  - customresourcedefinitions
  verbs:
  - "*"
- apiGroups:
  - voyager.appscode.com
  resources: ["*"]
  verbs: ["list"]
- apiGroups: [""]
  resources:
  - nodes
  verbs: ["list", "watch", "get"]
- apiGroups: [""]
  resources:
  - namespaces
  verbs: ["get", "list", "watch"]

This causes Voyager to throw tons of API errors, because it cannot query nodes, namespaces, etc.

@tamalsaha

This comment has been minimized.

Copy link
Member

commented Jun 26, 2018

I used the kubectl auth reconcile command, which is supposed to merge the 2 roles , not overwrite. https://github.com/appscode/voyager/blob/release-7.2/hack/deploy/voyager.sh#L401 . This is what I see on a fresh minikube cluster.

$ kubectl version --short
Client Version: v1.10.2
Server Version: v1.10.0

$ curl -fsSL https://raw.githubusercontent.com/appscode/voyager/7.2.0/hack/deploy/voyager.sh \
                                  | bash -s -- --provider=minikube
checking kubeconfig context
minikube

checking whether extended apiserver feature is enabled

VOYAGER_CLOUD_CONFIG=
VOYAGER_CLOUD_PROVIDER=minikube
VOYAGER_DOCKER_REGISTRY=appscode
VOYAGER_ENABLE_ANALYTICS=true
VOYAGER_ENABLE_RBAC=true
VOYAGER_ENABLE_VALIDATING_WEBHOOK=true
VOYAGER_HAPROXY_IMAGE_TAG=1.8.9-7.2.0-alpine
VOYAGER_IMAGE_PULL_POLICY=IfNotPresent
VOYAGER_IMAGE_PULL_SECRET=
VOYAGER_IMAGE_TAG=7.2.0
VOYAGER_INGRESS_CLASS=
VOYAGER_NAMESPACE=kube-system
VOYAGER_PURGE=0
VOYAGER_RESTRICT_TO_NAMESPACE=false
VOYAGER_ROLE_TYPE=ClusterRole
VOYAGER_RUN_ON_MASTER=0
VOYAGER_SERVICE_ACCOUNT=voyager-operator
VOYAGER_TEMPLATE_CONFIGMAP=
VOYAGER_UNINSTALL=0

Wrote ca certificates in  /home/tamal/go/src/github.com/kubedb/postgres
Wrote server certificates in  /home/tamal/go/src/github.com/kubedb/postgres
deployment.apps "voyager-operator" created
secret "voyager-apiserver-cert" created
service "voyager-operator" created
serviceaccount "voyager-operator" created
clusterrole.rbac.authorization.k8s.io "voyager-operator" reconciled
clusterrole.rbac.authorization.k8s.io "voyager-operator" reconciled
clusterrolebinding.rbac.authorization.k8s.io "voyager-operator" reconciled
clusterrolebinding.rbac.authorization.k8s.io "voyager-operator" reconciled
rolebinding.rbac.authorization.k8s.io "voyager-apiserver-extension-server-authentication-reader" reconciled
clusterrolebinding.rbac.authorization.k8s.io "voyager-apiserver-auth-delegator" reconciled
clusterrole.rbac.authorization.k8s.io "appscode:voyager:edit" reconciled
clusterrole.rbac.authorization.k8s.io "appscode:voyager:view" reconciled
apiservice.apiregistration.k8s.io "v1beta1.admission.voyager.appscode.com" created
validatingwebhookconfiguration.admissionregistration.k8s.io "admission.voyager.appscode.com" created

waiting until voyager operator deployment is ready
waiting until voyager apiservice is available
waiting until voyager crds are ready

Successfully installed Voyager in kube-system namespace!


$ kubectl get clusterrole | grep voyager
appscode:voyager:edit                                                  43s
appscode:voyager:view                                                  42s
voyager-operator                                                       43s

$ kubectl get clusterrole voyager-operator -o yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  creationTimestamp: 2018-06-26T19:14:01Z
  labels:
    app: voyager
  name: voyager-operator
  resourceVersion: "537"
  selfLink: /apis/rbac.authorization.k8s.io/v1/clusterroles/voyager-operator
  uid: 157d77be-7975-11e8-8eea-080027deeada
rules:
- apiGroups:
  - apiextensions.k8s.io
  resources:
  - customresourcedefinitions
  verbs:
  - '*'
- apiGroups:
  - voyager.appscode.com
  resources:
  - '*'
  verbs:
  - list
- apiGroups:
  - ""
  resources:
  - nodes
  verbs:
  - list
  - watch
  - get
- apiGroups:
  - ""
  resources:
  - namespaces
  verbs:
  - get
  - list
  - watch
- apiGroups:
  - voyager.appscode.com
  resources:
  - '*'
  verbs:
  - '*'
- apiGroups:
  - monitoring.coreos.com
  resources:
  - servicemonitors
  verbs:
  - '*'
- apiGroups:
  - apps
  resources:
  - deployments
  verbs:
  - '*'
- apiGroups:
  - extensions
  resources:
  - deployments
  verbs:
  - '*'
- apiGroups:
  - extensions
  resources:
  - daemonsets
  verbs:
  - '*'
- apiGroups:
  - extensions
  resources:
  - ingresses
  verbs:
  - '*'
- apiGroups:
  - ""
  resources:
  - replicationcontrollers
  verbs:
  - '*'
- apiGroups:
  - ""
  resources:
  - services
  verbs:
  - '*'
- apiGroups:
  - ""
  resources:
  - endpoints
  verbs:
  - '*'
- apiGroups:
  - ""
  resources:
  - configmaps
  verbs:
  - '*'
- apiGroups:
  - ""
  resources:
  - secrets
  verbs:
  - get
- apiGroups:
  - ""
  resources:
  - secrets
  verbs:
  - list
- apiGroups:
  - ""
  resources:
  - secrets
  verbs:
  - watch
- apiGroups:
  - ""
  resources:
  - secrets
  verbs:
  - create
- apiGroups:
  - ""
  resources:
  - secrets
  verbs:
  - update
- apiGroups:
  - ""
  resources:
  - secrets
  verbs:
  - patch
- apiGroups:
  - ""
  resources:
  - events
  verbs:
  - create
- apiGroups:
  - ""
  resources:
  - pods
  verbs:
  - list
- apiGroups:
  - ""
  resources:
  - pods
  verbs:
  - watch
- apiGroups:
  - ""
  resources:
  - pods
  verbs:
  - delete
- apiGroups:
  - ""
  resources:
  - pods
  verbs:
  - deletecollection
- apiGroups:
  - ""
  resources:
  - serviceaccounts
  verbs:
  - get
- apiGroups:
  - ""
  resources:
  - serviceaccounts
  verbs:
  - create
- apiGroups:
  - ""
  resources:
  - serviceaccounts
  verbs:
  - delete
- apiGroups:
  - ""
  resources:
  - serviceaccounts
  verbs:
  - patch
- apiGroups:
  - rbac.authorization.k8s.io
  resources:
  - rolebindings
  verbs:
  - get
- apiGroups:
  - rbac.authorization.k8s.io
  resources:
  - rolebindings
  verbs:
  - create
- apiGroups:
  - rbac.authorization.k8s.io
  resources:
  - rolebindings
  verbs:
  - delete
- apiGroups:
  - rbac.authorization.k8s.io
  resources:
  - rolebindings
  verbs:
  - patch
- apiGroups:
  - rbac.authorization.k8s.io
  resources:
  - roles
  verbs:
  - get
- apiGroups:
  - rbac.authorization.k8s.io
  resources:
  - roles
  verbs:
  - create
- apiGroups:
  - rbac.authorization.k8s.io
  resources:
  - roles
  verbs:
  - delete
- apiGroups:
  - rbac.authorization.k8s.io
  resources:
  - roles
  verbs:
  - patch

@dmegyesi

This comment has been minimized.

Copy link
Author

commented Jun 26, 2018

My bad! Yes indeed, I installed Voyager back in the early 5.0 times with some custom flags, so I couldn't use the shell script installer.
Now in order to upgrade to 7.2 and see the actual differences being done, I did the upgrade manually from the YAML files used in the automated script. I simply ran kubectl apply on the files, instead of reconcile. (Actually, didn't even know this exists! Thanks for pointing it out.)

I apologize for the false alarm.

@dmegyesi dmegyesi closed this Jun 26, 2018

@tamalsaha

This comment has been minimized.

Copy link
Member

commented Jun 26, 2018

Thanks for using Voyager !

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
2 participants
You can’t perform that action at this time.