RBAC on GKE - extra step needed

[gytisgreitai]

What did you do?

hack/cluster-monitoring/deploy

What did you expect to see?

No errors

What did you see instead? Under which circumstances?

namespace "monitoring" created
clusterrolebinding "prometheus-operator" configured
serviceaccount "prometheus-operator" created
deployment "prometheus-operator" created
Error from server (Forbidden): error when creating "manifests/prometheus-operator/prometheus-operator-cluster-role.yaml": clusterroles.rbac.authorization.k8s.io "prometheus-operator" is forbidden: attempt to grant extra privileges: [{[create] [extensions] [thirdpartyresources] [] []} {[*] [monitoring.coreos.com] [alertmanagers] [] []} {[*] [monitoring.coreos.com] [prometheuses] [] []} {[*] [monitoring.coreos.com] [servicemonitors] [] []} {[*] [apps] [statefulsets] [] []} {[*] [] [configmaps] [] []} {[*] [] [secrets] [] []} {[list] [] [pods] [] []} {[delete] [] [pods] [] []} {[get] [] [services] [] []} {[create] [] [services] [] []} {[update] [] [services] [] []} {[get] [] [endpoints] [] []} {[create] [] [endpoints] [] []} {[update] [] [endpoints] [] []} {[list] [] [nodes] [] []} {[watch] [] [nodes] [] []}] user=&{gytis@tripcreator.com  [system:authenticated] map[]} ownerrules=[{[create] [authorization.k8s.io] [selfsubjectaccessreviews] [] []} {[get] [] [] [] [/api /api/* /apis /apis/* /healthz /swaggerapi /swaggerapi/* /version]}] ruleResolutionErrors=[]
Waiting for Operator to register third party objects...done!
daemonset "node-exporter" created
service "node-exporter" created
clusterrolebinding "kube-state-metrics" configured
deployment "kube-state-metrics" created
serviceaccount "kube-state-metrics" created
service "kube-state-metrics" created
Error from server (Forbidden): error when creating "manifests/kube-state-metrics/kube-state-metrics-cluster-role.yaml": clusterroles.rbac.authorization.k8s.io "kube-state-metrics" is forbidden: attempt to grant extra privileges: [{[list] [] [nodes] [] []} {[watch] [] [nodes] [] []} {[list] [] [pods] [] []} {[watch] [] [pods] [] []} {[list] [] [resourcequotas] [] []} {[watch] [] [resourcequotas] [] []} {[list] [extensions] [daemonsets] [] []} {[watch] [extensions] [daemonsets] [] []} {[list] [extensions] [deployments] [] []} {[watch] [extensions] [deployments] [] []} {[list] [extensions] [replicasets] [] []} {[watch] [extensions] [replicasets] [] []}] user=&{gytis@tripcreator.com  [system:authenticated] map[]} ownerrules=[{[create] [authorization.k8s.io] [selfsubjectaccessreviews] [] []} {[get] [] [] [] [/api /api/* /apis /apis/* /healthz /swaggerapi /swaggerapi/* /version]}] ruleResolutionErrors=[]
secret "grafana-credentials" created
secret "grafana-credentials" configured
configmap "grafana-dashboards" created
deployment "grafana" created
service "grafana" created
clusterrolebinding "prometheus" configured
configmap "prometheus-k8s-rules" created
serviceaccount "prometheus-k8s" created
servicemonitor "alertmanager" configured
servicemonitor "kube-apiserver" configured
servicemonitor "k8s-apps-http" configured
servicemonitor "kube-state-metrics" configured
servicemonitor "kubelet" configured
servicemonitor "node-exporter" configured
servicemonitor "prometheus" configured
service "prometheus-k8s" created
prometheus "k8s" configured
Error from server (Forbidden): error when creating "manifests/prometheus/prometheus-cluster-role.yaml": clusterroles.rbac.authorization.k8s.io "prometheus" is forbidden: attempt to grant extra privileges: [{[get] [] [nodes] [] []} {[list] [] [nodes] [] []} {[watch] [] [nodes] [] []} {[get] [] [services] [] []} {[list] [] [services] [] []} {[watch] [] [services] [] []} {[get] [] [endpoints] [] []} {[list] [] [endpoints] [] []} {[watch] [] [endpoints] [] []} {[get] [] [pods] [] []} {[list] [] [pods] [] []} {[watch] [] [pods] [] []} {[get] [] [configmaps] [] []} {[get] [] [] [] [/metrics]}] user=&{gytis@tripcreator.com  [system:authenticated] map[]} ownerrules=[{[create] [authorization.k8s.io] [selfsubjectaccessreviews] [] []} {[get] [] [] [] [/api /api/* /apis /apis/* /healthz /swaggerapi /swaggerapi/* /version]}] ruleResolutionErrors=[]
secret "alertmanager-main" created
service "alertmanager-main" created
alertmanager "main" configured

Environment

• Kubernetes version information:

Client Version: version.Info{Major:"1", Minor:"6", GitVersion:"v1.6.2", GitCommit:"477efc3cbe6a7effca06bd1452fa356e2201e1ee", GitTreeState:"clean", BuildDate:"2017-04-19T20:33:11Z", GoVersion:"go1.7.5", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"6", GitVersion:"v1.6.2", GitCommit:"477efc3cbe6a7effca06bd1452fa356e2201e1ee", GitTreeState:"clean", BuildDate:"2017-04-19T20:22:08Z", GoVersion:"go1.7.5", Compiler:"gc", Platform:"linux/amd64"}

• Kubernetes cluster kind:

  GKE

According to Google Container Engine docs:

Because of the way Container Engine checks permissions when you create a Role or ClusterRole, you must first create a RoleBinding that grants you all of the permissions included in the role you want to create.

An example workaround is to create a RoleBinding that gives your Google identity a cluster-admin role before attempting to create additional Role or ClusterRole permissions.

This is a known issue in the Beta release of Role-Based Access Control in Kubernetes and Container Engine version 1.6.

So in order to proceed without error, cluster-admin role should be added to current executing user, eg:

kubectl create clusterrolebinding your-user-cluster-admin-binding --clusterrole=cluster-admin --user=your.google.cloud.email@example.org

Could this be added as a faq/hint etc. somewhere?

Thank you!

[brancz]

Yes I think it would definitely be worth starting an FAQ/Troubleshooting document in the Documentation folder. Do you want to do a PR for this @gytisgreitai ?

[gytisgreitai]

sure :)

[gytisgreitai]

Created #360

[brancz]

Closing here as #360 is merged. Thanks a lot @gytisgreitai !

[jason-riddle]

So I followed the instructions in #360, but I'm still stuck with "forbidden" access. Is there anything else need to be done? Do the credentials need to be recreated or do any other commands need to be ran?

[gytisgreitai]

Should be fine just with that command.

• What Kubernetes version are you using?
• Do you see your clusterrolebinding when doing kubectl get clusterrolebinding ?
• What does kubectl get clusterrolebinding [yourclusterindingname] -o json output?

As a quick hack you can do kubectl proxy and apply those failed yamls via create

[jason-riddle]

• kubectl version

$ kubectl version --short
Client Version: v1.6.0
Server Version: v1.6.2

• I do see my clusterrolebinding.

$ kubectl get clusterrolebinding | grep myname-cluster-admin-binding
myname-cluster-admin-binding                   17h

• kubectl get clusterrolebinding myname-cluster-admin-binding -o json

{
    "apiVersion": "rbac.authorization.k8s.io/v1beta1",
    "kind": "ClusterRoleBinding",
    "metadata": {
        "creationTimestamp": "2017-05-14T23:09:25Z",
        "name": "myname-cluster-admin-binding",
        "resourceVersion": "188788",
        "selfLink": "/apis/rbac.authorization.k8s.io/v1beta1/clusterrolebindingsmyname-cluster-admin-binding",
        "uid": "<snipped>"
    },
    "roleRef": {
        "apiGroup": "rbac.authorization.k8s.io",
        "kind": "ClusterRole",
        "name": "cluster-admin"
    },
    "subjects": [
        {
            "apiGroup": "rbac.authorization.k8s.io",
            "kind": "User",
            "name": "<my-gmail-username>@gmail.com"
        }
    ]
}

I literally used the name myname-cluster-admin-binding, but that shouldn't be an issue correct?

[gytisgreitai]

Well I've just tested this on 1.6.2 and it seems to work fine for me.
And gcloud info | grep Account outputs <my-gmail-username>@gmail.com yes?

What access rights does this email have on iAM ?

[jason-riddle]

• Yes, gcloud info | grep Account outputs my email.
• This user is an Owner on the project.

I'll let this issue die out since it only seems to be my problem, but I'll report back if I find a fix.

[yuvipanda]

I ran into this, and turns out that even though gcloud info | grep Account tells me my username is yuvipanda@gmail.com, my username really is YuviPanda@gmail.com! The case matters, and I had to fix the case to make it work.

I could determine this by looking at the apiserver logs, which I could do on GKE with:

kubectl proxy
curl -s http://localhost:8001/logs/kube-apiserver.log > apiserverlogs

[ngtuna]

Yes, gcloud info | grep Account outputs my email.

That's not always true. For example, the case of @yuvipanda showed that the username that GKE recognized was capitalized at some characters. So my two questions:

1. why does capitalization happen ?
2. How do we retrieve that modified/capitalized/real username by code ? I am looking for support from client-go for example...