New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

system:default not sufficient for helm, but works for kubectl #2687

Closed
jayunit100 opened this Issue Jul 14, 2017 · 22 comments

Comments

Projects
None yet
@jayunit100

jayunit100 commented Jul 14, 2017

PROBLEM

Helm reports lack of ACL's as the reason that it can't do list configmaps/pods/etc... However, the kubeconfig which it uses is valid. This is confusing b/c it mixes Tiller failures w/ client failures, and its not clear wether the helm client is breaking, or wether its that the Tiller is breaking.

i.e. the tiller logs show essentially the same thing as the client failure.

SOLUTION

Make the helm client more defensive.

  • Helm client operations should fail fast, and explain that they are failing b/c tiller isn't working.
  • Helm client failures should always report wether its a downstream error (tiller) or wether its an error in the client.
  • Ideally I'd also like to know why my tiller is failing, but hopefully I can figure that out on my own :).

DETAILS

I have a linux machine where the admin is capable of listing pods and configmaps and so on:

ubuntu@ip-10-0-22-242:/opt$ kubectl  get configmaps
NAME             DATA      AGE
hub-config       6         2d
special-config   3         2d

using a standard kubectl client.... However, with helm, I get an error:

ubuntu@ip-10-0-22-242:/opt$ export KUBECONFIG=/home/ubuntu/kubeconfig
ubuntu@ip-10-0-22-242:/opt$ helm-exec list
Error: User "system:serviceaccount:kube-system:default" cannot list configmaps in the namespace "kube-system". (get configmaps)

Helm version detailsl

ubuntu@ip-10-0-22-242:/opt$ helm-exec version
Client: &version.Version{SemVer:"v2.5.0", GitCommit:"012cb0ac1a1b2f888144ef5a67b8dab6c2d45be6", GitTreeState:"clean"}
Server: &version.Version{SemVer:"v2.5.0", GitCommit:"012cb0ac1a1b2f888144ef5a67b8dab6c2d45be6", GitTreeState:"clean"}

Helm user seems correct (admin@kubernetes) but when looking deeper into tiller:

ubuntu@ip-10-0-22-242:/opt$ kubectl logs tiller-deploy-3703072393-pn7f6 --namespace=kube-system
[main] 2017/07/13 22:50:37 Starting Tiller v2.5.0 (tls=false)
[main] 2017/07/13 22:50:37 GRPC listening on :44134
[main] 2017/07/13 22:50:37 Probes listening on :44135
[main] 2017/07/13 22:50:37 Storage driver is ConfigMap
[storage] 2017/07/14 15:02:51 listing all releases with filter
[storage/driver] 2017/07/14 15:02:51 list: failed to list: User "system:serviceaccount:kube-system:default" cannot list configmaps in the namespace "kube-system". (get configmaps)
[storage] 2017/07/14 15:03:18 listing all releases with filter
[storage/driver] 2017/07/14 15:03:18 list: failed to list: User "system:serviceaccount:kube-system:default" cannot list configmaps in the namespace "kube-system". (get configmaps)
[storage] 2017/07/14 15:06:36 listing all releases with filter
[storage/driver] 2017/07/14 15:06:36 list: failed to list: User "system:serviceaccount:kube-system:default" cannot list configmaps in the namespace "kube-system". (get configmaps)
[storage] 2017/07/14 15:07:25 listing all releases with filter
[storage/driver] 2017/07/14 15:07:25 list: failed to list: User "system:serviceaccount:kube-system:default" cannot list configmaps in the namespace "kube-system". (get configmaps)
[storage] 2017/07/14 15:07:54 listing all releases with filter
[storage/driver] 2017/07/14 15:07:54 list: failed to list: User "system:serviceaccount:kube-system:default" cannot list configmaps in the namespace "kube-system". (get configmaps)
[storage] 2017/07/14 15:11:57 listing all releases with filter
[storage/driver] 2017/07/14 15:11:57 list: failed to list: User "system:serviceaccount:kube-system:default" cannot list configmaps in the namespace "kube-system". (get configmaps)
@jayunit100

This comment has been minimized.

jayunit100 commented Jul 14, 2017

I fixed my issue - it was related to running things in kube-system. I don't know why my helm client (using same kubeconfig as my kubectl) couldn't access that NS, but I'm thinking we should print a warning when installing helm w/o a custom namespace.

@shraderdm

This comment has been minimized.

shraderdm commented Jul 16, 2017

@jayunit100 can you please explain how you got around this issue exactly? by deploying tiller-deploy in a different namespace? I'm using an OpenShift cluster and it deployed tiller-deploy in the kube-system namespace automatically.

@jayunit100

This comment has been minimized.

jayunit100 commented Jul 17, 2017

@shraderdm

  • http://jayunit100.blogspot.com/2017/07/helm-on.html I wrote up my notes here. tiller-namespace= can switch the namespace for you, but likely,

  • if you're seeing failed to list erros, then the real issue you are hitting is that, by default, your cluster doesn't add a service account to pods which allows the tiller pods to list stuff.

  • so the real trick is that you have to make sure there is a corresponding service account, which grants privileges that helm needs i.e.

roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
  • One final note: The above svc account might be a superset of whats actually needed for helm, but it works, and was shared w/ me by the heptio folks, as its necessary on some clusters, depending on the default service accounts that pods get launched with /
@thomastaylor312

This comment has been minimized.

Collaborator

thomastaylor312 commented Jul 21, 2017

@jayunit100 Sorry for the delay in getting to this issue. Would you consider your initial proposal still valid after solving the issue? It looks really good and I'd love to keep it open for discussion so we can implement it down the road

@shraderdm

This comment has been minimized.

shraderdm commented Jul 24, 2017

@jayunit100 nice blog post! I had solved it similarly, but there were a few other issues I had to solve in OpenShift as well. It basically came down to the same issue though, rbac blocking my access, and needing to add cluster-admin to the service account that deploys the tiller. I also deployed it in a different namespace and had to remove some nodeselectors, etc. (although I was already past that point when I posted my question here). I hope your proposal goes through though, to allow others to more easily solve this issue in the future. :)

@jayunit100

This comment has been minimized.

jayunit100 commented Jul 24, 2017

Yeah definetely I hope so to. I'm happy to help implement as well ; might need some pointers on how to get ramped up on helm dev/testing . I'll check the docs first

@thomastaylor312

This comment has been minimized.

Collaborator

thomastaylor312 commented Jul 25, 2017

Sounds good @jayunit100. Let us know if you need any help.

@leejian0612

This comment has been minimized.

leejian0612 commented Oct 27, 2017

Per jayunit100 's advice, add role cluster-admin to serviceaccount kube-system:default works for me, below is the cmd.

kubectl create clusterrolebinding add-on-cluster-admin --clusterrole=cluster-admin --serviceaccount=kube-system:default
@seh

This comment has been minimized.

Collaborator

seh commented Oct 27, 2017

Why are you running Tiller with the "default" service account?

@bacongobbler

This comment has been minimized.

Member

bacongobbler commented Nov 1, 2017

this sounds related and will likely be closed via the proposed documentation in #2962.

@christhomas

This comment has been minimized.

christhomas commented Jan 8, 2018

@seh I am running helm with the defaults, as the instructions said to do a "helm init" and what I got is this situation, it installs tiller-deploy-xxxx into the kube-system namespace and I'm assuming that by default it tries to use the default service account too.

@bacongobbler

This comment has been minimized.

Member

bacongobbler commented Jan 8, 2018

I believe we can close this now that #3094 was merged and is available to view at https://docs.helm.sh/using_helm/#role-based-access-control. I'm going to close this but please let us know if there's something else we missed. Thanks!

@christhomas

This comment has been minimized.

christhomas commented Jan 8, 2018

Shouldn't the initialisation of helm automatically setup this account, since it cannot operate without it, it appears strange that I also need to setup the service account also, since its a prerequisite for it to function correctly?

@bacongobbler

This comment has been minimized.

Member

bacongobbler commented Jan 8, 2018

The default service account should already be there by... puts on shades... default. 😎

Is this not the case? I haven't heard of a case before where the default service account does not exist in kube-system.

For background information, Helm (and therefore tiller) have no idea what a service account is. They just happen to need one to talk to the kubernetes API server. There are multiple different use cases in RBAC environments, some of which can be seen in the docs. There are also other authentication policies such as Attribute-based access control (ABAC) that do not tie into RBAC roles at all. With that in mind, we decided not to bake RBAC support directly into helm init and pigeon-hole users into one scenario.

@christhomas

This comment has been minimized.

christhomas commented Jan 8, 2018

ok, so it needs to be part of the installation instructions, at the same location where helm init is described, since this actually is a make or break issue for the installation and not an advanced option, its literally the default thing most of us will have to do if we are running on kubernetes 1.8 with a default setup.

and its not described anywhere, except a separate page, which leads me to believe its a side issue, or an advanced topic, but this is really critical to getting it working in kubernetes default RBAC mode

@fossxplorer

This comment has been minimized.

fossxplorer commented Apr 13, 2018

@leejian0612 Thanks a lot, i was stuck for ~ 3 days with the issue #2464 (comment) and after solving that i got into this issue which was solved with your suggestion.
Also see #2464 (comment)
@chrisglass totally agree with you. It should be part of Helm doc, for instance a FAQ section.

@bacongobbler

This comment has been minimized.

Member

bacongobbler commented Apr 13, 2018

I agree. If someone wants to make a PR and cc myself I'd happily review it.

@shults

This comment has been minimized.

shults commented Aug 1, 2018

kubectl -n kube-system edit deploy/tiller-deploy

# add these two lines in container (tiller) scope
serviceAccount: admin-user
serviceAccountName: admin-user

This solution is more secure than creating clusterrolebinding with next params (--clusterrole=cluster-admin --serviceaccount=kube-system:default).

@falcocoris

This comment has been minimized.

falcocoris commented Aug 8, 2018

Hi

@shults
I've tried this but i'm still getting this error :
Error: configmaps is forbidden: User "system:serviceaccount:kube-system:default" cannot list configmaps in the namespace "kube-system"
Is there something i've missed ?

@shults

This comment has been minimized.

shults commented Aug 8, 2018

@falcocoris

  1. run helm init
  2. kubectl -n kube-system edit deploy/tiller-deploy and upper two lines
  3. start observe tiller pods kubectl get pods -w | grep tiller; they should not fail.
@aganesan94

This comment has been minimized.

aganesan94 commented Aug 8, 2018

No luck yet.! with the above ones

@falcocoris

This comment has been minimized.

falcocoris commented Aug 8, 2018

using these lines fixed my pb :

# kubectl create clusterrolebinding add-on-cluster-admin \
  --clusterrole=cluster-admin \
  --serviceaccount=kube-system:default
clusterrolebinding "add-on-cluster-admin" created
# helm reset     

Error: error unstalling Tiller: timed out waiting for the condition

Note the error at the end, i'm not sure this command did anything in the end but it worked with all of these.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment