Jenkins deployment with Kubernetes plugin does not have cluster access using RBAC

[carlosedp]

I've deployed the Jenkins chart that comes bundled with the Jenkins Kubernetes plugin but the plugin needs cluster access to create it's test pods.

Since the chart is deployed with the default serviceaccount, I need to grant full access to all serviceaccounts to the cluster with the command:

kubectl create clusterrolebinding permissive-binding --clusterrole=cluster-admin --user=admin --user=kubelet --group=system:serviceaccounts

It would be better if Jenkins could be deployed with a specific account and grant the cluster-admin role just to it.

[ptagr]

Is this required for the kubernetes plugin to spin up new slaves? I think I just ran into the same issue.

io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: POST at: https://kubernetes.default/api/v1/namespaces/npd-test/pods. Message: Forbidden!Configured service account doesn't have access. Service account may have been revoked..
	at io.fabric8.kubernetes.client.dsl.base.OperationSupport.requestFailure(OperationSupport.java:315)
	at io.fabric8.kubernetes.client.dsl.base.OperationSupport.assertResponseCode(OperationSupport.java:266)
	at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:237)
	at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:230)
	at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleCreate(OperationSupport.java:208)
	at io.fabric8.kubernetes.client.dsl.base.BaseOperation.handleCreate(BaseOperation.java:643)
	at io.fabric8.kubernetes.client.dsl.base.BaseOperation.create(BaseOperation.java:300)
	at org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud$ProvisioningCallback.call(KubernetesCloud.java:636)
	at org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud$ProvisioningCallback.call(KubernetesCloud.java:581)
	at jenkins.util.ContextResettingExecutorService$2.call(ContextResettingExecutorService.java:46)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)```

[ptagr]

Which service account is the plugin using? Is it possible to specify a different service account to use for the kubernetes plugin? It does not seem to give that option when I add a Kubernetes Service Account credentials.

[lachie83]

You can create a service account like this with rbac enabled. I don't think it needs cluster-admin but I haven't narrowed it down.

https://gist.github.com/lachie83/17c1fff4eb58cf75c5fb11a4957a64d2

Then go configure k8s service account credentials in Jenkins and update the credentials config under the cloud section.

[carlosedp]

I confirm this works. Created the SA manually and assigned it to the Jenkins configuration. Will this be integrated to the chart?

[lachie83]

I feel that service account provisioning and configuration is outside the scope of a chart. You would also need to automate the create of the credentials inside Jenkins which we're currently not doing. I think the best step forward is to document the setup steps in the readme.

[lachie83]

I think this is reasonable and can be made conditional. See linked PR

[lachie83]

@carlosedp Would this be helpful?

[carlosedp]

Yes, absolutely @lachie83. Thanks!

[ReSearchITEng]

Hello @carlosedp @lachie83 !
While trying this, I noticed that the kubernetes-plugin is not getting configured to use the newly created service-account.
More than this, I could not find a way to point the kubernetes-plugin to a specific k8s service-account.
It seems that it's always looking to the default service account for that namespace:

Error connecting to [edited]: Failure executing: GET at: [edited]/api/v1/namespaces/jenkins/pods. Message: Forbidden!Configured service account doesn't have access. Service account may have been revoked. User "system:serviceaccount:jenkins:default" cannot list pods in the namespace "jenkins".. 

while the helm chart creates a service account like: jenkins-jenkins

[campbelldgunn]

I just tried the fix that @lachie83 mentions in Gist. I then created the kubernetes service account as jenkins. When I go and update the configuration and test it, I get the following error:

Error connecting to https://kubernetes.default: Failure executing: GET at: https://kubernetes.default/api/v1/namespaces/default/pods. Message: Forbidden!Configured service account doesn't have access. Service account may have been revoked. pods is forbidden: User "system:serviceaccount:default:default" cannot list pods in the namespace "default".

Any ideas, what I could to do more to find out what is going on?

[jchrisweaver]

How do you configure the plugin to use a specific Service Account that's been created for jenkins?

chris@Murdock:/Users/chris/code/victory/pacecar> k get serviceaccounts/cd-jenkins
NAME         SECRETS   AGE
cd-jenkins   1         33m
chris@Murdock:/Users/chris/code/victory/pacecar> k describe serviceaccounts/cd-jenkins
Name:                cd-jenkins
Namespace:           default
Labels:              app=cd-jenkins
                     chart=jenkins-0.16.1
                     heritage=Tiller
                     release=cd
Annotations:         <none>
Image pull secrets:  <none>
Mountable secrets:   cd-jenkins-token-wc2sj
Tokens:              cd-jenkins-token-wc2sj
Events:              <none>

root@default-dzspv:~# kubectl auth can-i get pods
no - Unknown user "system:serviceaccount:default:default"

[Al-Cheen]

I followed the steps here: https://www.blazemeter.com/blog/how-to-setup-scalable-jenkins-on-top-of-a-kubernetes-cluster

However when setting up the connection to the Kubernetes URL in the Cloud configuration I would receive the error mentioned previously:

Error connecting to https://kubernetes.default: Failure executing: GET at: https://kubernetes.default/api/v1/namespaces/default/pods. Message: Forbidden!Configured service account doesn't have access. Service account may have been revoked. pods is forbidden: User "system:serviceaccount:default:default" cannot list pods in the namespace "default".

I was able to run the command listed by @carlosedp and I was able to connect successfully:

kubectl create clusterrolebinding permissive-binding --clusterrole=cluster-admin --user=admin --user=kubelet --group=system:serviceaccounts

[vinixaavier]

The commands below will allow permission for all service accounts of cluster:
kubectl create clusterrolebinding permissive-binding --clusterrole=cluster-admin --user=admin --user=kubelet --group=system:serviceaccounts

The official documentation of Kubernetes, not approve this practice:
Warning: The following policy allows ALL service accounts to act as cluster administrators. Any application running in a container receives service account credentials automatically, and could perform any action against the API, including viewing secrets and modifying permissions. This is not a recommended policy.

Use this permissive binding for only namespace which the jenkins was deployed: i.e jenkins, kubernetes-plugin

example:
kubectl create clusterrolebinding permissive-binding --clusterrole=cluster-admin --user=admin --user=kubelet --group=system:serviceaccounts:jenkins
kubectl create clusterrolebinding permissive-binding --clusterrole=cluster-admin --user=admin --user=kubelet --group=system:serviceaccounts:<custom-namespace>

Therefore, the commands above will allow only permissions to serviceaccounts inside of the namespace which you specified.