Allow clusterwide actions within apb

[karmab]

Feature:

as one can be done with say helm, it would be useful to be able to create namespaces within the apb or define cluster role bindings .
this is useful for example for controllers trying to gather custom resource definitions cluster wide

What happened:
gets a permission denied

What you expected to happen:
sample code to succeed

How to reproduce it:
use the following sample code in an apb

- name: Create crd Role Binding
  k8s_v1beta1_cluster_role_binding:
    name: fission-crd
    role_ref_kind: ClusterRole
    role_ref_name: cluster-admin
    subjects:
    - kind: ServiceAccount
      name: fission-svc
      namespace: "{{ namespace }}"

[rthallisey]

IRC Discussion: https://paste.fedoraproject.org/paste/-bnngTfuX3Kr6qgKQ11emg

[LorbusChris]

I'm all for finding a way to enable APBs that are specifically targeted at cluster admins.

I ran into this as well with my shot at an acme-controller-apb (there was also another problem with the asb modules, so I didn't actually encouter this denial)

Could deploying a dedicated ASB for these cluster-wide APBs be a solution? And then find a way to have this broker's APBs only be useable for/shown in the Service Catalog of privileged users?

[karmab]

maybe we can address it in a simpler way by optionally enabling the use of a dedicated namespace for the apb run using default sa there ?
that way we could just set the permissions from this specific namespace + sa and let the end user choose

[LorbusChris]

Looking at asb-installer, I kinda like the idea of using the APB format just to package privileged applications and install without actually using the broker.

[rthallisey]

Looking at asb-installer, I kinda like the idea of using the APB format just to package privileged applications and install without actually using the broker.

That's definitely a possibility.

Here's what I was thinking for this issue:

• The apb can send a message to the admin/broker saying that is requires a cluster-role with permissions to access namespace X, Y, and Z. The admin can audit the apb, and decide whether to provide a cluster role or not.

[karmab]

for testing purposes, cant we just have the apb running in a specific namespace with a specific (or default) sa ( i was thinking of the openshift-ansible-broker ns actually)

[rthallisey]

An example 'tech-preview' way to give an apb cluster-level access: https://github.com/ansibleplaybookbundle/kubevirt-apb/blob/master/roles/kubevirt-apb/tasks/provision.yml#L3-L13.

To summarize, the user pass login credentials to the apb. The apb will sign into a user that has cluster-admin permissions, which will allow your apb to have full access to cluster-resources.

[jmrodri]

The pastebin link from #576 (comment) is no longer valid. Looking through my IRC logs from Dec 5, 2017 I think this was the conversation that was intended. Adding it here for posterity.

--- Day changed Tue Dec 05 2017
05:32 < karimb> hello folks
05:32 < karimb> is there a way to enable deploying namespaces/projects from an apb ?
08:40 < karimb> hello guys
08:40 < karimb> how can i get rid of the following error:
08:40 < karimb> References a non-existent ClusterServicePlan (K8S: "" ExternalName: "default")
08:44 < rhallisey> karimb, could be there are no plans in the catalog
08:45 < karimb> oc get clusterserviceplans -o yaml | grep -i external
08:45 < karimb> i tried that and default did show up
08:45 < karimb> rhallisey *
08:45 < rhallisey> but the plan might not be associated with the app you wanted
08:46 < karimb> right
08:47 < karimb> how can i check that ?
08:47 < karimb> rhallisey this is my apb https://pastebin.com/TqExvvKG
08:48 < karimb> the full error message goes like this: fission2   ServiceInstance             Warning   ReferencesNonexistentServicePlan   service-catalog-controller-manager   References a non-existent ClusterServicePlan (K8S: "" ExternalName: "default") on ClusterServiceClass (K8S: "05c1be5dd8e2aefbe37e5afb80cb9b94" ExternalName: "apb-push-fission-apb") or there is more than one (found: 0)
08:52 < rhallisey> karimb, we need to look through the plans. oc get clusterserviceplan -o yaml | grep 05c1be5dd8e2aefbe37e5afb80cb9b94
08:53 < karimb> nope that doesnt get me anything indeed
08:53 < rhallisey> just to double check
08:53 < rhallisey> oc get clusterserviceclass | grep 05c1be5dd8e2aefbe37e5afb80cb9b94
08:54 < rhallisey> want to make sure that is the clusterserviceclass externalid
08:54 < karimb> yes this one does show up
08:54 < karimb> so no  i am clueless :)
08:55 < rhallisey> let's do an oc get clusterserviceplan -o yaml | grep fission-apb
08:55 < rhallisey> let's do an oc get clusterserviceplan -o yaml | grep fission
08:55 < karimb> indeed fission isnt there
08:56 < rhallisey> how did you load your apbs into the broker?
08:56 < rhallisey> did you point at a registry? Or did you use the apb tool
08:56 < karimb> with apb push
08:56 < karimb> they do show up in apb list
08:58 < karimb> dev_broker is set to true in the configmap
08:59 < rhallisey> karimb, the service-catalog logs might have some more info.  Can you paste the logs from the controller-manager?
09:00 < rhallisey> I'll read through those and see if I spot anything
09:01 < karimb> rhallisey https://paste.fedoraproject.org/paste/8zSmuRvO2PVhTnQO6teOXw
09:01 < karimb> let me know if you want something... shorter
09:02 < karimb> https://paste.fedoraproject.org/paste/fE~8juy-r4Eljn-mtP9A2g this one was filtered with grep fission
09:06 < rhallisey> karimb, thanks
09:07 < rhallisey> karimb, can you also post the broker logs?
09:07 < karimb> sure
09:08 < karimb> rhallisey https://paste.fedoraproject.org/paste/XCQZWX~ZOWoAhwszymtkTg
09:08 < rhallisey> thank you
09:11 < rhallisey> karimb, nothing sticks out to me at the moment. Everything looks like it was processed just fine. I'll try to reproduce locally
09:11 < rhallisey> karimb, see if you can produce the same thing a second time
09:12 < karimb> yeah, it s happening each time
09:12 < karimb> i m testing on  a brand new asb instance rhallisey
09:12 < rhallisey> ok
09:14 < rhallisey> karimb, another thing you can try, is pointing your broker at the registry with your fission apb container and see if you get the same result
09:14 < karimb> that s a bit cumbersome, i believe
09:15 < karimb> or can i declare several sources in the configmap rhallisey ?
09:16 < rhallisey> karimb, ya you can do that too
09:18 < karimb> rhallisey something like this https://pastebin.com/3zrKbthj ?
09:18 < rhallisey> karimb, ya that looks good
09:59 < rhallisey> karimb, I'm able to run provision on fission when I made it available with apb push
10:00 < karimb> you mean, using my apb.yml ?
10:00 < karimb> now that s weird...
10:16 < rhallisey> karimb, I cloned the fission repo. Did an apb push.  Both serviceclass and serviceplan showed up in the calot
10:16 < rhallisey> catalog*
10:16 < karimb> which repo? mine you mean ?
10:16 < rhallisey> ya
10:17 < rhallisey> I'm using the latest apb tool that I built locally
10:17 < rhallisey> the provision didn't work because the pod couldn't find the local docker image, but the catalog entries did show up
10:26 < karimb> ok, so it s a matter of my environment
10:26 < karimb> i ve written the roles from scratch this morning
10:26 < karimb> so let s see if they actually work :=)
10:26 < karimb> on a brand new install
10:27 < rhallisey> it might be
11:21 < karimb> rhallisey indeed, it s working now....
11:21 < karimb> sorry for the disturbance .(
11:21 < rhallisey> karimb, nice!
11:21 < rhallisey> no worries
11:26 < karimb> rhallisey you commented someday that namespace creation was forbidden by default right ?
11:26 < karimb> is it like static stuff or can be adjusted in the configmap
11:27 < karimb> by setting elevate: auto or something like this
11:39 < karimb> also is there a way to delete serviceinstances manually ( that is, when the deprovisioning role has errored)
11:52 < rhallisey> karimb, the broker doesn't create it
11:52 < rhallisey> karimb, you can delete them manually with oc delete serviceinstance <xxxx>.  But, sometimes it gets stuck
11:53 < rhallisey> when the deprovision errors you're stuck. The catalog doesn't clean up the instnace
11:54 < karimb> rhallisey yes indeed
11:54 < karimb> the broker doesnt create it, the namespaces you say ?
11:54 < karimb> no way i can have them created ?
11:55 < rhallisey> karimb, iirc we got rid of that. You can create it before hand
11:55 < karimb> rhallisey i dont get the rationale behind it
11:55 < rhallisey> shurley, we don't create a namespace for an apb anymore right?
11:56 < shurley> rhallisey: no we expect that the target namespace has been created, you should see an error in the broker log
11:56 < rhallisey> karimb, I think it was a security choice
11:57 < rhallisey> we don't want the broker to be touching someones project
11:57 < karimb> yeah but i think it makes some products deployment too complex
11:58 < karimb> plus you have to create things before hand
11:58  * karimb believes it should be configurable
11:58 < rhallisey> karimb, you can create the project in the apb
11:58 < karimb> and defaulting to forbidden for security issues
11:58 < karimb> rhallisey then i dont get it
11:58 < karimb> ah ok
11:58 < karimb> you mean the namespace variable
11:58 < karimb> this one is fine
11:59 < karimb> but i can create additional namespaces within my apb code ?
11:59 < rhallisey> oh I didn't get you original q
12:00 < rhallisey> karimb, so you're getting an error when you create it?
12:00 < karimb> no, it s just something i had in mind
12:01 < karimb> but let me test it then
12:01 < rhallisey> I think you can do it
12:20 < karimb> rhallisey actually it s failing with Failed to retrieve requested object: User \"system:serviceaccount:apb-push-fission-apb-prov-z4cs9:apb-37f6115d-0bb1-40a6-ba51-d24e87d07141\" cannot get namespaces in the namespace \"fission-function\": User \"system:serviceaccount:apb-push-fission-apb-prov-z4cs9:apb-37f6115d-0bb1-40a6-ba51-d24e87d07141\" cannot get namespaces in project \"fission-function\
12:37 < karimb> what are the possible values for sandbox_role ?
12:37 < karimb> i ve set it to admin but cluster admin tasks are failing
12:57 < karimb> anyone :)
12:57 < karimb> ?
13:08 < rhallisey> karimb, you can set it to things in oc get clusterrole
13:08 < rhallisey> so it can be cluster-admin, admin, edit , ect...
13:11 < karimb> yeah tried cluster-admin actually
13:11 < karimb> but it doesnt work rhallisey
13:11 < karimb> fails early in provisioning
13:11 < rhallisey> karimb, same permission error?
13:11 < karimb> no
13:12 < karimb> it will not spawn the apb  container with this role
13:13 < rhallisey> karimb, so the broker is blocking you from spawning it then? Set auto_escalate to true
13:13 < rhallisey> in the broker config
13:13 < karimb> rhallisey let see
13:18 < karimb> rhallisey doesnt work, this is the traceback from asb pod https://pastebin.com/aw47VH6q
13:19 < shurley> karimb: rhallisey I don't think that auto_escalate will change anything here. I think that the cluster-admin role is only granted to certain namespaces, namespace A and namespace B I wonder if you can not create a namespace unless you have been granted access to do that
13:19 < karimb> rhallisey and the broker-config config map currently in usehttps://pastebin.com/RWWc7scB
13:20 < karimb> shurley doesnt that mean that you cant create namespaces ?
13:20 < karimb> actually i can create namespaces
13:20 < karimb> i have issues with cluster role bondings assignments rather
13:21 < rhallisey> karimb, so you create the namespace then you are trying to create a role for that namespace?
13:21 < karimb> yes
13:21 < karimb> exactly
13:22 < shurley> that makes sense, the svc acct that is being run does not have access to create a cluster role binding I am pretty sure. (does not get a clusterrolebinding as the sandbox role, just gets a namespaced rolebinding)
13:22 < shurley> could you instead create a rolebinding for the new namespace?
13:22 < karimb> shurley is there a way to give it access ?
13:22 < rhallisey> but what if you used the admin role
13:22 < karimb> normal role bindings do work
13:22 < karimb> i do both actually in my playbook
13:22 < karimb> the cluster wide action is the one failing
13:23 < rhallisey> do you need the clusterrolbinding vs rolebinding?
13:23 < karimb> i need both
13:23 < karimb> depending on the part of the playbook
13:23 < rhallisey> I guess you do since it's 2 namesapces
13:24 < shurley> is there anyway to not have a clusterrolebinding?
13:24 < karimb> i m not a fission expert, just translating their helm chart to an apb
13:24 < rhallisey> shurley, if he's using 2 namespaces, I don't think so
13:24 < karimb> but my understanding of the technology is that they dedicate a specific namespace to launch functions
13:25 < karimb> and that from this namespace, you need to be able to reach functions defined in each of the namespaces
13:25 < karimb> ( or something like that )
13:25 < karimb> actually i can find out, but pretty sure this wouldnt work without the cluster role binding
13:26 < karimb> plus i think it will be a common case
13:26 < rhallisey> karimb, that error in the broker log I thinik is because the broker is trying to create an elevated role.  The broker is using admin. You can try using admin
13:26 < karimb> ?
13:26 < rhallisey> sandbox_role: admin
13:26 < rhallisey> instead of cluster-admin
13:26 < karimb> yes, that s what i previously had
13:27 < rhallisey> still failed to launch the apb?
13:27 < shurley> I think that we need to be really smart about how we allow clusterrolebindings to be created. I would prefer that you grant access to desired namespaces for the svc acct that fission is running as rather then giving all the access to that svc account
13:28 < dzager> karimb: You'll need to elevate the role of the broker's service account to "cluster-admin" by default it is just an "admin"
13:28 < karimb> :)
13:28 < dzager> but I'm coming into the conversation late :/
13:28 < karimb> i think i ve never been able to launch an apb container with cluster-admin role
13:28 < karimb> shurley i think i disagree
13:28 < rhallisey> karimb, you would have to redeploy the broker
13:29 < rhallisey> to do what dzager is saying
13:29 < karimb> rhallisey i generally delete all pods from the ansible-service-broker namespace after each edit of the broker-config configmap
13:29 < karimb> ah ok dzager you re proposing a different thing
13:30 < karimb> i get it
13:30 < karimb> let me try that
13:30 < rhallisey> karimb, you could do this oc edit clusterrolebinding asb
13:31 < rhallisey> s/admin/cluster-admin/
13:31 < karimb> rhallisey no, that doesnt work
13:33 < karimb> rather went with oc adm policy add-cluster-role-to-user cluster-admin -z asb -n ansible-service-broker
13:33 < rhallisey> that works
13:34 < shurley> karimb: I am pretty sure that none of those things are going to work. (I could be wrong) but the issue is that svc-acct running the abp (which is not the asb svc-acct) is granted the sandbox-role ("edit", "admin", cluster-admin) to two specific namespaces.
13:34 < shurley> I do not think that a svc-acct, user, group can create cluster level resources (clusterrolebindings) unless they have a cluster role binding that gives them accecss to create those resources
13:35 < shurley> I could be wrong here, but that is my understanding
13:35 < karimb> isnt it the purpose of setting cluster-admin role to the sandbox_role  ?
13:35  * dzager nods in agreement. That makes sense to me. The apb has rolebindings, not clusterrolebindings...so it can't create/modify anything in the cluster scope as far as I understand
13:36 < shurley> to allow for an APB to create a clusterrolebing then we would have to give access to the APB to do anything it wants in the entire cluster with that sandbox role
13:36 < rhallisey> shurley, the reason we're having him set the asb service account to cluster-admin is so the sandbox_role can be cluster-admin
13:36 < rhallisey> shurley, I do agree with what you're saying about cluster role
13:36 < rhallisey> apbs should not have access to them or we're giving away a root kit
13:37 < karimb> but then again doesnt it imply that cluster-admin for the sandbox_role will never work ?
13:38 < dzager> karimb, having the cluster-admin for the sandbox_role gives you ultimate authority over that namespace to do some things you wouldn't be able to do with just "admin" or "edit" permissions.
13:38 < rhallisey> karimb, I don't know. I thought admin would allow you to create a clusterrolebinding
13:39 < rhallisey> karimb, can you paste the clusterrole you're creating
13:39 < karimb> rhallisey sure
13:39 < dzager> but the cluster-admin is just a role, if I give that to a namespaced role-binding (as we are)...then the use of the service account will not have cluster level privileges
13:39 < karimb> rhallisey https://github.com/karmab/fission-apb/blob/master/roles/provision-fission-apb/tasks/main.yml
13:40 < shurley> from the docs: A Role can only be used to grant access to resources within a single namespace. Here’s an example Role in the “default” namespace that can be used to grant read access to pods:
13:40 < shurley> A ClusterRole can be used to grant the same permissions as a Role, but because they are cluster-scoped, they can also be used to grant access to:
13:40 < rhallisey> dzager, that's right we're only creating a role
13:40 < rhallisey> then it won't work
13:41 < karimb> but then it means there are no way to do actions clusterwide from the apb ?
13:41 < karimb> right ?
13:41 < shurley> https://kubernetes.io/docs/admin/authorization/rbac/
13:42 < karimb> ( the reason clusterwide is needed is because of the use of custom resource definitions actually)
13:42 < dzager> I think shurley had a suggestion for how this would look if the APB were to be run by the Ansible Broker. I think another alternative is to not use the Ansible Broker to run this APB
13:43 < karimb> and now its when i realize why the cluster role is called ... fission-crd ....
13:45 < shurley> I don't know if we will be allowing an APB, through the broker, to get a cluster-admin cluster role.
13:45 < shurley> Is there no way to grant access to the fission-svc account to a namespace that has the function and it not work in that way?
13:46 < shurley> ^^ * to create a cluster-admin cluster role *
13:46 < karimb> no
13:47 < shurley> maybe I am wrong and we should have larger discussion about this.
13:47 < karimb> but just because of the architecture of the product
13:47 < karimb> that seems to be quite common those days
13:47 < karimb> having crds created by end users in their namespace
13:47 < karimb> and then controllers gathering them from anywhere
13:48 < dzager> karimb have you tried simply `oc run`ing the APB?
13:48 < karimb> dzager to manually run commands like oc new-project ?
13:50 < shurley> karimb: do you mind if we open an issue and use your APB as an example?
13:50 < shurley> I think this should warrent a larger discussion
13:50 < karimb> shurley i can open the issue myself no problem
13:50 < dzager> karimb: no, something like:
13:50 < dzager> # Start the nginx container using a different command and custom arguments.
13:50 < dzager>   oc run nginx --image=nginx --command -- <cmd> <arg1> ... <argN>
13:51 < shurley> karimb: ok thanks, I see why this should be a completly reasonable APB service
13:51 < karimb> ah so that i see which privileges i get or somethink
13:51 < dzager> instead say: oc run my_apb --image=docker.io/me/my_apb --command --->
13:51 < shurley> dzager: don't forget the extra-vars ^
13:54 < dzager> something closer: `oc run my_apb --image=docker.io/me/my_apb --command provision --extra-vars i=forget how=these are=mapped`
13:57 < karimb> actually i tried to oc rsh in the apb while executing ( adding a pause task)
13:57 < karimb> and indeed i cant do anything cluster related
13:58 < rhallisey> what would be nice is a role that allows you to rw from every namesapce you created
14:03 < dzager> karimb: I think there is a difference between 1) having a pause task and execing into the running apb pod and 2) using `oc run` to run the apb. I **think** in case 2, the pod running the APB would have all the permissions that you (you as in `oc whoami`) have
14:04 < dzager> I **think** that in case 2, you would have more freedom than we would ever want an APB to have in case 1
14:10 < karimb> dzager yes this is right
14:11 < jmontleon> shurley, is there any way to associate a clusterserviceclass to a registry adapter in the broker?
14:14 < rhallisey> shurley or karimb, can one of you create an issue for what we talked about?
14:27 < karimb> rhallisey inot i
14:27 < karimb> into it
14:29 < karimb> rhallisey https://github.com/openshift/ansible-service-broker/issues/576
14:30 < rhallisey> thank you karimb

[siamaksade]

Is there any workaround for this other than asking for user/pwd as apb params?

[djzager]

@siamaksade I have seen work to have namespaced brokers that are capable of doing cluster level actions. I have also worked on https://github.com/openshift/ansible-service-broker/blob/master/apb/install.yaml which runs the broker-apb in a pod. That broker-apb does everything from creating cluster role bindings to custom resource definitions.

To answer your question though, if your goal is to have an APB accessed via the service-catalog UI in OpenShift that has this kind of power, the answer is no. In that case you would need to use one of the workarounds like user/pwd as the apb's parameters.

[siamaksade]

@djzager I saw the broker installer but that is sort of contradictory to the goals of creating an APB. We could always run playbooks inside a pod, no reason to turn it into an APB

[djzager]

@siamaksade there is currently no additional workarounds that I am aware of and I believe you will be waiting for namespaced brokers before you could expect the Broker to support this in an intelligent way. Here is the proposal PR against the service-catalog kubernetes-retired/service-catalog#1826.