make bundle does not include roles and clusterroles in csv

[DTMad]

Bug Report

What did you do?

When running make bundle operator-sdk creates the CSV file and uses the base to fill fields like the description.
However with a folder structure like this

config/
-- /Common
--- /someFeature
---- roles.yaml
---- sa.yaml
---- ...
-- /Kubernetes
--- /someFeature
---- roles.yaml
---- sa.yaml
---- ...
-- /Openshift
--- /someFeature
---- roles.yaml
---- sa.yaml
---- ...

the bundle command only places the corresponding role and clusterrole for the operator itself into the CSV but all the others do not get moved into the CSV. When defining a specific namespace on the ClusterRoleBinding this leads to a problem because if installed into a different namespace with OLM the ClusterRoleBinding, Role and ServiceAccount still directs to the hard coded namespace.

What did you expect to see?

All the RBACs should be moved into the CSV to not depend on hard coded namespaces within bindings.

Environment

Operator type:

operator-sdk version: "v1.5.0",
commit: "98f30d59ade2d911a7a8c76f0169a7de0dec37a0",
kubernetes version: "v1.19.4",
go version: "go1.16.1"

/language go

[estroz]

@DTMad namespace is not considered when choosing which Roles/ClusterRoles to add to a CSV, only ServiceAccount name. If your operator's Deployment pod spec encodes a serviceAccountName ("default" is used if empty), all abound Roles/ClusterRoles that match via their Bindings are added to the CSV; all others are written to separate files in the bundle as-is. Are you saying this is not happening in your case, and namespace affects what gets added to the CSV?

/triage support

[DTMad]

This is exactly the problem.
We want to create more than one serviceAccount within the same namespace where the operator gets created. If those RBACs do not get pulled within the CSV they will always have the hard coded namespace and will not make it to the same namespace the operator is living in.

Is there a way to disable this behaviour?

[estroz]

If you are talking about the RBAC written directly to bundle/manifests, OLM should add the correct installation namespace at runtime; in fact, operator-sdk generate bundle strips namespaces from all manifests directly written to the bundle. Are you not seeing that happen? Do you have a concrete example to link that I can look at?

There is no way to disable ServiceAccount-to-RBAC linking, as all RBAC required by the operator's controllers should be bound to the ServiceAccount the operator is installed in.

[DTMad]

Exactly - the RBACs get written directly into the bundle/manifest folder and the namespaces set on the RoleBindings stay as they are. So this means they would get applied and reference a namespace which would probably not get created when the operator gets installed into a different namespace.

Here's our repository where we experience that problem:
https://github.com/Dynatrace/dynatrace-operator

FYI - we modified the makefile a bit to place the output folder within config/olm/<kubernetes/openshift>

[estroz]

Right, that makes sense, and is a completely valid configuration requirement for any operator. After doing some digging, I found that OLM does not configure namespaces in binding subjects, nor does it intend to; in fact, OLM wants to get away from configuring bundled manifests within its resolver/apply logic, and give configuration freedom to operator devs like yourself. Right now, this effectively means that an operator should be able to create RBAC needed outside of its own service account, namespaced according to runtime requirements. A few "helpers" will likely be produced to easily implement runtime configuration, perhaps in the form of:

• A library in https://github.com/operator-framework/operator-lib/, that wraps Manager.Add() so objects can be created before controllers start.
• A library to run something similar to kubectl kustomize <kustomize dir> so you can dynamically configure objects from manifests within the operator itself (at startup or within a controller).
• A separate service, installable by either operator devs or OLM, to do some sort of templating.

In the meantime, I will find out whether placing permissions and clusterPermissions in a CSV with a different service account than the operator's deployment (a) will work as intended, and (b) is idiomatic. If both are true, then a feature could probably be added to generate bundle to include RBAC bound to those extra service accounts in the CSV.

[estroz]

Followup:

(a) will work as intended, and (b) is idiomatic

Both (a) and (b) are true. So given something like the following CSV (assuming neither ServiceAccount exists), both ServiceAccounts foo and bar will be created with bindings within the operator's installation namespace:

apiVersion: operators.coreos.com/v1alpha1
kind: ClusterServiceVersion
metadata:
  name: foo-operator.v0.0.1
spec:
  install:
    spec:
      deployments:
      - name: foo-operator-controller-manager
        spec:
          template:
            spec:
              ...
              serviceAccountName: foo
      permissions:
      - rules:
        ...
        serviceAccountName: foo
      - rules:
        ...
        serviceAccountName: bar

Therefore this is a feature I'd like to add to generate bundle. Perhaps something like --extra-service-accounts <list>, where each ServiceAccount in the list that is bound to a role will be added to the CSV under permissions or clusterPermissions (depending on Role or ClusterRole), aggregated by serviceAccountName. Does that seem reasonable @DTMad?

/cc @jmrodri

[DTMad]

@estroz sounds great! Thanks for investigating.
Looking forward to that solution