Scope Kubeflow components in given namespace

[Jeffwan]

In my current company, there're few orgs/platforms like to leverage KFP. Besides multi-user KFP, I am also evaluating if it's possible to deploy KFP per namespace since users are ok to share experiments in the same namespace.

If we see instruction to install Kubeflow in single-user mode. There're some cluster-scoped-resources.
https://github.com/kubeflow/pipelines/blob/master/manifests/kustomize/cluster-scoped-resources/kustomization.yaml#L10-L12

Besides CRD, I see there's some cluster-role and bindings in cache-deployer
https://github.com/kubeflow/pipelines/tree/master/manifests/kustomize/base/cache-deployer/cluster-scoped

Seems the code level already support NAMESPACE_TO_WATCH that means cluster scope permissions is not needed. I think I can file a PR to remove it?

Does anyone know pitfalls to use KFP per namespace?

/kind question

[numerology]

/assign @Ark-kun

Do you know if there's any potential caveat besides caching in this case?

[Bobgy]

pipelines/manifests/kustomize/base/cache-deployer/cluster-scoped/cache-deployer-clusterrole.yaml

Line 21 in ec721fe

- mutatingwebhookconfigurations

is a cluster-scoped resource and RBAC for cluster-scoped resource can only be granted via ClusterRole: https://kubernetes.io/docs/reference/access-authn-authz/rbac/#role-and-clusterrole.

That was the reason we included those

[Bobgy]

If a namespaced install do not need caching feature, then you can install the CRDs and multiple namespaced installations.
Namespaced installation is indeed a feature we support.

[Jeffwan]

pipelines/manifests/kustomize/base/cache-deployer/cluster-scoped/cache-deployer-clusterrole.yaml

Line 21 in ec721fe

- mutatingwebhookconfigurations

is a cluster-scoped resource and RBAC for cluster-scoped resource can only be granted via ClusterRole: https://kubernetes.io/docs/reference/access-authn-authz/rbac/#role-and-clusterrole.
That was the reason we included those

I see. that would be the blocker to create multiple namespaced installations. We can either remove cache-deployer as you suggest, or make some changes in deployer to create different webhooks like cache-webhook-${namespace} and use NamespaceSelector in request matching, multiple namespaced installation can share same cluster role but to create different cluster role bindings.. This is not elegant and I think removing cache makes more sense

[Bobgy]

If making a pure namespaced mode KFP is of high value to you, we can accept a PR for a KFP env without cache.

[Jeffwan]

Sounds good. I file a PR #4796

[Ark-kun]

make some changes in deployer to create different webhooks like cache-webhook-${namespace}

The cache deployer already does that.

pipelines/backend/src/cache/deployer/deploy-cache-service.sh

Line 28 in b0a87e7

MUTATING_WEBHOOK_CONFIGURATION_NAME="cache-webhook-${NAMESPACE}"

Do you think this solves your issue?

use NamespaceSelector in request matching

This is pretty easy to do, but please note that NamespaceSelector cannot match on namespace name, only on labels. 🤦

P.S. I wonder about scoping other services like Minio and Argo.

[Jeffwan]

Do you think this solves your issue?

Yes, I checked the source and the webhook get created using given namespace. cache-deployer still need cluster level resources. As I said In the #4781 (comment), each installation can share same cluster role but still need to create different cluster role bindings. The real world case is tenancy can not create cluster resource and their permission is scoped to the namespace. I feel like in this case, it's better to get ride of any cluster level resources

P.S. I wonder about scoping other services like Minio and Argo.

Argo supported managed namespace https://argoproj.github.io/argo/managed-namespace/. I think minio or mysql doesn't need to be scoped.

[Ark-kun]

I feel like in this case, it's better to get ride of any cluster level resources

That would be ideal, but might not be always feasible. For example, CRDs like Argo Workflow are cluster scoped.

I really wish Kubernetes had support for namespace-scoped mutating webhooks.

Another alternative would be to integrate hook support into Argo.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[stale]

This issue has been automatically closed because it has not had recent activity. Please comment "/reopen" to reopen it.

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.