[Multi User] Integrate KFP multi user with KF 1.1

[Bobgy]

Part of #1223 (that's the main issue tracker).

I'm using this issue as a tracker for all the efforts of merging KFP multi user related changes to upstream kubeflow repo.

/priority p0
/assign @Bobgy
/kind feature

My fork to merge kfp multi user manifests with KF 1.0: https://github.com/Bobgy/manifests/tree/kfp-multi-user-master

• Enable istio-injection in kubeflow namespace. kubeflow#3866 blocked by
  • Two examples for disabling istio injection in kubeflow namespace Two examples for disabling istio injection in kubeflow namespace manifests#703
  • Add sidecar.istio.io/inject false annotations to all stateful sets in kubeflow ns Add sidecar.istio.io/inject false annotations to all stateful sets in kubeflow ns manifests#711
  • Add istio sidecar inject false to deployments in kubeflow ns Add istio sidecar inject false to deployments in kubeflow ns manifests#712
  • Added istio sidecar false annotation to metadata deployments Added istio sidecar false annotation to metadata deployments manifests#804
  • [Backend] Always disable istio sidecar injection [Backend] Always disable istio sidecar injection #3305
  • Disable istio sidecar injection in simple tfjob Disable istio sidecar injection in simple tfjob test training-operator#1148
  • istio injection in Kubeflow namespace breaks KF 1.0 configs kfctl#296
  • Add kubeflow namespace resource to enable istio injection Add kubeflow namespace resource to enable istio injection manifests#1153
• [Deferred] Secure KFAM with mutual TLS and istio authorization rules Secure KFAM with mutual TLS and istio authorization rules kubeflow#4871 blocked by
  • [KFAM] LivenessProbe doesn’t work with istio mutual TLS kubeflow#4865
  • Profiles deployment has missing port kubeflow/manifest#976 Port is required for istio sidecar to intercept traffic for the pod.
• Migrate Kubeflow's KFP manifests using kustomize v3
  • Argo kustomize v3 Argo V3 Kustomize Manifest manifests#1090
  • KFP kustomize v3 [Pipelines] Reuse KFP standalone manifests in kubeflow/manifests manifests#1048
• Enable multi user mode for Kubeflow Pipelines [KF 1.1] Enable KFP multi user mode without istio security #4159 [KF 1.1] Secure KFP multi user mode with istio #4160 feat: enable istio sidecar for kubeflow namespace GoogleCloudPlatform/kubeflow-distribution#80
• Some last mile bug fixes feat: Add application and common labels to KFP and various fixes manifests#1374
• Integrate with non GCP platforms [Pipelines Multi User] integrate with non GCP platforms manifests#1364

[nrchakradhar]

@Bobgy Though this is not strictly multi-user, #2858, this needs to be addressed since users may want to use different namespace.

[Bobgy]

@nrchakradhar Thanks for bringing this up, we have addressed it in #2665 (but we are putting together all pieces of multi user support, you cannot use it right now). This issue is meant to track efforts that need to be merged into Kubeflow, so this particular issue isn't tracked here.

[nrchakradhar]

@Bobgy Thanks for the reference.

[Jeffwan]

@Bobgy I see code reference to multi-user but have not tested it. There're also a few issues still opening. just like to know do we have a timeline or milestone when multi-user in pipeline will be available?

[Bobgy]

Hi @Jeffwan! Please see the main issue tracker for progress: #1223.
And you may try phase 1 multi user support in #1223 (comment)
(We are working on phase 2)

[Jeffwan]

Hi @Jeffwan! Please see the main issue tracker for progress: #1223.
And you may try phase 1 multi user support in #1223 (comment)
(We are working on phase 2)

Cool. I will have a check on phase I first and leave some feedbacks

[jlewi]

@Bobgy Could you please clarify what this issue is tracking? What is the remaining work? You asked me #1223 (comment)
to take a look. What is it you want me to take a look at?

[Bobgy]

@jlewi This tracks remaining work to merge manifests to kubeflow manifests master. The unchecked items above are remaining. (I guess you'd want a breakdown for Enable multi user mode for Kubeflow Pipelines, I'll keep updating this when working on this.)

[Bobgy]

I'm now using https://github.com/kubeflow/pipelines/projects/5 to track efforts merging multi user mode to KF 1.1. Closes this.

[Bobgy]

In fact, this issue is still clearer on the steps. Keep using this one

[Bobgy]

TODO: I also need to cherry pick a bunch of multi user mode PRs to 1.1 branch.

[jlewi]

@Bobgy Do we have an ETA for when all of the PRs will be on the 1.1 branch?

[Bobgy]

@jlewi GoogleCloudPlatform/kubeflow-distribution#80 and kubeflow/manifests#1374 are the only ones left.
If they are merged, I can create cherry pick PRs tomorrow.

[jlewi]

GoogleCloudPlatform/kubeflow-distribution#80 Per comment in this issue turning on istio-sidecar injection in the kubeflow namespace is a pretty big change. Do we really need to it this late in the release cycle?

[Bobgy]

@jlewi oooops, I just realized ASM 1.4 doesn't support RBAC v1alpha1: https://cloud.google.com/service-mesh/docs/archive/1.4/docs/supported-features#authorization_policy.

I'll port the istio RBAC policies to Authorization v1beta1.

EDIT: I tried in the cluster, servicerole and servicerolebinding CRDs exist in the cluster, but they don't have any real effect. Requests not satisfying these requirements also just pass through.

[Bobgy]

It will be the same problem for any component that use servicerole and servicerolebindings to authorize, e.g. jupyter notebooks.

[Bobgy]

It will be the same problem for any component that use servicerole and servicerolebindings to authorize, e.g. jupyter notebooks.

Hmmmm, I was wrong. I tried a notebook, when accessing it from another account without permission. I got RBAC: access denied as expected. So something is not working correctly for KFP. I'll troubleshoot it.

[Bobgy]

OK, I found the culprit, it's knative's istio rule: https://github.com/kubeflow/manifests/blob/c300ee9ca30d8b606cd9713d120721f36e86b0ec/knative/knative-serving-install/base/service-role-binding.yaml#L11.

It basically allowed any network access from any user to any services in the namespace knative was deployed to.

But knative was already moved to its own namespace, so this should no longer be a problem, we can test latest version.

[Bobgy]

/cc @jlewi
I found the cause, see above comment.

[Bobgy]

I'll keep this open until verified on GCP platform

[Bobgy]

UPDATE: verified KFP 1.0.0 with multi user mode working properly on GCP platform.