Kubeflow 1.9 Enterprise grade Agenda

[juliusvonkohout]

For general Kubeflow:

• Missing namespace in authorizationpolicy Missing namespace in authorizationpolicy manifests#2298
• performance issues with admission webhook #6708
• Renaming Notebooks to workbenches as discussed with Kimonas Sotirchos. There are not only Jupyterlabs but also Rstudios and Vscodes and other customer specific applications (e.g. Labelstudio). If possible make the istio-virtualservice redirect path configurable instead of always using NB_PREFIX.
• Docker -> OCI Update form-image.component.html Docker -> OCI Update form-image.component.html #6673
• The kserve logo does have a different color than the other logos chore(icons): the model icon is grey instead of black #6656
• oidc-authservice https://github.com/kubeflow/manifests/pull/2150 and https://github.com/kubeflow/kubeflow/pull/6609 for proper programatic authentication via tokens and serviceaccounts
• Remember the most recently selected namespace Remember the most recently selected namespace #6663
• There is also another meta issue now Notebooks WG Roadmap for KF 1.7 #6672

Security:

• Harden the istio 1.16 configuration manifests#2357
• Add authorization mechanisms in new Katib UI backend katib#1983.
• Protection from fake kubeflow-userid header impersonation #6702
• Add tensorboards-web-app.yaml networkpolicy manifests#2304
• Are you open to a rootless istio (istio-cni) for Kubeflow 1.7 ? Currently almost everything runs insecurely as root. I could also add offical Kubernetes podsecuritystandards: restricted then for Kubernetes 1.23+. https://www.arrikto.com/blog/securing-your-kubeflow-deployment-with-kyverno-policies/?utm_content=222104897&utm_medium=social&utm_source=linkedin&hss_channel=lis-ZPwp5HFOSx is also interesting. But they seem to not know how dangerous root containers are. I think for kubeflow PSA /PSS/ Kyverno are the best options.
• Namespace sharing is completely broken from a security perspective. You can easily escalate from default-viewer to default-editor. Furthermore a malicious collaborator can easily extract the serviceaccount token, leave the company and still login with that as bearer token and impersonate other users. Very dangerous. I tried a lot, but the only sane way if found is to add a checkbox on the collaborator page that says "Delete all secrets, serviceaccounts and pods when removing a collborator via kubectl delete serviceaccounts,secrets,pods --all --force. The default secrets and serviceaccounts will be immideately recreated by the Kubeflow profile controllers, but manually created ones are your own resposibility. This might lead to a very short service interuption for e.g. KFP and other Kubeflow components." We could just display this as Information or try to implement it in the profile controller.
• Katib Narrow and explicit RBAC Narrow and explicit RBAC katib#1639
• The pipeline UI allows reading other peoples artifacts. The artifact proxy in the user namespace is insecure and obsolete. In the UI you can just get the artifact link from another user, remove the ?namespace=xxx parameter at the end and the UI server will fake the corresponding user for you. So if you know the S3/GCS path you can read other guys artifacts.

For Pipelines maybe :

• [feature] Seek alternative for object store solution other than MinIO [feature] Seek alternative for object store solution other than MinIO pipelines#7878
• Proper cache configuration for v2 pipelines too feat(backend) V1 Pipelines add maximum_cache_staleness and default_cache_staleness pipelines#8270
• Support big (5+ GB) s3 artifacts in pipelines EOF when many steps using same big input artifact argoproj/argo-workflows#9525 and Download large file from S3 cause memory rise in the init container argoproj/argo-workflows#1322 (they are not solved)
• feat(backend): isolate artifacts per namespace/profile/user using only one bucket feat(backend): isolate artifacts per namespace/profile/user using only one bucket pipelines#7725 We need to add gcs and aws s3 support to get it merged
• implement a caching strategy for PVC data_passing_method. (https://kubeflow-pipelines.readthedocs.io/en/latest/_modules/kfp/dsl/_pipeline.html?highlight=data_passing_method#). For example if one uses a pvc named after the pipeline for all executions of that pipeline versions this could be useful.
• [Multi User] Support separate metadata for each namespace pipelines#4790

[jbottum]

@juliusvonkohout thanks for this list! Dominik and I will review.

[kimwnasptd]

I would also add securing the Katib web app's backend, with SubjectAccessReviews, as well kubeflow/katib#1983.

Right now folks can manipulate (CRUD) Experiment CRs in any namespace via the browser

[juliusvonkohout]

Good first issues

• listruns performance is bad because of shitty SQL queries [backend] severe performance problem in listruns API pipelines#9890 please use fix(backend): Fix performance issue within a mysql request pipelines#9680 as a reference. Here the tale begins https://github.com/kubeflow/pipelines/blob/ad058b5321f5fe74bbd10845778e6627fb9aa5cb/backend/src/apiserver/storage/run_store.go#L109. Maybe the frontend is not using pagination as well. I do not know where the bottleneck is.
• Try out kfp v2 rootless fix(backend): set correct permissions for local directory pipelines#7916
• SDK support for v1 namespaced pipelines feat(sdk): Upload namespaced pipeline definitions. Part of #4197 pipelines#8512
• Reviewing Improve the performance of most Kubeflow backends web-apps: Implement watch to avoid constant polling from backend to K8s #7080
• Workbenches as successor to notebooks Kubeflow Notebooks 2.0 (Kubeflow Workspaces) #7156
• General culling in old "noteboks" Notebook Culling for VSCode/CodeServer style notebooks #6920
• Support big (5+ GB) s3 artifacts in pipelines EOF when many steps using same big input artifact argoproj/argo-workflows#9525 and Download large file from S3 cause memory rise in the init container argoproj/argo-workflows#1322 (they are not solved)
• admission webhook performance performance issues with admission webhook #6708
• Ray istio/headless [Feature] Support Istio ray-project/kuberay#1005

[yurkoff-mv]

It would also be nice to add Dex protection against multiple requests that eventually crash the cluster. (dexidp/dex#1292)

In Kubeflow 1.8 there was no external access to Inference Services.
(kubeflow/manifests#2576)

[bhack]

We really need to prioritize the lowering of the installation and upgrade management overhead:
#3173

[juliusvonkohout]

@bhack feel free to join the manifests WG meeting

[juliusvonkohout]

Closed in favor of kubeflow/manifests#2727