Rootless Kubeflow

[juliusvonkohout]

Rootless Kubeflow

This is an issue to track the progress of the following proposal https://github.com/kubeflow/manifests/blob/master/proposals/20200913-rootlessKubeflow.md.

Goals

We want to run Kubeflow as rootless as possible according to CNCF/Kubernetes best practices.
Most enterprise environments will require this as well.

Implementation details

The main steps are adding an additional profile for istio-cni and later ambient mesh, updating the documentation and manifest generation process.
Only istio-cni or istio ambient mesh can run rootless as explained here https://istio.io/latest/docs/setup/additional-setup/cni/.
Istio-cni will still nedd a deamonset in kube-system, but that is completly isolated from user workloads.
The ambient mesh should get rid of this as well and also has the benefit of removing the istio initcontainers and sidecars altogether.
Then adding the baseline and restricted PSS as kustomize component to /contrib and extending the profile controller to annotate user namespaces with configurable PSS labels.

We want to use a staged approach.

First Stage:

1. Implement Istio 1.17.5 and use it by default, because 1.17. is what we have planned to use for Kubeflow 1.8.
2. Implement istio-cni (--set components.cni.enabled=true --set components.cni.namespace=kube-system) as second option.
3. Add simple tests similar to tests/gh-actions/install_istio.sh and tests/gh-actions/install_knative.sh for istio-cni and support both rootfull and rootless istio at the same time and give users one release to test

Second stage:
4. Add pod security standards (https://kubernetes.io/docs/concepts/security/pod-security-standards/) base/restricted to manifests/contrib
5. Enforce PSS baseline (here you can still build OCI containers via Podman and buildah). It works with any istio
6. Enable Warnings for violations of restricted PSS
7. Add tests to make sure that the PSS are used and tested in the CICD
8. Optionally Enforce PSS restricted (this is where minor corner cases are affected)

Third stage:
9. Upgrade Istio to 1.19 to make the ambient mesh available
10. Add istio-ambient as an option to the next Kubeflow release.

fourth stage:
11. Use the ambient service mesh by default in Kubeflow 1.10.

Non-Goals

This does not cover Application level CVEs, only cluster level security.

Does this break any existing functionality?

So far not. Only PSS restricted may block the incredibly dangerous and unprofessional Docker in Docker.
This is a rarely used feature from the KFP SDK.
With PSS baseline you can still build OCI images with Podman for example.
We should replace Docker with the cli compatible podman in the KFP SDK https://kubeflow-pipelines.readthedocs.io/en/1.8.22/source/kfp.containers.html?highlight=kfp.containers.build_image_from_working_dir#kfp.containers.build_image_from_working_dir.

Does this fix/solve any outstanding issues?

We are not following best practices and this is forbidden in most enterprise environments.
The progress is tracked in #2528

[juliusvonkohout]

The first stage has been implemented in #2455

[juliusvonkohout]

superseedes #2014

[kimwnasptd]

Adding a comment after our live discussion with @juliusvonkohout . Let's confirm we can't run baseline PSS with the default Istio, to further evaluate how/if to enable Istio CNI by default for 1.9.

[peterj]

I have experience with Istio/Ambient mesh and would love to help out if kubeflow is planning to adopt ambient in the future.

[juliusvonkohout]

@peterj please reach out on the Kubeflow Slack or Linkedin.

[diegolovison]

Should we have also a definition of done for this issue being able to deploy using kind and podman?

[juliusvonkohout]

Istio 1.22 #2714 is related

[juliusvonkohout]

@biswajit-9776 is working on this issue within GSOC

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[juliusvonkohout]

/lifecycle frozen