Rootless Kubeflow part one (istio-cni) #2455

juliusvonkohout · 2023-05-09T08:37:47Z

Which issue is resolved by this Pull Request:
Resolves #2014

Description of your changes:

The main steps are adding an additional profile for istio-cni or ambient mesh, updating the documentation and manifest generation process. Then adding the baseline and restricted PSS as kustomize component to /contrib and extending the profile controller to annotate user namespaces with configurable PSS labels.

First Stage:

Implement Istio 1.17.5 and use it by default. This is important for the Kubeflow 1.8 feature freeze
Implement istio-cni (--set components.cni.enabled=true --set components.cni.namespace=kube-system) as second option.
Add simple tests similar to tests/gh-actions/install_istio.sh and tests/gh-actions/install_knative.sh for istio-cni and support both rootfull and rootless istio at the same time and give users one release to test

Second stage in a second PR:
4. Add pod security standards (https://kubernetes.io/docs/concepts/security/pod-security-standards/) base/restricted to manifests/contrib
5. Add istio-ambient to Kubeflow 1.9
6. Enforce PSS baseline (here you can still build OCI containers via Podman and buildah) in Kubeflow 1.9. It works with any istio
7. Optionally Enforce PSS restricted (this is where minor corner cases are affected) in Kubeflow 1.10

Third stage:
10. Upgrade Istio to 1.19 and use ambient service mesh by default in Kubeflow 1.10

Checklist:

Unit tests pass:
Make sure you have installed kustomize == 3.2.1
1. make generate-changed-only
2. make test

google-cla · 2023-05-09T08:37:50Z

Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

View this failed invocation of the CLA check for more information.

For the most up to date status, view the checks section at the bottom of the pull request.

Talador12 · 2023-05-09T15:42:06Z

To clarify, this pull request is the rootful daemonset with rootless sidecar solution?

juliusvonkohout · 2023-05-10T07:00:01Z

To clarify, this pull request is the rootful daemonset with rootless sidecar solution?

yes. The damonset is completely isolated and can be managed by the cluster operators. It resides in kube-system so it becomes part of the underlying cluster then. There is no network interaction at all

juliusvonkohout · 2023-05-12T16:47:11Z

@agilgur5 do you have any thoughts about this PR?

moorthy156

We are also running istio-cni to remove sidecar root requirement privileges, it is working fine us

Talador12 · 2023-06-16T15:19:56Z

We ended up shifting to use a dedicated physical cluster instead of a virtual cluster for kubeflow, but we still want to try out this model to improve our security footprint. Looking forward to when this work is merged

juliusvonkohout · 2023-06-19T16:03:33Z

@moorthy156 do you want to help with an upstream implementation? If yes, please contact me on slack.

kimwnasptd · 2023-07-13T14:37:06Z

@juliusvonkohout what's the state of this PR? Is it ready for review?

My only concern with the CNI is the initContainer redirection, which could break KServe etc that also has init container that fetches data #2014 (comment).

What's the state with this?

moorthy156 · 2023-07-14T18:09:03Z

@moorthy156 do you want to help with an upstream implementation? If yes, please contact me on slack.

I would love too. Can i get directions on this

MuhammadUsman90 · 2023-07-16T13:36:55Z

@moorthy156 do you want to help with an upstream implementation? If yes, please contact me on slack.

I would love too. Can i get directions on this

I would also like to be part of it. @juliusvonkohout @moorthy156 Can we arrange a small point on this?

juliusvonkohout · 2023-07-19T10:43:16Z

Yes, always reach out in the Kubeflow slack.

The istio package is upgraded to the latest stable version, 1.18.1. This upgrade is needed for running Kubeflow with rootless containers, as there are a lot of improvements in the latest istio version for that. Signed-off-by: Tzvetomir Stoyanov (VMware) <tz.stoyanov@gmail.com>

annajung · 2023-08-10T17:15:34Z

.github/workflows/kserve_kind_cni_test.yaml

@@ -0,0 +1,59 @@
+name: Build & Apply KServe manifests in KinD, using istio CNI
+on:
+  workflow_dispatch:


There should be a trigger here based on changes to specific directories, not just workflow_dispatch.

@tzstoyanov will check this on monday

As this Istio CNI functionality is still experimental and not enabled by default yet, I added only a manual trigger of the github action. However, added also an automatic trigger on every change that uses Istio CNI.

If I understand the trigger correctly, only people with write access will have access to trigger the workflow based on workflow_dispatch. So, not sure if it's necessary to keep

Yes, but this would be the easiest way to run Istio CNI tests. Lets keep it at least for one release, until Istio CNI is not enabled by default. We can remove the manual trigger at any time.

Should this test be triggered when there is a change to istio-cni-*? Can we trigger the test to ensure it runs successfully?

Ok, that makes more sense than the current logic. Changed the trigger action, thanks Anna.

common/istio-1-17/istio-install/base/patches/istiod-remove-pdb.yaml

juliusvonkohout · 2023-08-11T13:02:36Z

It would be easier to review if the PR is separated into
* 1.17.5 upgrade

* Istio-cni

* contrib components
Also for contrib, can you edit to follow https://github.com/kubeflow/manifests/blob/master/proposals/20220926-contrib-component-guidelines.md?

I will just remove /contrib from this PR for now. I think the istio changes should stay together here.

The Istio CNI functionality is still experimental, that's why the github action is triggered manually for now. Added an automatic trigger as well - on every PR that uses ISTIO CNI code. Signed-off-by: Tzvetomir Stoyanov (VMware) <tz.stoyanov@gmail.com>

tzstoyanov · 2023-08-14T14:13:13Z

@annajung comments are addressed, please can you take a look again

tests/gh-actions/install_knative-cni.sh

common/istio-cni-1-17/README.md

annajung · 2023-08-16T13:28:26Z

.github/workflows/kserve_kind_cni_test.yaml

@@ -0,0 +1,59 @@
+name: Build & Apply KServe manifests in KinD, using istio CNI
+on:
+  workflow_dispatch:


If I understand the trigger correctly, only people with write access will have access to trigger the workflow based on workflow_dispatch. So, not sure if it's necessary to keep

Co-authored-by: Anna <antheaj@vmware.com>

Trigger the test on every PR which changes istio-cni code only. Signed-off-by: Tzvetomir Stoyanov (VMware) <tz.stoyanov@gmail.com>

rename to istio-cni

annajung · 2023-08-17T15:14:12Z

Thanks everyone!
/lgtm

cc @kimwnasptd

kimwnasptd · 2023-08-24T15:17:55Z

Thanks for the review @annajung! I'll also try to take a look within the weekend and merge this

mpaulgreen · 2023-09-05T01:30:31Z

@juliusvonkohout The knative serving and kubeflow pods are still stuck with isito-init failing as described in Issue #2338. I was trying to run on Openshift 4.12.x and 4.13.x.

ENVOY_PORT=
INBOUND_CAPTURE_PORT=
ISTIO_INBOUND_INTERCEPTION_MODE=
ISTIO_INBOUND_TPROXY_ROUTE_TABLE=
ISTIO_INBOUND_PORTS=
ISTIO_OUTBOUND_PORTS=
ISTIO_LOCAL_EXCLUDE_PORTS=
ISTIO_EXCLUDE_INTERFACES=
ISTIO_SERVICE_CIDR=
ISTIO_SERVICE_EXCLUDE_CIDR=
ISTIO_META_DNS_CAPTURE=
INVALID_DROP=

2023-09-04T11:38:02.320411Z	info	Istio iptables variables:
PROXY_PORT=15001
PROXY_INBOUND_CAPTURE_PORT=15006
PROXY_TUNNEL_PORT=15008
PROXY_UID=1337
PROXY_GID=1337
INBOUND_INTERCEPTION_MODE=REDIRECT
INBOUND_TPROXY_MARK=1337
INBOUND_TPROXY_ROUTE_TABLE=133
INBOUND_PORTS_INCLUDE=*
INBOUND_PORTS_EXCLUDE=15090,15021,15020
OUTBOUND_OWNER_GROUPS_INCLUDE=*
OUTBOUND_OWNER_GROUPS_EXCLUDE=
OUTBOUND_IP_RANGES_INCLUDE=*
OUTBOUND_IP_RANGES_EXCLUDE=
OUTBOUND_PORTS_INCLUDE=
OUTBOUND_PORTS_EXCLUDE=
KUBE_VIRT_INTERFACES=
ENABLE_INBOUND_IPV6=false
DNS_CAPTURE=false
DROP_INVALID=false
CAPTURE_ALL_DNS=false
DNS_SERVERS=[],[]
OUTPUT_PATH=
NETWORK_NAMESPACE=
CNI_MODE=false
HOST_NSENTER_EXEC=false
EXCLUDE_INTERFACES=

2023-09-04T11:38:02.320573Z	info	Writing following contents to rules file: /tmp/iptables-rules-1693827482320445198.txt2641253036
* nat
-N ISTIO_INBOUND
-N ISTIO_REDIRECT
-N ISTIO_IN_REDIRECT
-N ISTIO_OUTPUT
-A ISTIO_INBOUND -p tcp --dport 15008 -j RETURN
-A ISTIO_REDIRECT -p tcp -j REDIRECT --to-ports 15001
-A ISTIO_IN_REDIRECT -p tcp -j REDIRECT --to-ports 15006
-A PREROUTING -p tcp -j ISTIO_INBOUND
-A ISTIO_INBOUND -p tcp --dport 15090 -j RETURN
-A ISTIO_INBOUND -p tcp --dport 15021 -j RETURN
-A ISTIO_INBOUND -p tcp --dport 15020 -j RETURN
-A ISTIO_INBOUND -p tcp -j ISTIO_IN_REDIRECT
-A OUTPUT -p tcp -j ISTIO_OUTPUT
-A ISTIO_OUTPUT -o lo -s 127.0.0.6/32 -j RETURN
-A ISTIO_OUTPUT -o lo ! -d 127.0.0.1/32 -m owner --uid-owner 1337 -j ISTIO_IN_REDIRECT
-A ISTIO_OUTPUT -o lo -m owner ! --uid-owner 1337 -j RETURN
-A ISTIO_OUTPUT -m owner --uid-owner 1337 -j RETURN
-A ISTIO_OUTPUT -o lo ! -d 127.0.0.1/32 -m owner --gid-owner 1337 -j ISTIO_IN_REDIRECT
-A ISTIO_OUTPUT -o lo -m owner ! --gid-owner 1337 -j RETURN
-A ISTIO_OUTPUT -m owner --gid-owner 1337 -j RETURN
-A ISTIO_OUTPUT -d 127.0.0.1/32 -j RETURN
-A ISTIO_OUTPUT -j ISTIO_REDIRECT
COMMIT
2023-09-04T11:38:02.320602Z	info	Running command: iptables-restore --noflush /tmp/iptables-rules-1693827482320445198.txt2641253036
2023-09-04T11:38:02.321779Z	error	Command error output: xtables parameter problem: iptables-restore: unable to initialize table 'nat'

Error occurred at line: 1
Try `iptables-restore -h' or 'iptables-restore --help' for more information.
2023-09-04T11:38:02.321791Z	error	Failed to execute: iptables-restore --noflush /tmp/iptables-rules-1693827482320445198.txt2641253036, exit status 2```

juliusvonkohout · 2023-09-05T13:02:00Z

@mpaulgreen The istio-cni documentation says that for openshifts special cni implementation you need the networkattachmentdefinitions. But we will not cover this here.

And the logs you posted there look like the normal istio. This PR here is for istio-CNI. I used both istios on Openshift, so it works. But you have to configure both specifically for Openshift, since Openshift has some special defaults and a different CNI plugin that needs extra settings. As said in the other issue consulting is available, just reach out on slack or linkedin.

But please only discuss istio-CNI here. This PR is for development, not paid support.

mpaulgreen · 2023-09-05T15:19:40Z

@juliusvonkohout You are correct. There are some additional configuration required istio-cni for OpenShift. I will stop posting defects of kubeflow with istio here.

kimwnasptd · 2023-09-07T14:48:13Z

Tested as well in a KinD cluster and everything worked as expected. Great to see this in, thank you everyone that worked hard!

/lgtm
/approve

google-oss-prow · 2023-09-07T14:48:32Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: juliusvonkohout, kimwnasptd

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [kimwnasptd]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

google-oss-prow bot added the do-not-merge/work-in-progress label May 9, 2023

google-oss-prow bot added the size/XXL label May 9, 2023

google-oss-prow bot requested review from annajung, elikatsis and yanniszark May 9, 2023 08:38

juliusvonkohout mentioned this pull request May 9, 2023

Add proper PSPs to enforce security and safety for Kubeflow on Kubernetes #2014

Closed

juliusvonkohout force-pushed the rootless branch from 16acf49 to 408d8ec Compare May 9, 2023 08:58

kimwnasptd mentioned this pull request May 9, 2023

Manifests WG Roadmap for KF 1.8 #2456

Closed

moorthy156 reviewed Jun 15, 2023

View reviewed changes

juliusvonkohout mentioned this pull request Jul 25, 2023

[kubeflow 1.8] Kubeflow 1.8 Tracking Issue #2442

Closed

5 tasks

tzstoyanov force-pushed the rootless branch 2 times, most recently from 6854548 to a6ddca7 Compare July 26, 2023 10:45

juliusvonkohout and others added 5 commits July 26, 2023 16:20

Proof of concept

c22ca62

remove legacy stuff

7260176

Update README.md

eba9538

Remove legacy stuff in 1.18

2d5d31b

juliusvonkohout force-pushed the rootless branch from bbd3c70 to 2d5d31b Compare July 26, 2023 14:26

juliusvonkohout added 2 commits July 26, 2023 16:38

Update istio 1.17.3 documentation and enable patches

f184921

restore 1.6 and add knative-local-gateway comment

64bffe9

annajung reviewed Aug 10, 2023

View reviewed changes

common/istio-1-17/istio-install/base/patches/istiod-remove-pdb.yaml Outdated Show resolved Hide resolved

readd istiod-remove-pdb

e25b0ef

juliusvonkohout and others added 2 commits August 11, 2023 15:04

we will do /contrib changes in a new PR with capital one

124c5cc

annajung reviewed Aug 16, 2023

View reviewed changes

juliusvonkohout and others added 3 commits August 16, 2023 15:29

Update tests/gh-actions/install_knative-cni.sh

59f3688

Co-authored-by: Anna <antheaj@vmware.com>

Change the trigger for the Istio CNI github action

a893541

Trigger the test on every PR which changes istio-cni code only. Signed-off-by: Tzvetomir Stoyanov (VMware) <tz.stoyanov@gmail.com>

Update README.md

5fd16d0

rename to istio-cni

google-oss-prow bot assigned annajung Aug 17, 2023

google-oss-prow bot added the lgtm label Aug 17, 2023

juliusvonkohout mentioned this pull request Aug 18, 2023

profile controller fails to update profile owner kubeflow/kubeflow#6576

Open

This was referenced Aug 25, 2023

Kubeflow istio init issue #2338

Closed

Fix kustomize 5 warning for /common/istio-1-16/istio-install/ #2418

Closed

Adding oauth2-proxy as optional alternative to oidc-authservice #2409

Merged

kromanow94 mentioned this pull request Aug 30, 2023

oauth2-proxy and Istio mesh support #2516

Closed

google-oss-prow bot assigned kimwnasptd Sep 7, 2023

google-oss-prow bot added the approved label Sep 7, 2023

google-oss-prow bot merged commit b5b9a7b into kubeflow:master Sep 7, 2023
4 checks passed

juliusvonkohout changed the title ~~Rootless Kubeflow~~ Rootless Kubeflow part one (istio-cni) Sep 7, 2023

juliusvonkohout mentioned this pull request Sep 13, 2023

Rootless Kubeflow #2528

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Rootless Kubeflow part one (istio-cni) #2455

Rootless Kubeflow part one (istio-cni) #2455

juliusvonkohout commented May 9, 2023 •

edited

google-cla bot commented May 9, 2023

Talador12 commented May 9, 2023 •

edited

juliusvonkohout commented May 10, 2023 •

edited

juliusvonkohout commented May 12, 2023

moorthy156 left a comment

Talador12 commented Jun 16, 2023

juliusvonkohout commented Jun 19, 2023 •

edited

kimwnasptd commented Jul 13, 2023

moorthy156 commented Jul 14, 2023

MuhammadUsman90 commented Jul 16, 2023

juliusvonkohout commented Jul 19, 2023

annajung Aug 10, 2023

juliusvonkohout Aug 11, 2023

tzstoyanov Aug 14, 2023

annajung Aug 16, 2023

tzstoyanov Aug 16, 2023

annajung Aug 16, 2023

tzstoyanov Aug 17, 2023

juliusvonkohout commented Aug 11, 2023 •

edited

tzstoyanov commented Aug 14, 2023

annajung Aug 16, 2023

annajung commented Aug 17, 2023

kimwnasptd commented Aug 24, 2023

mpaulgreen commented Sep 5, 2023 •

edited

juliusvonkohout commented Sep 5, 2023

mpaulgreen commented Sep 5, 2023 •

edited

kimwnasptd commented Sep 7, 2023

google-oss-prow bot commented Sep 7, 2023

Rootless Kubeflow part one (istio-cni) #2455

Rootless Kubeflow part one (istio-cni) #2455

Conversation

juliusvonkohout commented May 9, 2023 • edited

google-cla bot commented May 9, 2023

Talador12 commented May 9, 2023 • edited

juliusvonkohout commented May 10, 2023 • edited

juliusvonkohout commented May 12, 2023

moorthy156 left a comment

Choose a reason for hiding this comment

Talador12 commented Jun 16, 2023

juliusvonkohout commented Jun 19, 2023 • edited

kimwnasptd commented Jul 13, 2023

moorthy156 commented Jul 14, 2023

MuhammadUsman90 commented Jul 16, 2023

juliusvonkohout commented Jul 19, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

juliusvonkohout commented Aug 11, 2023 • edited

tzstoyanov commented Aug 14, 2023

Choose a reason for hiding this comment

annajung commented Aug 17, 2023

kimwnasptd commented Aug 24, 2023

mpaulgreen commented Sep 5, 2023 • edited

juliusvonkohout commented Sep 5, 2023

mpaulgreen commented Sep 5, 2023 • edited

kimwnasptd commented Sep 7, 2023

google-oss-prow bot commented Sep 7, 2023

juliusvonkohout commented May 9, 2023 •

edited

Talador12 commented May 9, 2023 •

edited

juliusvonkohout commented May 10, 2023 •

edited

juliusvonkohout commented Jun 19, 2023 •

edited

juliusvonkohout commented Aug 11, 2023 •

edited

mpaulgreen commented Sep 5, 2023 •

edited

mpaulgreen commented Sep 5, 2023 •

edited