Istio does not work in cluster with restricted psp as default

[gitlawr]

Rancher Version
master 8/28

Similar to #20229, but for validating Istio.

Steps to reproduce

• Create a custom cluster with pod security policy enabled and the default set to restricted
• After the cluster has been created, enable Istio.

[gitlawr]

With the default configurations, Istio proxy sidecars require NET_ADMIN capability to set iptable rules, which is not ideal from security perspective.
To address this, A proper way is to use the Istio CNI plugin that operates iptables in the CNI layer, so that the sidecars can work without additional capabilities.

Here is the workflow to do that at the moment, which requires the system project to have a unrestricted PSP.

1. Create/Edit a RKE cluster with the following configuration:

services:
  kubelet:
    extra_binds:
      - '/lib/modules:/lib/modules'

So that the cni plugin can leverage the iptables module to set networking rules.
2. Make System project nonrestricted, or grant proper PSP so that Istio control plane can set up as expected.
3. Install the cni plugin to the cluster:

# The istio-cni chart is in istio/cni repo.
helm template install/kubernetes/helm/istio-cni --name=istio-cni --namespace=kube-system \
    --set logLevel=info \
    --set excludeNamespaces={"istio-system,kube-system"} | kubectl apply -f -

4. Enable Istio with a custom answer istio_cni.enabled: true

After these steps, Istio injected sidecars can work with restricted PSP.

NB: The Istio CNI plugin is in alpha phase at the moment, see.

[soumyalj]

Tested the above steps on a PSP enabled master setup. With the workaround, Istio is functional.Graphs are displayed correctly.

Note: With the Istio CNI plugin, only Istio-proxy sidecar is deployed. Istio-init sidecar is not deployed . (In a regular setup without PSP enabled, both istio-proxy and istio-init containers are deployed)

[strowi]

The namespace that uses istio also needs at least NET_ADMIN and NET_RAW psp-permissions, not only the default.
Might be worth mentioning in the docs until a fix is possible.

[deniseschannon]

@rancher-max Please follow these instructions to get PSPs working with Istio. https://rancher.com/docs/rancher/v2.x/en/cluster-admin/tools/istio/setup/enable-istio-in-cluster/enable-istio-with-psp/

[rancher-max]

These instructions have been validated as working exactly as is. Please note that this uses istio-cni, which is currently in Alpha status at the time of the writeup: https://istio.io/about/feature-stages/