Istio 100.0.0 v2 : book demo app deployment fails on SELinux RHEL 8.4 enabled on the clusters

[anupama2501]

What kind of request is this (question/bug/enhancement/feature request):
Bug

Steps to reproduce (least amount of steps as possible):
SELinux RHEL 8.4 docker-20.10.7, 0.2.rc1-RPM

1. Create a custom rke1 SELinux RHEL enabled cluster from the dashboard with 3 worker nodes, 1 etcd and 1 cp
2. Once the clusters are up and active
3. deploy istio v2 from the dashboard UI -->apps & Marketplace--> istio. Version: 100.0.0
4. Istio deploys successfully
5. create a namespace with istio sidecar injection enabled.
6. create the book demo app. The pods of the app are seen stuck in Updating.

Result:
istio-init container is in crashloopBackoff with the following error:

iptables-restore --noflush /tmp/iptables-rules-1624044808704787141.txt290487748
iptables-save
iptables-restore v1.6.1: iptables-restore: unable to initialize table 'nat'
Error occurred at line: 1
Try `iptables-restore -h' or 'iptables-restore --help' for more information.
panic: exit status 2
goroutine 1 [running]:
istio.io/istio/tools/istio-iptables/pkg/dependencies.(*RealDependencies).RunOrFail(0xd819c0, 0x9739cb, 0x10, 0xc00000cba0, 0x2, 0x2)
	istio.io/istio@/tools/istio-iptables/pkg/dependencies/implementation.go:44 +0x96
istio.io/istio/tools/istio-iptables/pkg/cmd.(*IptablesConfigurator).executeIptablesRestoreCommand(0xc0000efd30, 0x7f31b4f63001, 0x0, 0x0)
	istio.io/istio@/tools/istio-iptables/pkg/cmd/run.go:484 +0x3aa
istio.io/istio/tools/istio-iptables/pkg/cmd.(*IptablesConfigurator).executeCommands(0xc0000efd30)
	istio.io/istio@/tools/istio-iptables/pkg/cmd/run.go:491 +0x45
istio.io/istio/tools/istio-iptables/pkg/cmd.(*IptablesConfigurator).run(0xc0000efd30)
	istio.io/istio@/tools/istio-iptables/pkg/cmd/run.go:438 +0x2507
istio.io/istio/tools/istio-iptables/pkg/cmd.glob..func1(0xd5c740, 0xc0000d2700, 0x0, 0x10)
	istio.io/istio@/tools/istio-iptables/pkg/cmd/root.go:56 +0x14e
github.com/spf13/cobra.(*Command).execute(0xd5c740, 0xc00001e130, 0x10, 0x11, 0xd5c740, 0xc00001e130)
	github.com/spf13/cobra@v0.0.5/command.go:830 +0x2aa
github.com/spf13/cobra.(*Command).ExecuteC(0xd5c740, 0x40574f, 0xc000080058, 0x0)
	github.com/spf13/cobra@v0.0.5/command.go:914 +0x2fb
github.com/spf13/cobra.(*Command).Execute(...)
	github.com/spf13/cobra@v0.0.5/command.go:864
istio.io/istio/tools/istio-iptables/pkg/cmd.Execute()
	istio.io/istio@/tools/istio-iptables/pkg/cmd/root.go:284 +0x2d
main.main()
	istio.io/istio@/tools/istio-iptables/main.go:22 +0x20

Expected Result:
The istio-init container should come up active and the errors should not be seen

Other details that may be helpful:
If we set PSP to unrestricted for SELinux RHEL 8.4 hardened clusters and deploy the book demo app, the app deploys successfully without any errors.

Environment information

• Rancher version (rancher/rancher/rancher/server image tag or shown bottom left in the UI):

master-2ca2ad9c602f1d64e30a9ccfe093c6c39aefc728-head

• Installation option (single install/HA): Single

Cluster information

• Cluster type (Hosted/Infrastructure Provider/Custom/Imported): Node driver
• Kubernetes version (use kubectl version):

v1.20.7 

• Docker version (use docker version):

docker-20.10.7

#33285

[axeal]

I tested this with v2.6.2, a single all role node custom cluster running RHEL 8.4 (ami-0277fbe7afa8a33a6) and Kubernetes v1.21.6-rancher1-1 and it worked as expected with rancher-istio:100.0.2+up1.10.4 and SELinux enforcing, my steps:

Without PSP:

1. Provision a Rancher v2.6.2 instance and a single all role node customer cluster running RHEL 8.4, with PSP enabled and set to restricted by default (be sure to run systemctl disable nm-cloud-setup.service nm-cloud-setup.timer on the RHEL host before adding it to the cluster per Pod communication not working on RHEL 8.4 on cloud providers rke2#1053 (comment)).
2. Set PSP on default Project to unrestricted
3. Install rancher-istio:100.0.2+up1.10.4 with Pilot and Ingress Gateway
4. Deploy bookinfo app successfully https://istio.io/v1.10/docs/examples/bookinfo/#start-the-application-services

With PSP:

1. Remove bookinfo app deployed above
2. Upgrade rancher-istio chart to add CNI component
3. Set PSP on default Project to restricted
4. Deploy bookinfo app successfully https://istio.io/v1.10/docs/examples/bookinfo/#start-the-application-services

[deniseschannon]

@samkulkarni20 Can we have the QA team help validate if this is still an issue esp based on Alex's feedback?

[doflamingo721]

With reference to the steps mentioned by @axeal, the issue is working as expected on rancher 2.6.2 and 2.6-head.
Setup Configuration:

Rancher server:

• Installation option : Docker
• Provider: ec2 single node.

Information about the Cluster:

• Kubernetes version: v1.21.6-rancher-1-2
• Cluster Type : Single node Custom Cluster with 1 etcd, 1cp, 1 worker nodes. OS: RHEL-8.4 SElinux on, dockerversion: 20.10.7

Steps to reproduce without PSP :

1. Create rancher server
2. Create a single node custom cluster with all the roles running RHEL-8.4 with PSP enabled and set to restricted by default (Run systemctl disable nm-cloud-setup.service nm-cloud-setup.timer on the RHEL host before adding it to the cluster).
3. Set PSP on default project to unrestricted.
4. Install istio version 100.0.2+up1.10.4,
5. Create a namespace in default project and enable istio auto injection.
6. Deploy demo book app

Result:

• The book demo app is deployed successfully.

Steps to reproduce with PSP :

1. Create rancher server
2. Create a single node custom cluster with all the roles running RHEL-8.4 with PSP enabled and set to restricted by default. (Run systemctl disable nm-cloud-setup.service nm-cloud-setup.timer on the RHEL host before adding it to the cluster).
3. Set PSP on default project to restricted.
4. Install istio version 100.0.2+up1.10.4,
5. Create a namespace in default project and enable istio auto injection.
6. Deploy demo book app

Result:

• The book demo app is deployed.

The app is also getting deployed successfully on rancher 2.6-head with istio 100.1.0+up1.11.4 following the same steps mentioned above.

cc: @samkulkarni20 @deniseschannon

[vivek-shilimkar]

Validated the issue again. Validation indicates the results are inconsistent.

Setup Configuration:

Rancher server: v2.6.2

• Installation option : Docker
• Provider: ec2 single node.

Information about the Cluster:

• Kubernetes version: v1.21.7-rancher-1-1
• Cluster Type : Single node Custom Cluster with etcd, cp, and worker nodes.
• OS: RHEL-8.4 SElinux on, dockerversion: 20.10.7

Steps to reproduce without PSP :

1. Create rancher server
2. Create a single node custom cluster with all the roles running RHEL-8.4 with PSP enabled and set to restricted by default (Run systemctl disable nm-cloud-setup.service nm-cloud-setup.timer on the RHEL host before adding it to the cluster).
3. Set PSP on default project to unrestricted.
4. Install istio version 100.0.2+up1.10.4,
5. Create a namespace in default project and enable istio auto injection.
6. Deploy demo book app

Result:
istio-init container is in crashloopBackoff with the same error as Anupama observed.

[vivek-shilimkar]

Validated this issue on SELinux RHEL 8.4 docker-20.10.7

Steps followed to validate the issue
On rancher v2.6-head

1. Create a custom rke1 SELinux RHEL enabled cluster from the dashboard with 2 worker nodes, 1 etcd and 1 cp. with PSP enabled and set to restricted by default.
2. Once the clusters are up and active
3. Set PSP on default project to unrestricted.
4. Deploy monitoring and Istio v2 from the dashboard UI --> Apps & Marketplace--> Istio. Version: 100.1.0+up1.11.4
5. Istio deploys successfully
6. Create a namespace with Istio sidecar injection enabled.
7. Deploy the book demo app. Demo book app deploys successfully.

However, the app was not accessible on worker-node-ip:31380/productpage

For validation with PSP on.

1. Remove bookinfo app deployed above
2. Upgrade rancher-istio chart to add CNI component.

Istio upgrade failed with following errors.
Back-off restarting failed container
Container image "rancher/mirrored-istio-install-cni:1.11.4" already present on machine

Could not proceed to deploy demo book app.

[samkulkarni20]

I tested this with v2.6.2, a single all role node custom cluster running RHEL 8.4 (ami-0277fbe7afa8a33a6) and Kubernetes v1.21.6-rancher1-1 and it worked as expected with rancher-istio:100.0.2+up1.10.4 and SELinux enforcing, my steps:

Without PSP:

1. Provision a Rancher v2.6.2 instance and a single all role node customer cluster running RHEL 8.4, with PSP enabled and set to restricted by default (be sure to run systemctl disable nm-cloud-setup.service nm-cloud-setup.timer on the RHEL host before adding it to the cluster per Pod communication not working on RHEL 8.4 on cloud providers rke2#1053 (comment)).
2. Set PSP on default Project to unrestricted
3. Install rancher-istio:100.0.2+up1.10.4 with Pilot and Ingress Gateway
4. Deploy bookinfo app successfully https://istio.io/v1.10/docs/examples/bookinfo/#start-the-application-services

With PSP:

1. Remove bookinfo app deployed above
2. Upgrade rancher-istio chart to add CNI component
3. Set PSP on default Project to restricted
4. Deploy bookinfo app successfully https://istio.io/v1.10/docs/examples/bookinfo/#start-the-application-services

For the 1st scenario above, Istio installation works consistently using the latest Rancher-Istio chart (upstream v1.11.4).

The 2nd scenario fails though, since istio install/upgrade with CNI enabled, fails with below error:

Error: failed to create UDS listener: failed to listen on unix socket "/var/run/istio-cni/log.sock": listen unix /var/run/istio-cni/log.sock: bind: permission denied

The above issue is caused due to the hostPath volume mounted on the CNI daemonset at location /var/run/istio-cni. This hostPath directory is not having the correct SELinux context type label set on it during it's creation.

[root@ip-172-31-47-40 ec2-user]# ls -Z /var/run/
system_u:object_r:container_var_run_t:s0 istio-cni

As you see, it has type label container_var_run_t. The required type label is container_file_t, then CNI pod can to write to it.

A possible solution is to run below command on each cluster node before creating the cluster.

mkdir -p /var/run/istio-cni && semanage fcontext -a -t container_file_t /var/run/istio-cni && restorecon -v /var/run/istio-cni

@brendarearden @axeal
We can add this step in the chart README till an istio release supports to disable UDS logging feature. Shall we go ahead with this?

Note: Istio has a change in their master branch which enables setting env variable LOG_UDS_ADDRESS to emptyString. With that the UDS logging feature is disabled. Although the change isn't released in any of the release versions yet. Without this fix, CNI installer ignores the empty env vars.

CC: @doflamingo721

[brendarearden]

@samkulkarni20 we could add the temporary work around to https://github.com/rancher/charts/blob/dev-v2.6/packages/rancher-istio/rancher-istio/charts/app-readme.md in the Known Issues section. I would also think it needs to be added as a release note with the next rancher release.

[vivek-shilimkar]

Setup Configuration:

Rancher server v2.6-head:

• Installation option : Docker
• Provider: ec2 single node.

Information about the Cluster:

• Kubernetes version: v1.21.9-rancher-1-1
• Cluster Type : Single node Custom Cluster with 1 etcd, 1cp, 1 worker nodes. OS: RHEL-8.4 SElinux on, dockerversion: 20.10.7

Steps to reproduce without PSP :

1. Create rancher server
2. Create a single node custom cluster with all the roles running RHEL-8.4 with PSP enabled and set to restricted by default
3. Run following commands on the instance before adding it to the cluster

• systemctl disable nm-cloud-setup.service nm-cloud-setup.timer

• mkdir -p /var/run/istio-cni && semanage fcontext -a -t container_file_t /var/run/istio-cni && restorecon -v /var/run/istio-cni

4. Set PSP on default project to unrestricted.
5. Install istio version 100.1.1+up1.11.4
6. Create a namespace in default project and enable istio auto injection.
7. Deploy demo book app

Result:

• The book demo app is deployed successfully.

Steps to reproduce with PSP :

1. Create rancher server
2. Create a single node custom cluster with all the roles running RHEL-8.4 with PSP enabled and set to restricted by default. (Run systemctl disable nm-cloud-setup.service nm-cloud-setup.timer on the RHEL host before adding it to the cluster).
3. Set PSP on default project to restricted.
4. Install istio version 100.1.1+up1.11.4
5. Create a namespace in default project and enable istio auto injection.
6. Deploy demo book app

Result:
- The book demo app is deployed and accessible.

The app is also getting deployed successfully on rancher 2.6-head with istio 100.1.1+up1.11.4 following the same steps mentioned above.

[vivek-shilimkar]

Based on above validation, the issue is not active. Hence, closing the issue.

[themowski]

Our team encountered a similar issue that had the same core error message -- iptables-restore v1.6.1: iptables-restore: unable to initialize table 'nat'. Posting some info here in case anyone finds this issue while searching for this error message.

We encountered the error message when deploying Kubeflow v1.3.1 and v1.4.1 through its manifests deployment method, which injects Istio sidecars. Our target environment is a downstream Kubernetes v1.21.9-rancher1-1 cluster that is deployed by Rancher v2.6.3-patch1. Each node in our cluster has AlmaLinux 8.5 (one of the CentOS 8 derivatives) as its OS. SELinux and firewalld are both disabled on our nodes.

We found a solution in the comments on related issue istio/istio#23009. The answer was to ensure that a set of specific kernel modules is loaded. For more details, see my reply on the other issue: istio/istio#23009 (comment)