Istio support in rke2

[galal-hussein]

Istio needs to work on any supported downstream cluster in Rancher including K3S and RKE2

[galal-hussein]

I was able to get Istio working on Ubuntu, however in centos envoy ingress/egress pods are failing to start due to an issue with selinux k3s-io/k3s#2240

type=AVC msg=audit(1603818489.340:8261): avc:  denied  { write } for  pid=29290 comm="envoy" name="/" dev="tmpfs" ino=87633 scontext=system_u:system_r:container_t:s0:c538,c544 tcontext=system_u:object_r:container_runtime_tmpfs_t:s0 tclass=dir permissive=0
type=SYSCALL msg=audit(1603818489.340:8261): arch=c000003e syscall=257 success=no exit=-13 a0=ffffff9c a1=7ffd67b374d0 a2=a00c2 a3=180 items=0 ppid=29259 pid=29290 auid=4294967295 uid=1337 gid=1337 euid=1337 suid=1337 fsuid=1337 egid=1337 sgid=1337 fsgid=1337 tty=(none) ses=4294967295 comm="envoy" exe="/usr/local/bin/envoy" subj=system_u:system_r:container_t:s0:c538,c544 key=(null)

[davidnuzik]

@galal-hussein this issue is still in working state. It's indeed still being worked on? Any updates to report?

[cjellick]

Envoy selinux bug fixed. can retest

[rancher-max]

Used rke2 v1.18.12+rke2r1 and k3s v1.19.4+k3s1 for testing. This is working on ubuntu and centos and rhel systems in k3s, and on ubuntu systems in rke2.

It is working on rhel and centos systems in rke2 that DO NOT have selinux enforcing enabled, however, sidecar injection fails when selinux enforcing IS enabled. I believe it is related to istio/istio#19380 (comment) and probably shouldn't block this from being closed, but @cjellick @davidnuzik and @galal-hussein can advise.

See logs taken from the sidecar of one of the pods (bookinfo app, istio's recommended approach):

$ k logs -n istio-app-ns-95540 productpage-v1-67bfc95c4b-qbjrq -c istio-init -f
iptables-restore: line 22 failed
Environment:
------------
ENVOY_PORT=
INBOUND_CAPTURE_PORT=
ISTIO_INBOUND_INTERCEPTION_MODE=
ISTIO_INBOUND_TPROXY_MARK=
ISTIO_INBOUND_TPROXY_ROUTE_TABLE=
ISTIO_INBOUND_PORTS=
ISTIO_LOCAL_EXCLUDE_PORTS=
ISTIO_SERVICE_CIDR=
ISTIO_SERVICE_EXCLUDE_CIDR=

Variables:
----------
PROXY_PORT=15001
PROXY_INBOUND_CAPTURE_PORT=15006
PROXY_UID=1337
PROXY_GID=1337
INBOUND_INTERCEPTION_MODE=REDIRECT
INBOUND_TPROXY_MARK=1337
INBOUND_TPROXY_ROUTE_TABLE=133
INBOUND_PORTS_INCLUDE=*
INBOUND_PORTS_EXCLUDE=15020
OUTBOUND_IP_RANGES_INCLUDE=*
OUTBOUND_IP_RANGES_EXCLUDE=
OUTBOUND_PORTS_EXCLUDE=
KUBEVIRT_INTERFACES=
ENABLE_INBOUND_IPV6=false

Writing following contents to rules file:  /tmp/iptables-rules-1605923201250683860.txt049798106
* nat
-N ISTIO_REDIRECT
-N ISTIO_IN_REDIRECT
-N ISTIO_INBOUND
-N ISTIO_OUTPUT
-A ISTIO_REDIRECT -p tcp -j REDIRECT --to-ports 15001
-A ISTIO_IN_REDIRECT -p tcp -j REDIRECT --to-ports 15006
-A PREROUTING -p tcp -j ISTIO_INBOUND
-A ISTIO_INBOUND -p tcp --dport 22 -j RETURN
-A ISTIO_INBOUND -p tcp --dport 15020 -j RETURN
-A ISTIO_INBOUND -p tcp -j ISTIO_IN_REDIRECT
-A OUTPUT -p tcp -j ISTIO_OUTPUT
-A ISTIO_OUTPUT -o lo -s 127.0.0.6/32 -j RETURN
-A ISTIO_OUTPUT -o lo ! -d 127.0.0.1/32 -m owner --uid-owner 1337 -j ISTIO_IN_REDIRECT
-A ISTIO_OUTPUT -o lo -m owner ! --uid-owner 1337 -j RETURN
-A ISTIO_OUTPUT -m owner --uid-owner 1337 -j RETURN
-A ISTIO_OUTPUT -o lo ! -d 127.0.0.1/32 -m owner --gid-owner 1337 -j ISTIO_IN_REDIRECT
-A ISTIO_OUTPUT -o lo -m owner ! --gid-owner 1337 -j RETURN
-A ISTIO_OUTPUT -m owner --gid-owner 1337 -j RETURN
-A ISTIO_OUTPUT -d 127.0.0.1/32 -j RETURN
-A ISTIO_OUTPUT -j ISTIO_REDIRECT
COMMIT

iptables-restore --noflush /tmp/iptables-rules-1605923201250683860.txt049798106
iptables-save 
# Generated by iptables-save v1.6.1 on Sat Nov 21 01:46:41 2020
*nat
:PREROUTING ACCEPT [0:0]
:INPUT ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:POSTROUTING ACCEPT [0:0]
COMMIT
# Completed on Sat Nov 21 01:46:41 2020
panic: exit status 1

goroutine 1 [running]:
istio.io/istio/tools/istio-iptables/pkg/dependencies.(*RealDependencies).RunOrFail(0xd819c0, 0x9739cb, 0x10, 0xc000086b80, 0x2, 0x2)
	istio.io/istio@/tools/istio-iptables/pkg/dependencies/implementation.go:44 +0x96
istio.io/istio/tools/istio-iptables/pkg/cmd.(*IptablesConfigurator).executeIptablesRestoreCommand(0xc0000f7d30, 0x7f1f0fa36601, 0x0, 0x0)
	istio.io/istio@/tools/istio-iptables/pkg/cmd/run.go:484 +0x3aa
istio.io/istio/tools/istio-iptables/pkg/cmd.(*IptablesConfigurator).executeCommands(0xc0000f7d30)
	istio.io/istio@/tools/istio-iptables/pkg/cmd/run.go:491 +0x45
istio.io/istio/tools/istio-iptables/pkg/cmd.(*IptablesConfigurator).run(0xc0000f7d30)
	istio.io/istio@/tools/istio-iptables/pkg/cmd/run.go:438 +0x2507
istio.io/istio/tools/istio-iptables/pkg/cmd.glob..func1(0xd5c740, 0xc0000b0900, 0x0, 0x10)
	istio.io/istio@/tools/istio-iptables/pkg/cmd/root.go:56 +0x14e
github.com/spf13/cobra.(*Command).execute(0xd5c740, 0xc000098010, 0x10, 0x11, 0xd5c740, 0xc000098010)
	github.com/spf13/cobra@v0.0.5/command.go:830 +0x2aa
github.com/spf13/cobra.(*Command).ExecuteC(0xd5c740, 0x40574f, 0xc00006c058, 0x0)
	github.com/spf13/cobra@v0.0.5/command.go:914 +0x2fb
github.com/spf13/cobra.(*Command).Execute(...)
	github.com/spf13/cobra@v0.0.5/command.go:864
istio.io/istio/tools/istio-iptables/pkg/cmd.Execute()
	istio.io/istio@/tools/istio-iptables/pkg/cmd/root.go:284 +0x2d
main.main()
	istio.io/istio@/tools/istio-iptables/main.go:22 +0x20

Again, this is not the case when selinux is not enforcing on the rke2 nodes.

[dweomer]

When working on SELinux policy for RKE2 and k3s we ran into something similar with the klipper-lb image/container because it wants to load a kernel module if it is missing, which wont work under SELinux unless the container is privileged. My recommendation is to pre-load the kernel module so that the istio sidecar doesn't have to.

The best way to pinpoint which kernel module(s) (likley one of nf_tables or x_tables) to pre-load is to install istio on k3s and do an lsmod | sort and then do the same for an rke2 node. The missing module lines in the diff will likley be narrowed down to what must needs be pre-loaded.

[rancher-max]

Below I list the different values from the result of lsmod | sort for k3s and rke2 in an selinux enforcing rhel system, after attempting to run istio in both:

k3s:

ip_set                 49152  2 ip_set_hash_ip,xt_set
ip_set_hash_ip         36864  2
iptable_nat            16384  3
ipt_MASQUERADE         16384  6
ipt_REJECT             16384  0
nf_conntrack          155648  11 xt_conntrack,nf_conntrack_ipv6,nf_conntrack_ipv4,nf_nat,nf_nat_ipv6,ipt_MASQUERADE,nf_nat_ipv4,xt_nat,nf_conntrack_netlink,ip_vs,xt_REDIRECT
nf_conntrack_ipv4      16384  58
nf_conntrack_ipv6      20480  1
nf_defrag_ipv6         20480  2 nf_conntrack_ipv6,ip_vs
nf_nat                 36864  4 nf_nat_ipv6,nf_nat_ipv4,xt_nat,xt_REDIRECT
nf_nat_ipv6            16384  1 nft_chain_nat_ipv6
nf_tables             151552  877 nft_chain_route_ipv4,nft_compat,nft_chain_nat_ipv6,nft_chain_nat_ipv4,nft_counter
nft_chain_nat_ipv6     16384  4
nft_compat             20480  391
nft_counter            16384  229
overlay               126976  35
xt_addrtype            16384  3
xt_comment             16384  207
xt_conntrack           16384  8
xt_mark                16384  8
xt_multiport           16384  1
xt_nat                 16384  39
xt_owner               16384  12
xt_REDIRECT            16384  4
xt_statistic           16384  2

rke2:

ip_set                 49152  3 ip_set_hash_ip,xt_set,ip_set_hash_net
ip_set_hash_ip         36864  1
ip_set_hash_net        36864  2
iptable_nat            16384  0
ipt_MASQUERADE         16384  4
ipt_REJECT             16384  4
ipt_rpfilter           16384  1
nf_conntrack          155648  8 xt_conntrack,nf_conntrack_ipv4,nf_nat,ipt_MASQUERADE,nf_nat_ipv4,xt_nat,nf_conntrack_netlink,ip_vs
nf_conntrack_ipv4      16384  72
nf_defrag_ipv6         20480  1 ip_vs
nf_nat                 36864  2 nf_nat_ipv4,xt_nat
nf_tables             151552  1090 nft_chain_route_ipv4,nft_compat,nft_chain_nat_ipv4,nft_counter
nft_compat             20480  869
nft_counter            16384  465
overlay               126976  38
xt_addrtype            16384  1
xt_comment             16384  529
xt_conntrack           16384  41
xt_mark                16384  116
xt_multiport           16384  57
xt_nat                 16384  26