Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Istio support in rke2 #504

Closed
galal-hussein opened this issue Oct 26, 2020 · 12 comments
Closed

Istio support in rke2 #504

galal-hussein opened this issue Oct 26, 2020 · 12 comments
Assignees
Labels
kind/enhancement An improvement to existing functionality

Comments

@galal-hussein
Copy link
Contributor

Istio needs to work on any supported downstream cluster in Rancher including K3S and RKE2

@galal-hussein galal-hussein added kind/enhancement An improvement to existing functionality [zube]: Working labels Oct 26, 2020
@galal-hussein galal-hussein self-assigned this Oct 26, 2020
@davidnuzik davidnuzik added this to the December milestone Oct 26, 2020
@galal-hussein
Copy link
Contributor Author

I was able to get Istio working on Ubuntu, however in centos envoy ingress/egress pods are failing to start due to an issue with selinux k3s-io/k3s#2240

type=AVC msg=audit(1603818489.340:8261): avc:  denied  { write } for  pid=29290 comm="envoy" name="/" dev="tmpfs" ino=87633 scontext=system_u:system_r:container_t:s0:c538,c544 tcontext=system_u:object_r:container_runtime_tmpfs_t:s0 tclass=dir permissive=0
type=SYSCALL msg=audit(1603818489.340:8261): arch=c000003e syscall=257 success=no exit=-13 a0=ffffff9c a1=7ffd67b374d0 a2=a00c2 a3=180 items=0 ppid=29259 pid=29290 auid=4294967295 uid=1337 gid=1337 euid=1337 suid=1337 fsuid=1337 egid=1337 sgid=1337 fsgid=1337 tty=(none) ses=4294967295 comm="envoy" exe="/usr/local/bin/envoy" subj=system_u:system_r:container_t:s0:c538,c544 key=(null)

@davidnuzik davidnuzik modified the milestones: December, v1.19.4+rke2r2 Nov 6, 2020
@davidnuzik
Copy link
Contributor

@galal-hussein this issue is still in working state. It's indeed still being worked on? Any updates to report?

@cjellick
Copy link
Contributor

Envoy selinux bug fixed. can retest

@rancher-max
Copy link
Contributor

Used rke2 v1.18.12+rke2r1 and k3s v1.19.4+k3s1 for testing. This is working on ubuntu and centos and rhel systems in k3s, and on ubuntu systems in rke2.

It is working on rhel and centos systems in rke2 that DO NOT have selinux enforcing enabled, however, sidecar injection fails when selinux enforcing IS enabled. I believe it is related to istio/istio#19380 (comment) and probably shouldn't block this from being closed, but @cjellick @davidnuzik and @galal-hussein can advise.

See logs taken from the sidecar of one of the pods (bookinfo app, istio's recommended approach):

$ k logs -n istio-app-ns-95540 productpage-v1-67bfc95c4b-qbjrq -c istio-init -f
iptables-restore: line 22 failed
Environment:
------------
ENVOY_PORT=
INBOUND_CAPTURE_PORT=
ISTIO_INBOUND_INTERCEPTION_MODE=
ISTIO_INBOUND_TPROXY_MARK=
ISTIO_INBOUND_TPROXY_ROUTE_TABLE=
ISTIO_INBOUND_PORTS=
ISTIO_LOCAL_EXCLUDE_PORTS=
ISTIO_SERVICE_CIDR=
ISTIO_SERVICE_EXCLUDE_CIDR=

Variables:
----------
PROXY_PORT=15001
PROXY_INBOUND_CAPTURE_PORT=15006
PROXY_UID=1337
PROXY_GID=1337
INBOUND_INTERCEPTION_MODE=REDIRECT
INBOUND_TPROXY_MARK=1337
INBOUND_TPROXY_ROUTE_TABLE=133
INBOUND_PORTS_INCLUDE=*
INBOUND_PORTS_EXCLUDE=15020
OUTBOUND_IP_RANGES_INCLUDE=*
OUTBOUND_IP_RANGES_EXCLUDE=
OUTBOUND_PORTS_EXCLUDE=
KUBEVIRT_INTERFACES=
ENABLE_INBOUND_IPV6=false

Writing following contents to rules file:  /tmp/iptables-rules-1605923201250683860.txt049798106
* nat
-N ISTIO_REDIRECT
-N ISTIO_IN_REDIRECT
-N ISTIO_INBOUND
-N ISTIO_OUTPUT
-A ISTIO_REDIRECT -p tcp -j REDIRECT --to-ports 15001
-A ISTIO_IN_REDIRECT -p tcp -j REDIRECT --to-ports 15006
-A PREROUTING -p tcp -j ISTIO_INBOUND
-A ISTIO_INBOUND -p tcp --dport 22 -j RETURN
-A ISTIO_INBOUND -p tcp --dport 15020 -j RETURN
-A ISTIO_INBOUND -p tcp -j ISTIO_IN_REDIRECT
-A OUTPUT -p tcp -j ISTIO_OUTPUT
-A ISTIO_OUTPUT -o lo -s 127.0.0.6/32 -j RETURN
-A ISTIO_OUTPUT -o lo ! -d 127.0.0.1/32 -m owner --uid-owner 1337 -j ISTIO_IN_REDIRECT
-A ISTIO_OUTPUT -o lo -m owner ! --uid-owner 1337 -j RETURN
-A ISTIO_OUTPUT -m owner --uid-owner 1337 -j RETURN
-A ISTIO_OUTPUT -o lo ! -d 127.0.0.1/32 -m owner --gid-owner 1337 -j ISTIO_IN_REDIRECT
-A ISTIO_OUTPUT -o lo -m owner ! --gid-owner 1337 -j RETURN
-A ISTIO_OUTPUT -m owner --gid-owner 1337 -j RETURN
-A ISTIO_OUTPUT -d 127.0.0.1/32 -j RETURN
-A ISTIO_OUTPUT -j ISTIO_REDIRECT
COMMIT

iptables-restore --noflush /tmp/iptables-rules-1605923201250683860.txt049798106
iptables-save 
# Generated by iptables-save v1.6.1 on Sat Nov 21 01:46:41 2020
*nat
:PREROUTING ACCEPT [0:0]
:INPUT ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:POSTROUTING ACCEPT [0:0]
COMMIT
# Completed on Sat Nov 21 01:46:41 2020
panic: exit status 1

goroutine 1 [running]:
istio.io/istio/tools/istio-iptables/pkg/dependencies.(*RealDependencies).RunOrFail(0xd819c0, 0x9739cb, 0x10, 0xc000086b80, 0x2, 0x2)
	istio.io/istio@/tools/istio-iptables/pkg/dependencies/implementation.go:44 +0x96
istio.io/istio/tools/istio-iptables/pkg/cmd.(*IptablesConfigurator).executeIptablesRestoreCommand(0xc0000f7d30, 0x7f1f0fa36601, 0x0, 0x0)
	istio.io/istio@/tools/istio-iptables/pkg/cmd/run.go:484 +0x3aa
istio.io/istio/tools/istio-iptables/pkg/cmd.(*IptablesConfigurator).executeCommands(0xc0000f7d30)
	istio.io/istio@/tools/istio-iptables/pkg/cmd/run.go:491 +0x45
istio.io/istio/tools/istio-iptables/pkg/cmd.(*IptablesConfigurator).run(0xc0000f7d30)
	istio.io/istio@/tools/istio-iptables/pkg/cmd/run.go:438 +0x2507
istio.io/istio/tools/istio-iptables/pkg/cmd.glob..func1(0xd5c740, 0xc0000b0900, 0x0, 0x10)
	istio.io/istio@/tools/istio-iptables/pkg/cmd/root.go:56 +0x14e
github.com/spf13/cobra.(*Command).execute(0xd5c740, 0xc000098010, 0x10, 0x11, 0xd5c740, 0xc000098010)
	github.com/spf13/cobra@v0.0.5/command.go:830 +0x2aa
github.com/spf13/cobra.(*Command).ExecuteC(0xd5c740, 0x40574f, 0xc00006c058, 0x0)
	github.com/spf13/cobra@v0.0.5/command.go:914 +0x2fb
github.com/spf13/cobra.(*Command).Execute(...)
	github.com/spf13/cobra@v0.0.5/command.go:864
istio.io/istio/tools/istio-iptables/pkg/cmd.Execute()
	istio.io/istio@/tools/istio-iptables/pkg/cmd/root.go:284 +0x2d
main.main()
	istio.io/istio@/tools/istio-iptables/main.go:22 +0x20

Again, this is not the case when selinux is not enforcing on the rke2 nodes.

@dweomer
Copy link
Contributor

dweomer commented Nov 23, 2020

When working on SELinux policy for RKE2 and k3s we ran into something similar with the klipper-lb image/container because it wants to load a kernel module if it is missing, which wont work under SELinux unless the container is privileged. My recommendation is to pre-load the kernel module so that the istio sidecar doesn't have to.

The best way to pinpoint which kernel module(s) (likley one of nf_tables or x_tables) to pre-load is to install istio on k3s and do an lsmod | sort and then do the same for an rke2 node. The missing module lines in the diff will likley be narrowed down to what must needs be pre-loaded.

@rancher-max
Copy link
Contributor

Below I list the different values from the result of lsmod | sort for k3s and rke2 in an selinux enforcing rhel system, after attempting to run istio in both:

k3s:

ip_set                 49152  2 ip_set_hash_ip,xt_set
ip_set_hash_ip         36864  2
iptable_nat            16384  3
ipt_MASQUERADE         16384  6
ipt_REJECT             16384  0
nf_conntrack          155648  11 xt_conntrack,nf_conntrack_ipv6,nf_conntrack_ipv4,nf_nat,nf_nat_ipv6,ipt_MASQUERADE,nf_nat_ipv4,xt_nat,nf_conntrack_netlink,ip_vs,xt_REDIRECT
nf_conntrack_ipv4      16384  58
nf_conntrack_ipv6      20480  1
nf_defrag_ipv6         20480  2 nf_conntrack_ipv6,ip_vs
nf_nat                 36864  4 nf_nat_ipv6,nf_nat_ipv4,xt_nat,xt_REDIRECT
nf_nat_ipv6            16384  1 nft_chain_nat_ipv6
nf_tables             151552  877 nft_chain_route_ipv4,nft_compat,nft_chain_nat_ipv6,nft_chain_nat_ipv4,nft_counter
nft_chain_nat_ipv6     16384  4
nft_compat             20480  391
nft_counter            16384  229
overlay               126976  35
xt_addrtype            16384  3
xt_comment             16384  207
xt_conntrack           16384  8
xt_mark                16384  8
xt_multiport           16384  1
xt_nat                 16384  39
xt_owner               16384  12
xt_REDIRECT            16384  4
xt_statistic           16384  2

rke2:

ip_set                 49152  3 ip_set_hash_ip,xt_set,ip_set_hash_net
ip_set_hash_ip         36864  1
ip_set_hash_net        36864  2
iptable_nat            16384  0
ipt_MASQUERADE         16384  4
ipt_REJECT             16384  4
ipt_rpfilter           16384  1
nf_conntrack          155648  8 xt_conntrack,nf_conntrack_ipv4,nf_nat,ipt_MASQUERADE,nf_nat_ipv4,xt_nat,nf_conntrack_netlink,ip_vs
nf_conntrack_ipv4      16384  72
nf_defrag_ipv6         20480  1 ip_vs
nf_nat                 36864  2 nf_nat_ipv4,xt_nat
nf_tables             151552  1090 nft_chain_route_ipv4,nft_compat,nft_chain_nat_ipv4,nft_counter
nft_compat             20480  869
nft_counter            16384  465
overlay               126976  38
xt_addrtype            16384  1
xt_comment             16384  529
xt_conntrack           16384  41
xt_mark                16384  116
xt_multiport           16384  57
xt_nat                 16384  26

@rancher-max
Copy link
Contributor

Attempted to add ExecStartPre=-/sbin/modprobe nf_nat_ipv6 into the unit files of server and worker nodes: /lib/systemd/system/rke2-server.service and /lib/systemd/system/rke2-agent.service and restarted rke2 on the nodes. Then tried to install istio again and run bookinfo app with sidecar injection, and it failed again. This time I noticed the ingressgateway istio pod in the istio-system namespace failed to startup. The error is the same as in istio/istio#28045, which mentions:

This looks like you may have some lower level networking issues in your cluster possibly?

I believe we may need to load additional kernel modules? @dweomer what do you think?

@rancher-max
Copy link
Contributor

Validated able to get this working for RKE2 in Selinux enforcing systems with rancher/rancher#27377 (comment). NOTE: this uses Istio's CNI, which is in alpha state. They often suggest use of that in their main repo for issues similar to this one, so it looks like they are actively trying to get that beyond alpha at least. We should have this available in Rancher in the upcoming 2.5.4 release as well so everything should integrate with each other nicely now.

There are 2 important things I would like to note here as well:

  1. This required editing the cni-node daemonset post-install to set the container's securitycontext to privileged. This may not always be desired for all users, and to determine potential other methods that don't require this, we have an upstream issue with istio.
  2. I was not able to determine which exact kernel modules could be preloaded to avoid the need for using cni and privileged containers for istio, but there is likely some combination that would work. These would potentially be needed for other networking applications similar to istio that users might want to run in their cluster. At this time, I'll leave that up to users to find and report or submit issues relating to those applications, as that is beyond the scope of this issue.

@joshrwolf
Copy link
Contributor

was able to verify that preloading the missing kernel modules (xt_REDIRECT, xt_owner, and xt_statistic on my rhel8.3 test box) allowed istio-init to successfully modify iptables rules with selinux enforcing.

will modify the documentation as another potential (non istio-cni) workaround

@tallaxes
Copy link

I can confirm that preloading the above modules works

@BrandonALXEllisSS
Copy link

was able to verify that preloading the missing kernel modules (xt_REDIRECT, xt_owner, and xt_statistic on my rhel8.3 test box) allowed istio-init to successfully modify iptables rules with selinux enforcing.

will modify the documentation as another potential (non istio-cni) workaround

Any progress on documenting how to preload the kernel modules?

@brandond
Copy link
Contributor

brandond commented Sep 7, 2021

@BrandonALXEllisSS put them in etc/modules.conf?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind/enhancement An improvement to existing functionality
Projects
None yet
Development

No branches or pull requests

9 participants