v1.22.0 ambient inject iptables rules in pod failed for CentOS 7

[escoffier]

Is this the right place to submit this?

• This is not a security vulnerability or a crashing bug
• This is not a question about how to use Istio

Bug Description

When adding a pod into ambient mesh, istio-cni-node log the follwing err:

24-05-14T09:44:22.086970Z     info    ambient in pod mode - adding pod ambient-demo/sleep-75bbc86479-vc6ss to ztunnel 
2024-05-14T09:44:22.087458Z     info    iptables        Running iptables-legacy-restore with the following input:
\* mangle
-N ISTIO_PRERT
-N ISTIO_OUTPUT
-A PREROUTING -j ISTIO_PRERT
-A OUTPUT -j ISTIO_OUTPUT
-A ISTIO_PRERT -m mark --mark 0x539/0xfff -j CONNMARK --set-xmark 0x111/0xfff
-A ISTIO_PRERT -s 169.254.7.127 -p tcp -m tcp -j ACCEPT
-A ISTIO_PRERT ! -d 127.0.0.1/32 -p tcp -i lo -j ACCEPT
-A ISTIO_PRERT -p tcp -m tcp --dport 15008 -m mark ! --mark 0x539/0xfff -j TPROXY --on-port 15008 --tproxy-mark 0x111/0xfff
-A ISTIO_PRERT -p tcp -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A ISTIO_PRERT ! -d 127.0.0.1/32 -p tcp -m mark ! --mark 0x539/0xfff -j TPROXY --on-port 15006 --tproxy-mark 0x111/0xfff
-A ISTIO_OUTPUT -m connmark --mark 0x111/0xfff -j CONNMARK --restore-mark --nfmask 0xffffffff --ctmask 0xffffffff
COMMIT
\* nat
-N ISTIO_OUTPUT
-A OUTPUT -j ISTIO_OUTPUT
-A ISTIO_OUTPUT -d 169.254.7.127 -p tcp -m tcp -j ACCEPT
-A ISTIO_OUTPUT -p tcp -m mark --mark 0x111/0xfff -j ACCEPT
-A ISTIO_OUTPUT ! -d 127.0.0.1/32 -o lo -j ACCEPT
-A ISTIO_OUTPUT ! -d 127.0.0.1/32 -p tcp -m mark ! --mark 0x539/0xfff -j REDIRECT --to-ports 15001
COMMIT
2024-05-14T09:44:22.087510Z     info    Running command (with wait lock): iptables-legacy-restore --noflush -v --wait=30
2024-05-14T09:44:22.090028Z     error   Command error output: Warning: Extension CONNMARK is not supported, missing kernel module?

node os version:
CentOS Linux 7 (Core) 3.10.0-1160.el7.x86_64

node iptables version:

iptables --version
iptables v1.4.21

connmark module is loaded

$ lsmod | grep connmark
xt_connmark            12755  103 
nf_conntrack          139264  12 ip_vs,nf_nat,xt_state,nf_nat_ipv4,nf_nat_ipv6,xt_conntrack,nf_nat_masquerade_ipv4,nf_nat_masquerade_ipv6,nf_conntrack_netlink,xt_connmark,nf_conntrack_ipv4,nf_conntrack_ipv6

Version

$ istioctl version
client version: 1.22.0
control plane version: 1.22.0
data plane version: 1.22.0 (5 proxies)

$ kubectl version 
Client Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.0", GitCommit:"ab69524f795c42094a6630298ff53f3c3ebab7f4", GitTreeState:"clean", BuildDate:"2021-12-07T18:16:20Z", GoVersion:"go1.17.3", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.0", GitCommit:"ab69524f795c42094a6630298ff53f3c3ebab7f4", GitTreeState:"clean", BuildDate:"2021-12-07T18:09:57Z", GoVersion:"go1.17.3", Compiler:"gc", Platform:"linux/amd64"}

Additional Information

No response

[howardjohn]

I think you need xt_connmark https://istio.io/latest/docs/setup/platform-setup/prerequisites/#kernel-module-requirements-on-cluster-nodes. Note that doc will need to be updated to say it's needed for ambient as well

[howardjohn]

oops, missed where you said you have that

[howardjohn]

cc @bleggett

[bleggett]

This is an extremely old version of linux/iptables (1.4 is from 2012), and centos7 is ~1 month away from EOL.

It might be because the container iptables binary and the host kernel are just too far out of skew.

I'll see if I can repro locally with Centos7

[linsun]

Hi @escoffier - given centOS 7 is going to be end of life soon, could you reproduce it on a newer centOS version? Also, do you have this prob for sidecars?

[bleggett]

I can't repro this in a centos7 Docker image, with iptables 1.4.21:

[root@d48fd0199421 /]# iptables -t nat -N ISTIO_OUTPUT
[root@d48fd0199421 /]# iptables -t nat -A ISTIO_OUTPUT -p tcp -m mark --mark 0x111/0xfff -j ACCEPT
[root@d48fd0199421 /]# iptables -t nat -A ISTIO_OUTPUT ! -d 127.0.0.1/32 -p tcp -m mark ! --mark 0x539/0xfff -j REDIRECT --to-ports 15001
[root@d48fd0199421 /]# iptables --version
iptables v1.4.21

Naturally, since it's a docker image, the underlying kernel is much newer (6.8) - so I do think this is likely a kernel problem, not an iptables problem, or an istio problem.

Since 3.10 has been EOL'd for years (~2017), and CentOS has been keeping it alive in CentOS 7 by backporting fixes, it's entirely possible they didn't backport something we need, it's a very nonstandard kernel by this point.

@escoffier unless you can repro this with a kernel/OS that's not on the verge of being EOL'd, we probably aren't going to fix this.

[escoffier]

I can't repro this in a centos7 Docker image, with iptables 1.4.21:

[root@d48fd0199421 /]# iptables -t nat -N ISTIO_OUTPUT
[root@d48fd0199421 /]# iptables -t nat -A ISTIO_OUTPUT -p tcp -m mark --mark 0x111/0xfff -j ACCEPT
[root@d48fd0199421 /]# iptables -t nat -A ISTIO_OUTPUT ! -d 127.0.0.1/32 -p tcp -m mark ! --mark 0x539/0xfff -j REDIRECT --to-ports 15001
[root@d48fd0199421 /]# iptables --version
iptables v1.4.21

Naturally, since it's a docker image, the underlying kernel is much newer (6.8) - so I do think this is likely a kernel problem, not an iptables problem, or an istio problem.

Since 3.10 has been EOL'd for years (~2017), and CentOS has been keeping it alive in CentOS 7 by backporting fixes, it's entirely possible they didn't backport something we need, it's a very nonstandard kernel by this point.

@escoffier unless you can repro this with a kernel/OS that's not on the verge of being EOL'd, we probably aren't going to fix this.

thanks， i‘am trying to upgrade kernel.

[escoffier]

Hi @escoffier - given centOS 7 is going to be end of life soon, could you reproduce it on a newer centOS version? Also, do you have this prob for sidecars?

Thanks for responding. it works fine on newer centOS version and for sidecar.

[linsun]

Thanks @escoffier! Seems we should document this as a limitation for centOS 7 under https://istio.io/latest/docs/ambient/install/platform-prerequisites/? PR would be welcome!

[bleggett]

Thanks @escoffier! Seems we should document this as a limitation for centOS 7 under https://istio.io/latest/docs/ambient/install/platform-prerequisites/? PR would be welcome!

No, we need to update https://istio.io/latest/docs/setup/platform-setup/prerequisites/#kernel-module-requirements-on-cluster-nodes as @howardjohn mentioned.

Also, this isn't ambient specific, really. I can do a pass on that doc.

Edit: istio/istio.io#15109

[bleggett]

Doc PR: istio/istio.io#15121

[bleggett]

Closing this as WONTFIX for centos7 specifically - doc PR is merged.