Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

nftables / CentOS8 support #306

Closed
iptizer opened this issue Jun 28, 2020 · 3 comments
Closed

nftables / CentOS8 support #306

iptizer opened this issue Jun 28, 2020 · 3 comments

Comments

@iptizer
Copy link

iptizer commented Jun 28, 2020

I am not 100% sure how things are related between iptables and nftables, so I though I'll highlight my problem here, maybe there is a solution.

Setup

I am using Kubernetes with Calico on CentOS8. CentOS8 uses nftables. Felix (the Calico component for setting iptables on a host) is setup with backend=NFT. This works, but some things are not working 100% so I thought let's fix this using python.

My iptables command is linked to xtables:

[root@myserver myuser]# ll /usr/sbin/iptables
lrwxrwxrwx. 1 root root 17 Apr 24 13:51 /usr/sbin/iptables -> xtables-nft-multi

Problem

Listing chains with iptables -L -t nat and iptc.easy.dump_table('nat', ipv6=False) show different results:

>>> iptc.easy.dump_table('nat', ipv6=False)
{'PREROUTING': [{'comment': {'comment': 'kubernetes service portals'}, 'target': 'KUBE-SERVICES', 'counters': (810333, 46201389)}], 'INPUT': [], 'OUTPUT': [{'comment': {'comment': 'kubernetes service portals'}, 'target': 'KUBE-SERVICES', 'counters': (4088850, 246510358)}], 'POSTROUTING': [{'comment': {'comment': 'kubernetes postrouting rules'}, 'target': 'KUBE-POSTROUTING', 'counters': (4542742, 273781783)}], 'KUBE-FIREWALL': [{'target': 'KUBE-MARK-DROP', 'counters': (0, 0)}], 'KUBE-LOAD-BALANCER': [{'target': 'KUBE-MARK-MASQ', 'counters': (0, 0)}], 'KUBE-MARK-DROP': [], 'KUBE-MARK-MASQ': [{'target': {'MARK': {'set-xmark': '0x4000/0x4000'}}, 'counters': (0, 0)}], 'KUBE-NODE-PORT': [{'protocol': 'tcp', 'comment': {'comment': 'Kubernetes nodeport TCP port for masquerade purpose'}, 'set': {'match-set': ['KUBE-NODE-PORT-TCP', 'dst']}, 'target': 'KUBE-MARK-MASQ', 'counters': (0, 0)}], 'KUBE-POSTROUTING': [{'comment': {'comment': 'kubernetes service traffic requiring SNAT'}, 'mark': {'mark': '0x4000/0x4000'}, 'target': {'MASQUERADE': {'random-fully': ''}}, 'counters': (0, 0)}, {'comment': {'comment': 'Kubernetes endpoints dst ip:port, source ip for solving hairpin purpose'}, 'set': {'match-set': ['KUBE-LOOP-BACK', 'dst,dst,src']}, 'target': 'MASQUERADE', 'counters': (2, 120)}], 'KUBE-SERVICES': [{'src': '!10.233.64.0/18', 'comment': {'comment': 'Kubernetes service cluster ip + port for masquerade purpose'}, 'set': {'match-set': ['KUBE-CLUSTER-IP', 'dst,dst']}, 'target': 'KUBE-MARK-MASQ', 'counters': (0, 0)}, {'addrtype': {'dst-type': 'LOCAL'}, 'target': 'KUBE-NODE-PORT', 'counters': (53, 2948)}, {'set': {'match-set': ['KUBE-CLUSTER-IP', 'dst,dst']}, 'target': 'ACCEPT', 'counters': (11, 660)}]}
[root@ns3032787 moritz]# iptables -L -t nat | grep -i chain
Chain PREROUTING (policy ACCEPT)
Chain INPUT (policy ACCEPT)
Chain POSTROUTING (policy ACCEPT)
Chain OUTPUT (policy ACCEPT)
Chain KUBE-MARK-DROP (0 references)
Chain KUBE-MARK-MASQ (0 references)
Chain KUBE-POSTROUTING (1 references)
Chain KUBE-KUBELET-CANARY (0 references)
Chain cali-PREROUTING (1 references)
Chain cali-POSTROUTING (1 references)
Chain cali-OUTPUT (1 references)
Chain cali-fip-dnat (2 references)
Chain cali-fip-snat (1 references)
Chain cali-nat-outgoing (1 references)
Chain CNI-HOSTPORT-SETMARK (19 references)
Chain CNI-HOSTPORT-MASQ (1 references)
Chain CNI-HOSTPORT-DNAT (2 references)
Chain CNI-DN-ff88843368fae4d5cdfae (0 references)
Chain CNI-DN-f0f13904d22032106b1e7 (0 references)
Chain CNI-DN-442e7af5e08e6bf46621a (0 references)
Chain CNI-DN-e4582bd2b3ff7ed196a28 (0 references)
Chain CNI-DN-365d9a185cff3e5ea9379 (0 references)
Chain CNI-DN-90b43263af9d462e04f9b (0 references)
Chain CNI-DN-f54bc33afc414d6e64133 (1 references)
Chain CNI-DN-fc95996fce81df369bc47 (1 references)
Chain CNI-DN-ea59c359e9d718f7116dc (0 references)
Chain CNI-DN-9a7663babfc285ea01c15 (1 references)
Chain CNI-DN-fc387103aa44e3b189d09 (1 references)

So, some chains are showing up, some not.

Question: Is this even supposed to work? Or are the differences between iptables and nftables so large, that using a different python library is recommended?
@jllorente
Copy link
Collaborator

Hi @iptizer ,
can you please provide the output of # iptables -t nat -S ?

@iptizer
Copy link
Author

iptizer commented Jun 30, 2020

Hi @iptizer ,
can you please provide the output of # iptables -t nat -S ?

sure thing!

[root@myserver ~]# iptables -t nat -S
-P PREROUTING ACCEPT
-P INPUT ACCEPT
-P POSTROUTING ACCEPT
-P OUTPUT ACCEPT
-N KUBE-MARK-DROP
-N KUBE-MARK-MASQ
-N KUBE-POSTROUTING
-N KUBE-KUBELET-CANARY
-N cali-PREROUTING
-N cali-POSTROUTING
-N cali-OUTPUT
-N cali-fip-dnat
-N cali-fip-snat
-N cali-nat-outgoing
-N CNI-HOSTPORT-SETMARK
-N CNI-HOSTPORT-MASQ
-N CNI-HOSTPORT-DNAT
-N CNI-DN-ff88843368fae4d5cdfae
-N CNI-DN-f0f13904d22032106b1e7
-N CNI-DN-442e7af5e08e6bf46621a
-N CNI-DN-e4582bd2b3ff7ed196a28
-N CNI-DN-365d9a185cff3e5ea9379
-N CNI-DN-90b43263af9d462e04f9b
-N CNI-DN-f54bc33afc414d6e64133
-N CNI-DN-fc95996fce81df369bc47
-N CNI-DN-ea59c359e9d718f7116dc
-N CNI-DN-9a7663babfc285ea01c15
-N CNI-DN-fc387103aa44e3b189d09
-N CNI-DN-bcfdb00a9541b4df781a0
-N CNI-DN-2a1d84389734e54983abb
-N CNI-DN-38f4d538cd34246b81019
-A PREROUTING -m comment --comment "cali:6gwbT8clXdHdC1b1" -j cali-PREROUTING
-A PREROUTING -m addrtype --dst-type LOCAL -j CNI-HOSTPORT-DNAT
-A POSTROUTING -m comment --comment "cali:O3lYWMrLQYEMJtB5" -j cali-POSTROUTING
-A POSTROUTING -m comment --comment "CNI portfwd requiring masquerade" -j CNI-HOSTPORT-MASQ
-A POSTROUTING -m comment --comment "kubernetes postrouting rules" -j KUBE-POSTROUTING
-A OUTPUT -m comment --comment "cali:tVnHkvAo15HuiPy0" -j cali-OUTPUT
-A OUTPUT -m addrtype --dst-type LOCAL -j CNI-HOSTPORT-DNAT
-A KUBE-MARK-DROP -j MARK --set-xmark 0x8000/0x8000
-A KUBE-MARK-MASQ -j MARK --set-xmark 0x4000/0x4000
-A KUBE-POSTROUTING -m comment --comment "kubernetes service traffic requiring SNAT" -m mark --mark 0x4000/0x4000 -j MASQUERADE --random-fully
-A cali-PREROUTING -m comment --comment "cali:r6XmIziWUJsdOK6Z" -j cali-fip-dnat
-A cali-POSTROUTING -m comment --comment "cali:Z-c7XtVd2Bq7s_hA" -j cali-fip-snat
-A cali-POSTROUTING -m comment --comment "cali:nYKhEzDlr11Jccal" -j cali-nat-outgoing
-A cali-POSTROUTING -o tunl0 -m comment --comment "cali:SXWvdsbh4Mw7wOln" -m addrtype ! --src-type LOCAL --limit-iface-out -m addrtype --src-type LOCAL -j MASQUERADE --random-fully
-A cali-OUTPUT -m comment --comment "cali:GBTAv2p5CwevEyJm" -j cali-fip-dnat
-A cali-nat-outgoing -m comment --comment "cali:flqWnvo8yq4ULQLa" -m set --match-set cali40masq-ipam-pools src -m set ! --match-set cali40all-ipam-pools dst -j MASQUERADE --random-fully
-A CNI-HOSTPORT-SETMARK -m comment --comment "CNI portfwd masquerade mark" -j MARK --set-xmark 0x2000/0x2000
-A CNI-HOSTPORT-MASQ -m mark --mark 0x2000/0x2000 -j MASQUERADE
-A CNI-HOSTPORT-DNAT -p tcp -m comment --comment "dnat name: \"cni0\" id: \"e05c0b7dc624c20da56c8db60492744381eb27b7449e28e87f8efa6da8210a9e\"" -m multiport --dports 10081,8443 -j CNI-DN-bcfdb00a9541b4df781a0
-A CNI-HOSTPORT-DNAT -p tcp -m comment --comment "dnat name: \"cni0\" id: \"24b63cb1cbda96c6c89e5fb89700141460323010decfb98160b6434700e3a9a4\"" -m multiport --dports 10081,8443 -j CNI-DN-2a1d84389734e54983abb
-A CNI-HOSTPORT-DNAT -p tcp -m comment --comment "dnat name: \"cni0\" id: \"c6578acfe6765587433b73388f55424e1f6e8996f3bb38c1e7ab573995f34245\"" -m multiport --dports 10081,8443 -j CNI-DN-38f4d538cd34246b81019
-A CNI-DN-f0f13904d22032106b1e7 -s 10.233.124.132/32 -p tcp -m tcp --dport 10080 -j CNI-HOSTPORT-SETMARK
-A CNI-DN-f0f13904d22032106b1e7 -s 127.0.0.1/32 -p tcp -m tcp --dport 10080 -j CNI-HOSTPORT-SETMARK

-A CNI-DN-f0f13904d22032106b1e7 -p tcp -m tcp --dport 10080 -j DNAT --to-destination 10.233.124.132:80
-A CNI-DN-442e7af5e08e6bf46621a -s 10.233.124.122/32 -p tcp -m tcp --dport 10080 -j CNI-HOSTPORT-SETMARK
-A CNI-DN-442e7af5e08e6bf46621a -s 127.0.0.1/32 -p tcp -m tcp --dport 10080 -j CNI-HOSTPORT-SETMARK
-A CNI-DN-442e7af5e08e6bf46621a -p tcp -m tcp --dport 10080 -j DNAT --to-destination 10.233.124.122:80
-A CNI-DN-e4582bd2b3ff7ed196a28 -s 10.233.124.141/32 -p tcp -m tcp --dport 10081 -j CNI-HOSTPORT-SETMARK
-A CNI-DN-e4582bd2b3ff7ed196a28 -s 127.0.0.1/32 -p tcp -m tcp --dport 10081 -j CNI-HOSTPORT-SETMARK
-A CNI-DN-365d9a185cff3e5ea9379 -s 127.0.0.1/32 -p tcp -m tcp --dport 10081 -j CNI-HOSTPORT-SETMARK
-A CNI-DN-365d9a185cff3e5ea9379 -p tcp -m tcp --dport 10081 -j DNAT --to-destination 10.233.124.245:80
-A CNI-DN-90b43263af9d462e04f9b -s 10.233.124.192/32 -p tcp -m tcp --dport 10081 -j CNI-HOSTPORT-SETMARK
-A CNI-DN-90b43263af9d462e04f9b -s 127.0.0.1/32 -p tcp -m tcp --dport 10081 -j CNI-HOSTPORT-SETMARK
-A CNI-DN-90b43263af9d462e04f9b -p tcp -m tcp --dport 10081 -j DNAT --to-destination 10.233.124.192:80
-A CNI-DN-90b43263af9d462e04f9b -s 10.233.124.192/32 -p tcp -m tcp --dport 8443 -j CNI-HOSTPORT-SETMARK
-A CNI-DN-90b43263af9d462e04f9b -s 127.0.0.1/32 -p tcp -m tcp --dport 8443 -j CNI-HOSTPORT-SETMARK
-A CNI-DN-90b43263af9d462e04f9b -p tcp -m tcp --dport 8443 -j DNAT --to-destination 10.233.124.192:443
-A CNI-DN-9a7663babfc285ea01c15 -s 10.233.124.151/32 -p tcp -m tcp --dport 10081 -j CNI-HOSTPORT-SETMARK
-A CNI-DN-9a7663babfc285ea01c15 -s 127.0.0.1/32 -p tcp -m tcp --dport 10081 -j CNI-HOSTPORT-SETMARK
-A CNI-DN-9a7663babfc285ea01c15 -p tcp -m tcp --dport 10081 -j DNAT --to-destination 10.233.124.151:80
-A CNI-DN-9a7663babfc285ea01c15 -s 10.233.124.151/32 -p tcp -m tcp --dport 8443 -j CNI-HOSTPORT-SETMARK
-A CNI-DN-9a7663babfc285ea01c15 -s 127.0.0.1/32 -p tcp -m tcp --dport 8443 -j CNI-HOSTPORT-SETMARK
-A CNI-DN-9a7663babfc285ea01c15 -p tcp -m tcp --dport 8443 -j DNAT --to-destination 10.233.124.151:443
-A CNI-DN-fc387103aa44e3b189d09 -s 10.233.124.36/32 -p tcp -m tcp --dport 10081 -j CNI-HOSTPORT-SETMARK
-A CNI-DN-fc387103aa44e3b189d09 -s 127.0.0.1/32 -p tcp -m tcp --dport 10081 -j CNI-HOSTPORT-SETMARK
-A CNI-DN-fc387103aa44e3b189d09 -p tcp -m tcp --dport 10081 -j DNAT --to-destination 10.233.124.36:80
-A CNI-DN-fc387103aa44e3b189d09 -s 10.233.124.36/32 -p tcp -m tcp --dport 8443 -j CNI-HOSTPORT-SETMARK
-A CNI-DN-fc387103aa44e3b189d09 -s 127.0.0.1/32 -p tcp -m tcp --dport 8443 -j CNI-HOSTPORT-SETMARK
-A CNI-DN-fc387103aa44e3b189d09 -p tcp -m tcp --dport 8443 -j DNAT --to-destination 10.233.124.36:443
-A CNI-DN-bcfdb00a9541b4df781a0 -s 10.233.124.50/32 -p tcp -m tcp --dport 10081 -j CNI-HOSTPORT-SETMARK
-A CNI-DN-bcfdb00a9541b4df781a0 -s 127.0.0.1/32 -p tcp -m tcp --dport 10081 -j CNI-HOSTPORT-SETMARK
-A CNI-DN-bcfdb00a9541b4df781a0 -p tcp -m tcp --dport 10081 -j DNAT --to-destination 10.233.124.50:80
-A CNI-DN-bcfdb00a9541b4df781a0 -s 10.233.124.50/32 -p tcp -m tcp --dport 8443 -j CNI-HOSTPORT-SETMARK
-A CNI-DN-bcfdb00a9541b4df781a0 -s 127.0.0.1/32 -p tcp -m tcp --dport 8443 -j CNI-HOSTPORT-SETMARK
-A CNI-DN-bcfdb00a9541b4df781a0 -p tcp -m tcp --dport 8443 -j DNAT --to-destination 10.233.124.50:443
-A CNI-DN-2a1d84389734e54983abb -s 10.233.124.73/32 -p tcp -m tcp --dport 10081 -j CNI-HOSTPORT-SETMARK
-A CNI-DN-2a1d84389734e54983abb -s 127.0.0.1/32 -p tcp -m tcp --dport 10081 -j CNI-HOSTPORT-SETMARK
-A CNI-DN-2a1d84389734e54983abb -p tcp -m tcp --dport 10081 -j DNAT --to-destination 10.233.124.73:80
-A CNI-DN-2a1d84389734e54983abb -s 10.233.124.73/32 -p tcp -m tcp --dport 8443 -j CNI-HOSTPORT-SETMARK
-A CNI-DN-2a1d84389734e54983abb -s 127.0.0.1/32 -p tcp -m tcp --dport 8443 -j CNI-HOSTPORT-SETMARK
-A CNI-DN-2a1d84389734e54983abb -p tcp -m tcp --dport 8443 -j DNAT --to-destination 10.233.124.73:443
-A CNI-DN-38f4d538cd34246b81019 -s 10.233.124.236/32 -p tcp -m tcp --dport 10081 -j CNI-HOSTPORT-SETMARK
-A CNI-DN-38f4d538cd34246b81019 -s 127.0.0.1/32 -p tcp -m tcp --dport 10081 -j CNI-HOSTPORT-SETMARK
-A CNI-DN-38f4d538cd34246b81019 -p tcp -m tcp --dport 10081 -j DNAT --to-destination 10.233.124.236:80
-A CNI-DN-38f4d538cd34246b81019 -s 10.233.124.236/32 -p tcp -m tcp --dport 8443 -j CNI-HOSTPORT-SETMARK
-A CNI-DN-38f4d538cd34246b81019 -s 127.0.0.1/32 -p tcp -m tcp --dport 8443 -j CNI-HOSTPORT-SETMARK
-A CNI-DN-38f4d538cd34246b81019 -p tcp -m tcp --dport 8443 -j DNAT --to-destination 10.233.124.236:443
# Warning: iptables-legacy tables present, use iptables-legacy to see them

@ldx
Copy link
Owner

ldx commented Jul 1, 2020

My understanding is, even though nftables provides a compatibility layer and the CLI is similar, under the hood it's a different library. Python-iptables interacts with iptables via the C libraries, so it doesn't work with nftables.

Recently the official nftables project added python bindings it seems: https://git.netfilter.org/nftables/tree/py so you might want to give that a try. I haven't tested it personally though.

@ldx ldx closed this as completed Aug 7, 2020
@ldx ldx mentioned this issue Aug 7, 2020
Closed
@ldx ldx mentioned this issue Apr 27, 2021
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@ldx @jllorente @iptizer