Tables not belonging to firewalld gets flushed on firewalld reload/restart

[nicolas-goudry]

What happened:

Default tables (ip filter, ip nat, ip mangle, ect…) not belonging to firewalld gets flushed on firewall-cmd --reload or systemctl restart firewalld or systemctl reload firewalld

What you expected to happen:

Only firewalld table gets flushed

How to reproduce it (as minimally and precisely as possible):

1. Create a rule in ip filter table: nft add rule ip filter INPUT ip saddr 10.1.1.9 drop
2. Check rule is present: nft list chain ip filter INPUT
3. Reload firewalld: firewall-cmd --reload
4. Check rule has been flushed: nft list chain ip filter INPUT

Same happens when using iptables command (please note that on RHEL8 based OS, iptables uses nf_tables API, effectively adding rules into nftables):

1. Create a rule in ip filter table: iptables -A INPUT -s 10.1.1.9 -j DROP
2. Check rule is present in iptables: iptables-save
3. Check rule is present in nftables: nft list chain ip filter INPUT
4. Reload firewalld: firewall-cmd --reload
5. Check rule is present in iptables: iptables-save
6. Check rule has been flushed: nft list chain ip filter INPUT

Anything else we need to know?:

This doesn’t happen when creating a custom table. You can also read this serverfault post for better context understanding.

Environment:

• Firewalld Version (if Fedora based dnf info firewalld or commit hash if developing from git git log -n1 --format=format:"%H"):

$ firewall-cmd --version
0.8.2

• Firewalld Backend (cat /etc/firewalld/firewalld.conf | grep FirewallBackend):

$ cat /etc/firewalld/firewalld.conf | grep FirewallBackend
# FirewallBackend
FirewallBackend=nftables

• OS (e.g: cat /etc/os-release):

$ cat /etc/os-release
NAME="Rocky Linux"
VERSION="8.4 (Green Obsidian)"
ID="rocky"
ID_LIKE="rhel fedora"
VERSION_ID="8.4"
PLATFORM_ID="platform:el8"
PRETTY_NAME="Rocky Linux 8.4 (Green Obsidian)"
ANSI_COLOR="0;32"
CPE_NAME="cpe:/o:rocky:rocky:8.4:GA"
HOME_URL="https://rockylinux.org/"
BUG_REPORT_URL="https://bugs.rockylinux.org/"
ROCKY_SUPPORT_PRODUCT="Rocky Linux"
ROCKY_SUPPORT_PRODUCT_VERSION="8"

• Others:

$ iptables -V
iptables v1.8.4 (nf_tables)

[erig0]

There are a couple things going on here.

• The nft variant of iptables will add rules to nftables in the known tables: filter, nat, raw, etc.
• firewalld always flushes all iptables rules (e.g. iptables -F -t filter) which flushes all chains in the filter table. This corresponds to the known nftables table names mentioned above.

Firewalld's nftables backend only touches the firewalld table, but tables filter, nat, raw, etc. are touched indirectly via iptables-nft to support iptables direct rules. This behavior cannot be changed due to compatibility. Firewalld will not touch other nftables tables though, e.g. a table named foobar would not get flushed.

Unfortunately we can't do anything here. We either a) maintain compatibility and let the iptables named tables get flushed or b) drop iptables direct rule support. Option A is the most user friendly.

[erig0]

As an aside, I don't know what you're trying to do with direct rules, but firewalld v0.9.0+ supports policies which cover a lot of use cases typically done via direct rules.
https://firewalld.org/2020/09/policy-objects-introduction

Policy Objects should be in the upcoming RHEL-8.5.

[nicolas-goudry]

@erig0 Thanks for you quick reply and sorry for my delay responding to you!

I understand that this is for backward compatibility purposes, but would it be acceptable/possible to provide an option in firewalld preventing it to flush iptables rules? I know it's an edge case, but I should say “nothing ventured, nothing gained”.

Regarding your comment on policies, I think it’s a great feature but it doesn’t match my use case because the issue is related to a third party in a Kubernetes environment. See this serverfault post to better understand my use case.

Anyway, thanks again and keep up the good work with firewalld! It’s a great software 🙂

[erig0]

@erig0 Thanks for you quick reply and sorry for my delay responding to you!

I understand that this is for backward compatibility purposes, but would it be acceptable/possible to provide an option in firewalld preventing it to flush iptables rules? I know it's an edge case, but I should say “nothing ventured, nothing gained”.

Maybe FlushAllOnReload=no in /etc/firewalld/firewalld.conf would help your use case.

       FlushAllOnReload
           Flush all runtime rules on a reload. In previous releases some
           runtime configuration was retained during a reload, namely;
           interface to zone assignment, and direct rules. This was confusing
           to users. To get the old behavior set this to "no". Defaults to
           "yes".

However, this does not address flushing on a full stop/start, e.g. systemctl restart firewalld.

[nicolas-goudry]

@erig0 Thanks for the workaround, I will definitely try that!

[champtar]

@erig0 it would be nice to have an option to disable iptables usage

For people looking for a workaround, you can hide iptables from firewalld:

cat > /etc/systemd/system/firewalld.service.d/override.conf <<'EOF'
[Service]
InaccessiblePaths=/usr/sbin/xtables-nft-multi
EOF

[mangelajo]

we are also hitting the same issue with MicroShift, firewalld flushes iptables on reload or restart (even if you use the .conf settings to prevent that, but probably I misunderstood those). Flush also happens under certain events, like DHCP assignment of a new address or hostname (https://issues.redhat.com/browse/USHIFT-789).

The workaround described in #863 (comment) seems to help, but I agree with @champtar, a setting to avoid flushing iptables would help with this.
We are tracking those issues here: https://issues.redhat.com/browse/NP-641

@erig0 I understand that the workaround would work as long as MicroShift/OVN doesn't generate incompatible rules to firewalld. But I would love to have a moment to talk about this issue.

[champtar]

Just a caveat with my workaround, it triggers SELinux: https://bugzilla.redhat.com/show_bug.cgi?id=2126583

[mangelajo]

Just a caveat with my workaround, it triggers SELinux: https://bugzilla.redhat.com/show_bug.cgi?id=2126583

Thanks for the heads up. It seems not to be very especially noisy, but yes, better if we can handle this in firewalld.

In which context did you hit this issue with firewalld @champtar ?

[champtar]

Single node K8S 'appliance', we're using firewalled to protect the host. We open some ports dynamically by looking at the hostPorts, and we allow users to change firewalld config as they see fit. Too many ways to trigger a flush, we needed a way to make it fool proof. (And I don't like to put warnings in docs, nobody read docs)

[erig0]

FWIW, as of firewalld v1.0.0 the iptables backend is optional. It can be disabled at build time. This configuration is covered by CI.

e.g.

# ./configure IPTABLES=/bin/false IPTABLES_RESTORE=/bin/false IP6TABLES=/bin/false IP6TABLES_RESTORE=/bin/false

[erig0]

We may be able to skip flushing iptables if the FirewallBackend is nftables and there haven't been any direct rules used.

[erig0]

@erig0 I understand that the workaround would work as long as MicroShift/OVN doesn't generate incompatible rules to firewalld. But I would love to have a moment to talk about this issue.

No. What happens is iptables becomes unusable (from firewalld's perspective). So when firewalld starts it detects that and the iptables backend is disabled. This is functionally equivalent to the build hack I mentioned above.

firewalld should throw an INFO level log (syslog or /var/log/firewalld):

iptables is not usable.
ip6tables is not usable.
ebtables is not usable.

[champtar]

@erig0 most people don't want to rebuild core packages of their distro ;)
No direct rules == no flush would be nice

[erig0]

Reopened. We can attempt to avoid touching iptables (i.e. skip flush on shutdown) if FirewallBackend=nftables and there are no --direct rules.

[mangelajo]

We may be able to skip flushing iptables if the FirewallBackend is nftables and there haven't been any direct rules used.

What do you mean by direct rules? Direct iptables rules? Direct nftables rules?

[mangelajo]

We may be able to skip flushing iptables if the FirewallBackend is nftables and there haven't been any direct rules used.

What do you mean by direct rules? Direct iptables rules? Direct nftables rules?

Ah sorry, you mean iptable rules added via direct firewalld rules.

[mangelajo]

Reopened. We can attempt to avoid touching iptables (i.e. skip flush on shutdown) if FirewallBackend=nftables and there are no --direct rules.

Thanks for re-prioritizing this, I see you added the labels but the bug is still in "Closed" state.

[erig0]

@mangelajo, #1082 addresses this. It would be great if you could provide feedback, e.g. test in your environment. Otherwise I'll just merge it since there is reasonable test coverage. ;)

[mangelajo]

@mangelajo, #1082 addresses this. It would be great if you could provide feedback, e.g. test in your environment. Otherwise I'll just merge it since there is reasonable test coverage. ;)

Thank you, I am testing it now on the environment.