Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Failed to load nf_conntrack module on CentOS 7.7 container #519

Closed
FathiBenNasr opened this issue Sep 27, 2019 · 28 comments
Projects

Comments

@FathiBenNasr
Copy link

@FathiBenNasr FathiBenNasr commented Sep 27, 2019

Hello,
I am having trouble since I've upgraded a CentOS 7.6 container to CentOS 7.7 running on ProxMox 6.0.
The problem is that firewalld no more starts complaining about nf_conntrack module as follows.
ERROR: Failed to load nf_conntrack module: modprobe: ERROR: could not find module by name='nf_conntrack'
modprobe: ERROR: could not insert 'nf_conntrack': Function not implemented
modprobe: ERROR: Error running install command for nf_conntrack...
ERROR: Raising SystemExit in run_server

The output of modinfo nf_conntrack is
modinfo: ERROR: Module alias nf_conntrack not found.

While on a VPS running at french provider OVH, i have as output:
filename: /lib/modules/3.10.0-1062.1.1.el7.x86_64/kernel/net/netfilter/nf_conntrack.ko.xz
license: GPL
retpoline: Y
rhelversion: 7.7
srcversion: 03A8408E58BFA6E173F2FE6
depends: libcrc32c
intree: Y
vermagic: 3.10.0-1062.1.1.el7.x86_64 SMP mod_unload modversions
signer: CentOS Linux kernel signing key
sig_key: 34:1A:1E:7B:06:D6:87:15:3E:3A:E9:8D:3E:B5:6E:0E:CD:30:DB:79
sig_hashalgo: sha256
parm: tstamp:Enable connection tracking flow timestamping. (bool)
parm: acct:Enable connection tracking flow accounting. (bool)
parm: nf_conntrack_helper:Enable automatic conntrack helper assignment (default 1) (bool)
parm: expect_hashsize:uint

On the ProxMox server side, i have also a positive output:
filename: /lib/modules/5.0.21-2-pve/kernel/net/netfilter/nf_conntrack.ko
license: GPL
alias: nf_conntrack-10
alias: nf_conntrack-2
alias: ip_conntrack
srcversion: ECF2FC78962840323375B8C
depends: nf_defrag_ipv6,libcrc32c,nf_defrag_ipv4
retpoline: Y
intree: Y
name: nf_conntrack
vermagic: 5.0.21-2-pve SMP mod_unload modversions
parm: tstamp:Enable connection tracking flow timestamping. (bool)
parm: acct:Enable connection tracking flow accounting. (bool)
parm: nf_conntrack_helper:Enable automatic conntrack helper assignment (default 0) (bool)
parm: expect_hashsize:uint

The output of rpm -qf /lib/modules/3.10.0-1062.1.1.el7.x86_64/kernel/net/netfilter/nf_conntrack.ko.xz on the centos 7.7 vps is
kernel-3.10.0-1062.1.1.el7.x86_64

Now, on a proxmox container, there is no kernel installed as it is a container.
As this is an urgent case, I removed firewalld firewalld-filesystem and python-firewall all 0.6.3-2 and reinstalled from centos vault those of the 7.6 version of centos (0.5.3-5). And now the firewalld service starts.

Now, on the centos 7.7 container:
lsmod | grep nf_
nf_reject_ipv4 16384 1 ipt_REJECT
nf_reject_ipv6 20480 1 ip6t_REJECT
nf_nat_ipv6 16384 2 ip6table_nat,ip6t_MASQUERADE
nf_nat_ipv4 16384 2 ipt_MASQUERADE,iptable_nat
nf_nat 36864 2 nf_nat_ipv6,nf_nat_ipv4
nf_conntrack 139264 6 xt_conntrack,nf_nat,ip6t_MASQUERADE,nf_nat_ipv6,ipt_MASQUERADE,nf_nat_ipv4
nf_defrag_ipv6 24576 1 nf_conntrack
nf_defrag_ipv4 16384 1 nf_conntrack
libcrc32c 16384 4 nf_conntrack,nf_nat,dm_persistent_data,btrfs

and on the proxmox hypervisor:
lsmod | grep nf_
nf_reject_ipv4 16384 1 ipt_REJECT
nf_reject_ipv6 20480 1 ip6t_REJECT
nf_nat_ipv6 16384 2 ip6table_nat,ip6t_MASQUERADE
nf_nat_ipv4 16384 2 ipt_MASQUERADE,iptable_nat
nf_nat 36864 2 nf_nat_ipv6,nf_nat_ipv4
nf_conntrack 139264 6 xt_conntrack,nf_nat,ip6t_MASQUERADE,nf_nat_ipv6,ipt_MASQUERADE,nf_nat_ipv4
nf_defrag_ipv6 24576 1 nf_conntrack
nf_defrag_ipv4 16384 1 nf_conntrack
libcrc32c 16384 4 nf_conntrack,nf_nat,dm_persistent_data,btrfs

So nf_conntrack seems to be loaded on both the hypervisor and the container and beeing useable by firewalld 0.5.3 but not by firewalld 0.6.3.

Could someone help me solve this issue please ?
TIA
Fathi Ben Nasr

@cominderace

This comment has been minimized.

Copy link

@cominderace cominderace commented Oct 15, 2019

We have got this problem too. VPS under VirtOS.
Interested in gettings thsi issue diagnostic and fix.

@BlackMage2

This comment has been minimized.

Copy link

@BlackMage2 BlackMage2 commented Oct 19, 2019

i have the same issue on my VPS under OpenVZ and i have the firewalld Version : 0.6.3 and the CentOS Linux release is 7.3.1611 (and now with 7.7.1908 the same) and kernel is 2.6.32-042stab127.2

/var/log/firewalld

2019-10-18 13:22:30 WARNING: ipset not usable, disabling ipset usage in firewall.
2019-10-18 13:22:30 ERROR: Failed to load nf_conntrack module: modprobe: ERROR: could not find module by name='nf_conntrack'
modprobe: ERROR: could not insert 'nf_conntrack': Function not implemented
modprobe: ERROR: Error running install command for nf_conntrack
modprobe: ERROR: could not insert 'nf_conntrack': Operation not permitted

2019-10-18 13:22:30 ERROR: Raising SystemExit in run_server

and what means 'WARNING: ipset not usable, disabling ipset usage in firewall.'? what is ipset?

@JeroenSteen

This comment has been minimized.

Copy link

@JeroenSteen JeroenSteen commented Oct 21, 2019

Failed to load nf_conntrack module: modprobe: ERROR: could not find module by name='nf_conntrack'.

Same issue on 3.10.0-042stab140.1.

@erig0

This comment has been minimized.

Copy link
Collaborator

@erig0 erig0 commented Oct 23, 2019

@erig0 erig0 added this to Priority bugs in firewalld Oct 23, 2019
@BlackMage2

This comment has been minimized.

Copy link

@BlackMage2 BlackMage2 commented Oct 23, 2019

@erig0 i have no access to the redhat bugzilla bug and the workaround described at #430 doesn´t work for me

@erig0

This comment has been minimized.

Copy link
Collaborator

@erig0 erig0 commented Oct 23, 2019

@erig0 i have no access to the redhat bugzilla bug and the workaround described at #430 doesn´t work for me

I didn't mean to imply they provided a workaround. Just that they're related to this issue.

@BlackMage2

This comment has been minimized.

Copy link

@BlackMage2 BlackMage2 commented Oct 24, 2019

@erig0 ohh okay, I have probably misinterpreted then

@alexgit2k

This comment has been minimized.

Copy link

@alexgit2k alexgit2k commented Oct 24, 2019

Workaround is to downgrade firewalld to 7.6:

wget http://vault.centos.org/7.6.1810/os/x86_64/Packages/firewalld-0.5.3-5.el7.noarch.rpm http://vault.centos.org/7.6.1810/os/x86_64/Packages/firewalld-filesystem-0.5.3-5.el7.noarch.rpm http://vault.centos.org/7.6.1810/os/x86_64/Packages/python-firewall-0.5.3-5.el7.noarch.rpm
yum downgrade firewalld-0.5.3-5.el7.noarch.rpm firewalld-filesystem-0.5.3-5.el7.noarch.rpm python-firewall-0.5.3-5.el7.noarch.rpm
@BlackMage2

This comment has been minimized.

Copy link

@BlackMage2 BlackMage2 commented Oct 24, 2019

then
ERROR: Failed to read file "/proc/sys/net/netfilter/nf_conntrack_helper": [Errno 2] No such file or directory: '/proc/sys/net/netfilter/nf_conntrack_helper WARNING: Failed to get and parse nf_conntrack_helper setting WARNING: ebtables not usable, disabling ethernet bridge firewall. log( /var/log/firewalld )...

@alexgit2k

This comment has been minimized.

Copy link

@alexgit2k alexgit2k commented Oct 24, 2019

Getting the same error, but at least iptables is running after the downgrade to 0.5.3 unlike 0.6.3:
iptables -nvL

@JeroenSteen

This comment has been minimized.

Copy link

@JeroenSteen JeroenSteen commented Oct 24, 2019

When I try the workaround of alexgit2k, I get ERROR: '/usr/sbin/iptables-restore -w -n' failed: iptables-restore v1.4.21: Kernel module xt_set is not loaded in. ERROR: COMMAND_FAILED: '/usr/sbin/iptables-restore -w -n' failed: iptables-restore v1.4.21: Kernel module xt_set is not loaded in.

@alexgit2k

This comment has been minimized.

Copy link

@alexgit2k alexgit2k commented Oct 24, 2019

Working here (0.5.3 but not 0.6.3) on containers (I assume OpenVZ) with 7.7 and host kernels 3.10.0-042stab138.1 (x86_64) / 3.10.0 (x86_64).

@erig0

This comment has been minimized.

Copy link
Collaborator

@erig0 erig0 commented Oct 24, 2019

Problem:

  • The various modules (nf_conntrack, nf_conntrack_ftp) are manually loaded by firewalld, however they may not be modprobe-able due to various reasons; builtin, insufficient privileges (containers)

Solution:

  • don't bother with modinfo. Trust what's in the helper definitions.
  • Assume nf_conntrack_helper=0, no harm in adding the rules/modules even if it's =1.
    • avoids manual load of nf_conntrack and checking /proc for nf_conntrack_helper at start()
  • must still manually load nat modules as they won't be implicitly loaded by helper rules, so might as well manually load the helpers too.
  • remove all the special casing for nf_conntrack_helper=1.
  • ignore failures to load modules, warn in logs but no error
@JeroenSteen

This comment has been minimized.

Copy link

@JeroenSteen JeroenSteen commented Oct 24, 2019

Which files needs to be touched erig0? For nf_conntrack_helper=0? Please be more clear.

@erig0

This comment has been minimized.

Copy link
Collaborator

@erig0 erig0 commented Oct 24, 2019

Which files needs to be touched erig0? For nf_conntrack_helper=0? Please be more clear.

My comment above was more of a note for me/developers. It's a list of what needs to be done to fix the issues with module loading.

@FathiBenNasr

This comment has been minimized.

Copy link
Author

@FathiBenNasr FathiBenNasr commented Oct 26, 2019

Workaround is to downgrade firewalld to 7.6:

wget http://vault.centos.org/7.6.1810/os/x86_64/Packages/firewalld-0.5.3-5.el7.noarch.rpm http://vault.centos.org/7.6.1810/os/x86_64/Packages/firewalld-filesystem-0.5.3-5.el7.noarch.rpm http://vault.centos.org/7.6.1810/os/x86_64/Packages/python-firewall-0.5.3-5.el7.noarch.rpm
yum downgrade firewalld-0.5.3-5.el7.noarch.rpm firewalld-filesystem-0.5.3-5.el7.noarch.rpm python-firewall-0.5.3-5.el7.noarch.rpm

That's what I did to temporarily circumvent the issue.
Don't forget to "yum lock" these three packages so they don't get updated next time you run a yum update.

@BlackMage2

This comment has been minimized.

Copy link

@BlackMage2 BlackMage2 commented Oct 26, 2019

you mean "yum versionlock"(available in yum-plugin-versionlock)?
So all commands are:

wget http://vault.centos.org/7.6.1810/os/x86_64/Packages/firewalld-0.5.3-5.el7.noarch.rpm http://vault.centos.org/7.6.1810/os/x86_64/Packages/firewalld-filesystem-0.5.3-5.el7.noarch.rpm http://vault.centos.org/7.6.1810/os/x86_64/Packages/python-firewall-0.5.3-5.el7.noarch.rpm
yum downgrade firewalld-0.5.3-5.el7.noarch.rpm firewalld-filesystem-0.5.3-5.el7.noarch.rpm python-firewall-0.5.3-5.el7.noarch.rpm
yum -y install yum-versionlock
yum versionlock firewalld firewalld-filesystem python-firewall

@FathiBenNasr

This comment has been minimized.

Copy link
Author

@FathiBenNasr FathiBenNasr commented Oct 26, 2019

yes,
yum versionlock firewalld-0.5.3-5.el7
yum versionlock firewalld-filesystem-0.5.3-5.el7
yum versionlock python-firewall-0.5.3-5.el7
and then yum versionlock list to check it has been done.

@BlackMage2

This comment has been minimized.

Copy link

@BlackMage2 BlackMage2 commented Oct 26, 2019

You can also exclude all 3 in one versionlock command from an update. as I wrote in the previous post. The full package names with version numbers are not necessary, The currently installed version is automatically taken for the versionlock.

@FathiBenNasr

This comment has been minimized.

Copy link
Author

@FathiBenNasr FathiBenNasr commented Oct 28, 2019

yes,
yum versionlock firewalld-0.5.3-5.el7
yum versionlock firewalld-filesystem-0.5.3-5.el7
yum versionlock python-firewall-0.5.3-5.el7
and then yum versionlock list to check it has been done.

You can also exclude all 3 in one versionlock command from an update. as I wrote in the previous post. and the full package names with version numbers are not necessary. it will automatically lure the currently installed version

Thank you @BlackMage2 . Didn't know that.

@erig0 erig0 closed this in 88e76dd Oct 30, 2019
@erig0 erig0 moved this from Priority bugs to Done in firewalld Oct 30, 2019
@till

This comment has been minimized.

Copy link

@till till commented Nov 6, 2019

Sorry to comment on a closed ticket, but this seems like a major issue.

I know I can downgrade firewalld and I can yum versionlockit, but that defeats the purpose of running CentOS all together. I would expect a fix to land in CentOS 7.7. Can anyone clarify if will happen?

@erig0

This comment has been minimized.

Copy link
Collaborator

@erig0 erig0 commented Nov 6, 2019

Sorry to comment on a closed ticket, but this seems like a major issue.

I know I can downgrade firewalld and I can yum versionlockit, but that defeats the purpose of running CentOS all together. I would expect a fix to land in CentOS 7.7. Can anyone clarify if will happen?

Please follow the redhat bug report to know the status of it going to CentOS 7.7: https://bugzilla.redhat.com/show_bug.cgi?id=1754029

@till

This comment has been minimized.

Copy link

@till till commented Nov 6, 2019

Sorry to comment on a closed ticket, but this seems like a major issue.
I know I can downgrade firewalld and I can yum versionlockit, but that defeats the purpose of running CentOS all together. I would expect a fix to land in CentOS 7.7. Can anyone clarify if will happen?

Please follow the redhat bug report to know the status of it going to CentOS 7.7: https://bugzilla.redhat.com/show_bug.cgi?id=1754029

I don't have access to this link?

@alexgit2k

This comment has been minimized.

Copy link

@alexgit2k alexgit2k commented Nov 6, 2019

@till: I'm quite sure that it will be fixed in 7.7. The status is still assigned.

@erig0

This comment has been minimized.

Copy link
Collaborator

@erig0 erig0 commented Nov 6, 2019

Sorry to comment on a closed ticket, but this seems like a major issue.
I know I can downgrade firewalld and I can yum versionlockit, but that defeats the purpose of running CentOS all together. I would expect a fix to land in CentOS 7.7. Can anyone clarify if will happen?

Please follow the redhat bug report to know the status of it going to CentOS 7.7: https://bugzilla.redhat.com/show_bug.cgi?id=1754029

I don't have access to this link?

Right. Sorry. Short answers is; I'm trying to get it into 7.7, but it might not make it.

@BlackMage2

This comment has been minimized.

Copy link

@BlackMage2 BlackMage2 commented Nov 6, 2019

i have also no access to https://bugzilla.redhat.com/show_bug.cgi?id=1754029
What is it?

@erig0

This comment has been minimized.

Copy link
Collaborator

@erig0 erig0 commented Nov 6, 2019

i have also no access to https://bugzilla.redhat.com/show_bug.cgi?id=1754029
What is it?

It's basically a downstream (RHEL) report of this issue.

erig0 added a commit to erig0/firewalld that referenced this issue Nov 7, 2019
There are many cases in which module loading may fail:
 - builtin modules, but corrupt/missing modules.builtin database
 - CONFIG_MODULES=n
 - inside unprivileged container

Unfortunately, we have no way to detect these scenarios. The only thing
we can do is attempt to load the module and hope for the best.

Fixes: firewalld#430
Fixes: firewalld#519
(cherry picked from commit 88e76dd)
erig0 added a commit that referenced this issue Nov 8, 2019
There are many cases in which module loading may fail:
 - builtin modules, but corrupt/missing modules.builtin database
 - CONFIG_MODULES=n
 - inside unprivileged container

Unfortunately, we have no way to detect these scenarios. The only thing
we can do is attempt to load the module and hope for the best.

Fixes: #430
Fixes: #519
(cherry picked from commit 88e76dd)
@dvershinin

This comment has been minimized.

Copy link

@dvershinin dvershinin commented Nov 17, 2019

Till the fix has arrived to RHEL 7, here's a fix that automatically generates modules.builtin on non-stock kernels automatically, during startup.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
firewalld
  
Done
8 participants
You can’t perform that action at this time.