Module/Helper loading broken in Centos 7.4 for conntrack_proto_gre

[sll552]

Hi,

I encountered an error in a previous release (CentOS 7.3) documented in #242. As recommended in this Issue, I created a service and it worked until I upgraded to CentOS 7.4.

I have the following service defined:

<?xml version="1.0" encoding="utf-8"?>
<service>
  <protocol value="gre"/>
  <module name="nf_conntrack_proto_gre"/>
</service>

This was enough to get firewalld to load the required module.
After the upgrade I received the following error when trying to add the service to a zone:

Error: INVALID_HELPER: proto-gre

I figured that I need to create a helper module which looks like this:

<?xml version="1.0" encoding="utf-8"?>
<helper module="nf_conntrack_proto_gre">
</helper>

But now I get different error:

Error: INVALID_HELPER: 'nf_conntrack_proto_gre' is not available

By looking at https://github.com/firewalld/firewalld/blob/master/src/firewall/functions.py and the get_nf_conntrack_helpers() function, it seems like only conntrack modules which have an alias defined are considered.
But unfortunately the conntrack module for gre doesn't have an alias as modinfo shows:

filename:       /lib/modules/3.10.0-693.2.2.el7.x86_64/kernel/net/netfilter/nf_conntrack_proto_gre.ko.xz
license:        GPL
rhelversion:    7.4
srcversion:     916FE0F82D2357B5A8B68BC
depends:        nf_conntrack
intree:         Y
vermagic:       3.10.0-693.2.2.el7.x86_64 SMP mod_unload modversions
signer:         CentOS Linux kernel signing key
sig_key:        C0:8B:35:53:41:76:0F:03:0A:64:E3:C5:50:E1:13:CD:47:77:1E:A9
sig_hashalgo:   sha256

Is it possible to work around this issue somehow, or is this a Centos issue?

Best regards

[erig0]

IIIRC, all the other nf_conntrack_proto_* (e.g. sctp) are now built directly into the nf_conntrack module for newer kernels. I don't think firewalld currently handles nf_conntrack_proto_gre properly. I believe it's checking for the "helper" string in the modinfo output, but that does not present for nf_conntrack_proto_gre.

Try any of the following as a workaround:

1. manually load nf_conntrack_proto_gre before firewalld starts
2. Use automatic helpers. AutomaticHelpers=yes in /etc/firewalld/firewalld.conf

[sll552]

Thanks for the suggestions.

1. This did not work with the older firewalld version in 7.3, see Conntrack Modules unloaded at firewalld restart/complete-reload #242. But I will try again just to be sure.
2. As far as I know AutomaticHelpers only work if there is a port defined for the protocol, which is not the case for gre.

[sll552]

Hi,
just a quick update/follow-up:

1. As expected only works till the firewalld service gets restarted, then the module gets unloaded. Basically the same thats described in Conntrack Modules unloaded at firewalld restart/complete-reload #242
2. AutomaticHelpers also did not work. I removed the module from the service and set AutomaticHelpers=yes in /etc/firewalld/firewalld.conf and restarted firewalld.

[sll552]

Hi,
whats the status of this, my current workaround is a downgrade of the package.
Is this going to be fixed in the next CentOS release?

[erig0]

There is currently no one working on this bug.
It sounds like pre-loading the module works. If you use filewall-cmd --reload, then it won't unload modules.

[sll552]

It works until firewalld is restarted and that might happen implicitly at reboot or through chef-runs (which are applied in the environment I am working) so this is not an option for me.

Is there a specific reason why the modules are verified so extensively before considering them conntrack modules in firewalld? If not, why not just rely on the name or at least don't reject the module if some fields are not set in modinfo?

[erig0]

Is there a specific reason why the modules are verified so extensively before considering them conntrack modules in firewalld?

Unfortunately I don't know the history, but I'll look into it. It's possible these checks are no longer necessary.

[kvishnivetsky]

Hi, There.

I have found a "workaround" for this issue, which will automaticaly load module nf_conntrack_proto_gre in a rather "system" manner:
Create an executable file /etc/sysconfig/modules/nf_conntrack_proto_gre.modules contents /sbin/modprobe nf_conntrack_proto_gre > /dev/null 2>&1.
Wipe out all you added to filrewalld for loading helper module.
Reboot and have fun.

[sll552]

Please see #242 on why this is not a valid workaround.
In short: firewalld unloads all conntrack modules at service restart, so your workaround only works until the firewalld service is restarted.

[kvishnivetsky]

So, is http://www.firewalld.org/documentation/howto/add-a-service.html valid way to add module loading?

[sll552]

For additional modules to load it is required to define a http://www.firewalld.org/documentation/helper/ and add a module tag to the service http://www.firewalld.org/documentation/service/options.html

This works for most conntrack modules, but not for gre as you can see in the original issue description. The problem is that firewalld is validating the available modules to see if they are conntrack modules before loading and this validation fails for conntrack_gre. All of this has already been discussed in this thread.

[kvishnivetsky]

In which way nf_conntrack modules are validated in firewalld?

[sll552]

https://github.com/firewalld/firewalld/blob/master/src/firewall/functions.py#L334

[kvishnivetsky]

Oooookay, that was simple. I took a function get_nf_conntrack_helpers() from from the firewalld code and made a copy in my test script with all nessesary import header, then added a little debug output.
So, what do we have here for good and bad situation:
[root@pbx-01 ~]# modinfo nf_conntrack_sane
filename: /lib/modules/3.10.0-693.21.1.el7.x86_64/kernel/net/netfilter/nf_conntrack_sane.ko.xz
alias: nfct-helper-sane
....

[root@pbx-01 ~]# modinfo nf_conntrack_proto_gre
filename: /lib/modules/3.10.0-693.21.1.el7.x86_64/kernel/net/netfilter/nf_conntrack_proto_gre.ko.xz

Adn whre is an alias string???? Right you - it's absent.

So, all we need is to find out just how to set this alias for this module.
And here we are:
[root@pbx-01 3.10.0-693.21.1.el7.x86_64]# pwd
/lib/modules/3.10.0-693.21.1.el7.x86_64
[root@pbx-01 3.10.0-693.21.1.el7.x86_64]# grep nfct-helper modules.alias
alias nfct-helper-amanda nf_conntrack_amanda
alias nfct-helper-ftp nf_conntrack_ftp
alias nfct-helper-H.245 nf_conntrack_h323
alias nfct-helper-Q.931 nf_conntrack_h323
alias nfct-helper-RAS nf_conntrack_h323
alias nfct-helper-irc nf_conntrack_irc
alias nfct-helper-netbios_ns nf_conntrack_netbios_ns
alias nfct-helper-snmp nf_conntrack_snmp
alias nfct-helper-pptp nf_conntrack_pptp
alias nfct-helper-sane nf_conntrack_sane
alias nfct-helper-sip nf_conntrack_sip
alias nfct-helper-tftp nf_conntrack_tftp

[kvishnivetsky]

I have analyzed binary files of modules and found out thet nf_conntrack_proto_gre module is broken!
Thre was no any alias defined for it in it's binary image. :(
So, this is not a firewalld issue, but firewalld may add a boolean(Y/N) configuation flag named strict_modules_check to workaround this.

[sll552]

Or not check the alias at all. I don't understand the reasin behind checking it. Name of the module could be enough to check before loading

[kvishnivetsky]

Agree. I think, there was no nf_conntrack_proto_gre at the time this function was added. And Mr. Developer found an easy way to get the final internal module name from an alias.

[kvishnivetsky]

Work fine like this:

def get_nf_conntrack_helpers():
    kver = os.uname()[2]
    path = "/lib/modules/%s/kernel/net/netfilter/" % kver
    helpers = { }
    if os.path.isdir(path):
        for filename in sorted(os.listdir(path)):
            if not filename.startswith("nf_conntrack_"):
                continue
            module = filename.split(".")[0]
            (status, ret) = runProg(COMMANDS["modinfo"], [ module, ])
            if status != 0:
                continue
            print(filename)
-            for line in ret.split("\n"):
-                if line.startswith("alias:") and "-helper-" in line:
-                    helper = line.split(":")[1].strip()
-                    helper = helper.replace("nfct-helper-", "")
-                    helper = helper.replace("_", "-")
-                    helpers.setdefault(module, [ ]).append(helper)
+            helper = filename.split(".")[0].strip()
+            helper = helper.replace("_", "-")
+            helpers.setdefault(module, [ ]).append(helper)
    return helpers

Result list:

nf_conntrack_pptp
nf_conntrack_netlink
nf_conntrack_broadcast
nf_conntrack_ftp
nf_conntrack_amanda
nf_conntrack_proto_gre
nf_conntrack_snmp
nf_conntrack_netbios_ns
nf_conntrack_sip
nf_conntrack_h323
nf_conntrack_irc
nf_conntrack_tftp
nf_conntrack_sane

[kvishnivetsky]

In my firewalld on CentOS Linux release 7.4.1708 (Core) this path work just fine!!!

[sll552]

Thanks for the fix.

Do you know if the next release of firewalld will be in 7.5, or EPEL?

[erig0]

@sll552, RHEL 7.5 released today. So this fix definitely won't make 7.5, but should make 7.6. I'm in progress of backporting this fix to the stable-0.5 branch. I expect RHEL 7.6 will use the latest 0.5.x version of firewalld.

[sll552]

Thanks for the info , I just read the announcement of 7.5 after posting 😄.
I guess I will need to build from source then.