systemd-modules-load doesn't load nf_conntrack module

[morfikov]

I have the following modules to load at boot time:

coretemp
xt_cgroup
cls_cgroup
ifb
xt_connmark
sch_fq_codel
act_mirred
nf_conntrack
xt_recent
tun
eeprom
brcmsmac
mac80211
cfg80211
b43
nfs

But when I checked systemd-sysctl log, I could see the following messages:

# journalctl --no-tail -b -u systemd-sysctl
-- Logs begin at Fri 2015-08-14 12:07:50 CEST, end at Tue 2015-09-01 21:25:56 CEST. --
Sep 01 20:49:24 morfikownia systemd[1]: Starting Apply Kernel Variables...
Sep 01 20:49:24 morfikownia systemd-sysctl[430]: Couldn't write '30' to 'net/netfilter/nf_conntrack_icmpv6_timeout', ignoring: No such file or directory
Sep 01 20:49:24 morfikownia systemd-sysctl[430]: Couldn't write '20' to 'net/netfilter/nf_conntrack_tcp_timeout_time_wait', ignoring: No such file or directory
Sep 01 20:49:24 morfikownia systemd-sysctl[430]: Couldn't write '20' to 'net/netfilter/nf_conntrack_tcp_timeout_fin_wait', ignoring: No such file or directory
Sep 01 20:49:24 morfikownia systemd-sysctl[430]: Couldn't write '28800' to 'net/netfilter/nf_conntrack_tcp_timeout_established', ignoring: No such file or directory
Sep 01 20:49:24 morfikownia systemd-sysctl[430]: Couldn't write '30' to 'net/netfilter/nf_conntrack_icmp_timeout', ignoring: No such file or directory
Sep 01 20:49:24 morfikownia systemd[1]: Started Apply Kernel Variables.

I have also other sysctl settings, but only the above ones have some problems.

After the boot stage, I can issue sysctl -p and all the variables are set properly. So I checked the systemd-modules-load.service log, and there's something like this:

# journalctl --no-tail -b -u systemd-modules-load.service
-- Logs begin at Fri 2015-08-14 12:07:50 CEST, end at Tue 2015-09-01 21:25:56 CEST. --
Sep 01 20:49:22 morfikownia systemd-modules-load[341]: Inserted module 'loop'
Sep 01 20:49:22 morfikownia systemd-modules-load[341]: Inserted module 'lp'
Sep 01 20:49:22 morfikownia systemd-modules-load[341]: Inserted module 'ppdev'
Sep 01 20:49:22 morfikownia systemd-modules-load[341]: Inserted module 'parport_pc'
Sep 01 20:49:22 morfikownia systemd-modules-load[341]: Inserted module 'ecryptfs'
Sep 01 20:49:22 morfikownia systemd-modules-load[341]: Inserted module 'coretemp'
Sep 01 20:49:23 morfikownia systemd-modules-load[341]: Inserted module 'xt_cgroup'
Sep 01 20:49:23 morfikownia systemd-modules-load[341]: Inserted module 'cls_cgroup'
Sep 01 20:49:23 morfikownia systemd-modules-load[341]: Inserted module 'ifb'
Sep 01 20:49:23 morfikownia systemd-modules-load[341]: Inserted module 'xt_connmark'
Sep 01 20:49:23 morfikownia systemd-modules-load[341]: Inserted module 'sch_fq_codel'
Sep 01 20:49:23 morfikownia systemd-modules-load[341]: Inserted module 'act_mirred'
Sep 01 20:49:23 morfikownia systemd-modules-load[341]: Inserted module 'xt_recent'
Sep 01 20:49:23 morfikownia systemd-modules-load[341]: Inserted module 'tun'
Sep 01 20:49:23 morfikownia systemd-modules-load[341]: Inserted module 'eeprom'
Sep 01 20:49:24 morfikownia systemd-modules-load[341]: Inserted module 'brcmsmac'
Sep 01 20:49:24 morfikownia systemd-modules-load[341]: Inserted module 'b43'
Sep 01 20:49:24 morfikownia systemd-modules-load[341]: Inserted module 'nfs'
Sep 01 20:49:24 morfikownia systemd[1]: Started Load Kernel Modules.

When compared this to the modules list that you can see above, there's only one module missing: nf_conntrack. I don't know why because it's included in the /etc/modules file. Anyway, it's loaded later somehow:

# lsmod | grep nf_conntrack
nf_conntrack_ipv6      20480  6
nf_defrag_ipv6         36864  1 nf_conntrack_ipv6
nf_conntrack_ipv4      20480  24
nf_defrag_ipv4         16384  1 nf_conntrack_ipv4
nf_conntrack           94208  7 xt_CT,nf_nat,nf_nat_ipv4,xt_conntrack,xt_connmark,nf_conntrack_ipv4,nf_conntrack_ipv6

The question is why the module isn't loaded with all the other modules?

[poettering]

Does your initrd contain systemd-modules-load? If so, did you rebuild the initrd after changing the config file, and did you make sure all relevent .ko files are also in the initrd?

[morfikov]

I have no idea whether it contains systemd-modules-load -- how to check it? I'm using debian distro.

Anyway, I tried to add all the modules from /etc/modules file to /etc/initramfs-tools/modules. Previously, none of them was in the second file. I've compared the initrd content before and after the change, and all the modules are now included. But after reboot, it still can't set the settings in question -- still the same errors. This is the full tree view of the initrd content if you're interested.

[mbiebl]

@morfikov the xt_connmark module requires nf_conntrack and loading xt_connmark triggers the load of nf_conntrack.
Since you load that xt_connmark, nf_conntrack will already been loaded. So systemd-modules-load simply skips this module.

[mbiebl]

Let me rephrase that: xt_connmark comes before nf_conntrack in /etc/modules. Afaik, systemd-modules-load loads the modules in order, so it loads xt_connmark first (which triggers the auto-load of nf_conntrack). So at the point systemd-modules-load tries to load nf_conntrack, it's already loaded, so this entry is skipped.

[morfikov]

Ok, that makes sense. I even changed the order to be sure, and now it shows all the modules in the log. So that's fine, but the messages still are logged, so do you have any idea why the variables can't be set at the boot stage?

[mbiebl]

@morfikov systemd-sysctl.service has a After=systemd-modules-load.service.
Afaik, loading the module is asynchronous though, so the insmod call does not block until all device nodes, sysfs entries etc are there.
The only other solution I could think of would be to apply these settings via udev rules.

@kaysievers, @poettering do you have a bette suggestion how this could be addressed?

[morfikov]

According to the manual, I've created the following udev rule:

ACTION=="add", SUBSYSTEM=="module", KERNEL=="nf_conntrack", \
      RUN+="/lib/systemd/systemd-sysctl --prefix=/net/netfilter"

But it didn't work. Now I also have the messages in question in the systemd-udevd.service log.

I've looked into the /etc/sysctl.conf file in order to see what nf_conntrack variables there are, and I could see the following:

# egrep net.netfilter /etc/sysctl.conf
net.netfilter.nf_conntrack_timestamp = 1
net.netfilter.nf_conntrack_tcp_timeout_fin_wait = 20
net.netfilter.nf_conntrack_max = 32768
net.netfilter.nf_conntrack_tcp_timeout_established = 28800
net.netfilter.nf_conntrack_tcp_timeout_time_wait = 20
net.netfilter.nf_conntrack_icmp_timeout = 30
net.netfilter.nf_conntrack_icmpv6_timeout = 30

But when compared them to the sysctl log, it looks like that nf_conntrack_timestamp and nf_conntrack_max work just fine and the two options are set correctly without errors in the log, other five aren't -- that's weird.

[morfikov]

I've found it! To solve this issue, I had to add the two following modules to the /etc/modules file:

nf_conntrack_ipv4
nf_conntrack_ipv6

Now all the errors disappeared.

[poettering]

OK, closing then.

[bluikko]

For reference to anyone else fighting this on later distributions:

1. The aforementioned 2 modules have been merged to a single module nf_conntrack. So the "workaround" in systemd-modules-load doesn't load nf_conntrack module #1113 (comment) applies only to old distributions.
2. While systemd-sysctl may have after=systemd-modules-load setting the modules may still not be loaded in time for systemd-sysctl: on EL9 the service systemd-sysctl tries to run before systemd-modules-load loads the module nf_conntrack as listed in a /etc/modules-load.d/ file: 5 seconds earlier on my current test machine:

Jun 12 11:12:38 machine systemd-sysctl[312]: Couldn't write '0' to 'net/netfilter/nf_conntrack_tcp_loose', ignoring: No such file or directory
[...]
Jun 12 11:12:43 machine systemd-modules-load[624]: Inserted module 'nf_conntrack'

It would seem like other workarounds are needed on newer distributions such as EL9.

[kucerakov]

On Ubuntu 22.04, what we found as a workaround was to force the nf_conntrack and it's ip_vs dependency to be loaded earlier by adding them to the /etc/modules-load.d/modules.conf file. This ensured the file would be there when the kernel needed to write the value from sysctl.conf