Fails to load autofs module through autofs4 alias #9501
Comments
|
do you manage to load it manually with modprobe/insmode ? I am asking because right now, the "fuse" module in debian testing gives me "missing symbols" errors, so it's worth double-checking the basics... |
|
I don't have a 4.18 kernel to test this (yet), I'm merely forwarding the downstream bug report. |
|
systemd loads kmods by name during the early module loading, and doesn't process aliases. I am not convinced it's really a good idea to do the full alias magic this early. I mean, the alias stuff maps names to sets of kmods iirc, it's substantially more than just a way to rename modules... urks. Those kernel hackers with their "don't break userspace" mantra should maybe not break stuff like this so willynilly... /cc @raven-au |
|
Hmm, so the alias is added in this commit: The commit msg claims the alias is fine, but actually it isn't for cases like this one... If the kernel folks care that old userspace shall function correctly on new kernels they need to revert this one and a few others that are related, or find a different way out if they want to rename the kmod. /cc @torvalds I mean, we can certainly look into working around this kernel change in userspace, and either process aliases or load autofs too, but this is not going to fix old systemd versions on new kernels... Renaming kernel modules is problematic in general. There's lots of code in this world which checks for kernel features by accessing
Hence, besides the module loading itself, changing kmod names of generic kernel functionality will break stuff, all over the place. |
|
Quoting Andreas Henriksson from IRC: <ah[m]> https://sources.debian.org/src/systemd/239-3/src/core/kmod-setup.c/?hl=125#L125 <ah[m]> IMHO would have been nice if kmod did that internally but it's documented not to so likely won't change: https://sources.debian.org/src/kmod/25-1/libkmod/libkmod-module.c/?hl=314#L314 <ah[m]> Or simply replace kmod_module_new_from_name with kmod_module_new_from_lookup which should dtrt and is used elsewhere in systemd source. |
|
@mbiebl yeah sure, as I already said we can look into adding some fallbacks for this, but that's not going to fix old userspace on new kernels... anyway you look at it the userspace facing API of the kernel did change here... |
|
On Wed, 2018-07-04 at 02:38 -0700, Lennart Poettering wrote:
systemd loads kmods by name during the early module loading, and doesn't
process aliases. I am not convinced it's really a good idea to do the full
alias magic this early. I mean, the alias stuff maps names to sets of kmods
iirc, it's substantially more than just a way to rename modules... urks.
Those kernel hackers with their "don't break userspace" mantra should maybe
not break stuff like this so willynilly...
/cc @raven-au
Yes, this is an age old problem from the autofs fork of version 3
to version 4, well before I took on maintenance of autofs.
I have to apologize for not getting onto renaming the autofs4 module
to autofs for such a very long time so that module auto-loading would
work.
That has been done now and should work from kernel 4.17+.
I'm not sure that adding a MODULE_ALIAS("autofs") to the kernel source
in fs/autofs4/init.c (kernel < 4.17 of course) will be enough but it's
worth a try.
OTOH if you are loading modules by name and you're loading autofs4.ko,
as that was the module name, that should work ok without an alias for
pre 4.17 kernels ... so I'm not sure from above what the problem
actually is?
Perhaps it's the module rename of 4.17+ that is the problem, oops!
Linus really wanted to get rid of the autofs4 directory that I tried
to leave around for compatibility, to be removed in two subsequent
kernel revisions, when people had a chance to notice the change.
So it was removed!
If the rename has caused a problem I can only apologize for that too
but the likelihood is no-one would have noticed the effort I put into
the autofs4 Kconfig help message even if it had stayed and we would
still be here.
Ian
|
|
@raven-au well, fact is kernels starting with 4.18 have problems with any released version of systemd since systemd will try to load "autofs4" and the kmod with that name ceased to exist. The MODULES_ALIAS is not honoured by systemd, as we don't follow aliases at all. So, the rename is basically breaking all current userspace. And in my view that's a problem. And as I understand Linus' "don't break userspace" mantra this should also be considered a problem for the kernel folks. |
|
On Wed, 2018-07-04 at 06:06 -0700, Lennart Poettering wrote:
@raven-au well, fact is kernels starting with 4.18 have problems with any
released version of systemd since systemd will try to load "autofs4" and the
kmod with that name ceased to exist. The MODULES_ALIAS is not honoured by
systemd, as we don't follow aliases at all.
So, the rename is basically breaking all current userspace. And in my view
that's a problem. And as I understand Linus' "don't break userspace" mantra
this should also be considered a problem for the kernel folks.
Both Linus and myself didn't realize that systemd did this.
The remnant autofs4 stuff remaining after was ugly and was going to
need to be removed at some point so the problem would have occurred
at some point.
So what is it you would like to see happen given that 4.18 is still
in rc it could be changed?
OTOH, it isn't hard to fix and the remnants really should go at some
point, it's just wrong to carry them around IMHO.
I'm quite sure that the autofs module directory had been removed before
systemd was developed so fixing this amounts to little more than:
if autofs directory exists and autofs.ko exists load it
else if autofs4 directory exists and autofs4.ko exists load it
else error
Lennart, Linus, thoughts?
|
|
On Wed, Jul 4, 2018 at 5:23 PM Ian Kent ***@***.***> wrote:
On Wed, 2018-07-04 at 06:06 -0700, Lennart Poettering wrote:
> @raven-au well, fact is kernels starting with 4.18 have problems with any
> released version of systemd since systemd will try to load "autofs4" and the
> kmod with that name ceased to exist. The MODULES_ALIAS is not honoured by
> systemd, as we don't follow aliases at all.
Oh wow.
So I wonder what the excuse for that utter shite is _this_ time?
Both Linus and myself didn't realize that systemd did this.
Indeed. I am continually surprised by the garbage that is systemd.
"Let's replace the tools that used to work with stuff that doesn't".
It's not like the module alias stuff doesn't predate all of systemd,
and the original module tools did this properly.
What an unbelievable piece of shit.
So what is it you would like to see happen given that 4.18 is still
in rc it could be changed?
We'll make the kernel do an autofs4 module. We don't break user space.
Because unlike systemd, we actually care.
Does the attached patch "just fix it"? Can somebody who uses this check?
We can just continue to call the module "autofs4" forever, with a big
comment about why ("systemd is unbelievable garbage").
We'll make the module alias be "autofs", in the - probably useless -
hope that systemd will some day fix its bugs.
Linus
|
|
On Wed, Jul 4, 2018 at 6:17 PM Linus Torvalds ***@***.***> wrote:
We'll make the kernel do an autofs4 module. We don't break user space.
Because unlike systemd, we actually care.
Does the attached patch "just fix it"? Can somebody who uses this check?
Ok, that didn't work, because github doesn't actuall seem to react to
attachments.
So explicitly including Ben Hutchings and Michael Biebl in the email,
since they were active on the original Debian bugzilla entry, and thus
hopefully can test the kernel patch.
Trivial - but entirely untested, and this quite probably broken -
patch attached once more.
Ben, Michael? Does this make things work for you guys again?
Linus
|
|
Oh, wow. Linus being awful. Unlike on lkml we'd block him here for being like that. And just to make this clear: we are loading the kmod with libkmod, which isn't something the systemd folks invented but the official implementation of the kmod loader. Linus' remarks are hence entirely besides the point. The kmod alias stuff is usually used to match kmods to pci or usb vid/pid and similar things, not as a hack to rename specific modules. We just wanted to load four specific general purpose kmods which some distros build as module and hence we reference them by name, in particular has the alias stuff allows non-unique mappings which is not of interest for this. Anyway, if this is the level of conversation the kernel folks want then don't expect any further comment here from me on this topic. I am out on this one. |
|
On Wed, 2018-07-04 at 21:50 -0700, Lennart Poettering wrote:
Oh, wow. Linus being awful. Unlike on lkml we'd block him here for being like
that. And just to make this clear: we are loading the kmod with libkmod, which
isn't something the systemd folks invented but the official implementation of
the kmod loader. Linus' remarks are hence entirely besides the point.
The kmod alias stuff was is usually used to match kmods to pci or usb vid/pid
and similar things, not as a hack to rename specific modules. We just wanted
to load four specific general purpose kmods which some distros build as module
and hence we reference them by name, in particular has the alias stuff allows
non-unique mappings which is not of interest for this.
Anyway, if this is the level of conversation the kernel folks want then don't
expect any further comment here from me on this topic. I am out on this one.
Yeah, I've had my fair share of problems with systemd but I'm not berating
systemd, I deal with it as best I can.
More importantly this is a real problem I'd like to see fixed to the
satisfaction of everyone, just bowing out isn't going to help.
While Linus is quite harsh fairly often he has a habit of coming up
with innovative solutions to difficult problems so maybe a bit of
perseverance will be worth it.
I'm still wondering what the systemd developers think is a good approach?
Is a modification to cater for different module names doable given it would
take some time to get patches out and integrated?
Or is it going to be essential to keep something with the old name around?
Or something else completely different?
Thoughts?
|
|
On Thu, 2018-07-05 at 08:23 +0800, Ian Kent wrote:
On Wed, 2018-07-04 at 06:06 -0700, Lennart Poettering wrote:
> @raven-au well, fact is kernels starting with 4.18 have problems with any
> released version of systemd since systemd will try to load "autofs4" and the
> kmod with that name ceased to exist. The MODULES_ALIAS is not honoured by
> systemd, as we don't follow aliases at all.
> So, the rename is basically breaking all current userspace. And in my view
> that's a problem. And as I understand Linus' "don't break userspace" mantra
> this should also be considered a problem for the kernel folks.
Both Linus and myself didn't realize that systemd did this.
I've just loaded up the kernel I used to test the removal of the
autofs4 remnants.
Then I realized that my system (f27) is using systemd and autofs
as a module without any autofs4 cruft and this must have been the
case with at least one kernel build during my testing.
There may have been a message from systemd I didn't see and can't
find but the system booted, mounted binfmt_misc and the autofs
module it needs in spite of the reported problem.
So it's not accurate to say this change breaks all systemd installs
on kernels with this change.
Perhaps there's something more been done in 4.18 that I don't have
in my test kernel but that's unlikely or it's something debian
specific.
Once again, can someone give give me a reasonably detailed analysis
of what the problem is please!
If I can get this information I'll build a 4.18 rc kernel and see if
I can reproduce it on f27 too so we can properly understand what is
going on.
Or is it just the log noise that's annoying people?
Ian
|
|
On Thu, Jul 5, 2018 at 1:56 AM Ian Kent ***@***.***> wrote:
Then I realized that my system (f27) is using systemd and autofs
as a module without any autofs4 cruft and this must have been the
case with at least one kernel build during my testing.
Looking at the systemd sources, it looks like the only place that
cares about "autofs4" is kmod_setup(), which has a list of modules it
wants to load.
That code uses the kmod library - which *can* handle aliases, but you
need to actually ask it to look up the module using
kmod_module_new_from_lookup() and then walk the (possible) list of
results.
That's not what systemd does, it just does
r = kmod_module_new_from_name(...)
...
r = kmod_module_probe_insert_module(..)
and if that fails, I think you should find something like
Failed to insert module 'autofs4'
in your logs.
There may have been a message from systemd I didn't see and can't
find but the system booted, mounted binfmt_misc and the autofs
module it needs in spite of the reported problem.
I wonder if you had a later auto-module load that just worked, because
I think the only place this special "just load the name blindly
without any aliases" exists seems to be that special early
kmod_setup() phase.
I'm not even certain why systemd wants to do that early. It only does
it for a very few special modules. The comment says:
/* auto-loading on use doesn't work before udev is up */
so there seems to be some initialization dependency with udev, but I'm
not sure why udev would need autofs.
Looking through the systemd history for this code, the comment *used*
to be slightly more verbose:
/* This one we need to load explicitly, since
* auto-loading on use doesn't work before udev
* created the ghost device nodes, and we need it
* earlier than that. */
and the whole thing was added long long ago in 2010 by commit
11c3a4e ("kmod: automatically load a few kernel modules we need for
normal operation before udev is active") and before that systemd used
to just manually do "modprobe" when it started up its own automount
daemon.
Of course, the old "modprobe" way would have worked fine, since
modprobe does the whole alias thing.
Anyway, it is *possible* that the early loading is not even necessary,
and that the
m->dev_autofs_fd = open("/dev/autofs", O_CLOEXEC|O_RDONLY);
by the automount daemon just causes the module to be auto-loaded.
Maybe the udev-vs-autofs dependency doesn't exist any more? Or maybe
it only exists in Debian, not in Fedora?
Very annoying. This lack of alias handling really seems to be for a
hacky special case inside systemd. The *normal* systemd module loading
actually uses the proper kmod_module_new_from_lookup(), so it's
literally just this broken hack that really is broken garbage.
systemd even has a *convenience* function called "load_module()" that
does all that proper module name lookup for you. Of course, it's
static to src/modules-load/modules-load.c, so kmod_setup() can't use
it directly, but still.. This bug literally seems to be because of a
nasty and broken hack.
Linus
|
|
On Thu, Jul 5, 2018 at 10:35 AM Linus Torvalds ***@***.***> wrote:
Failed to insert module 'autofs4'
I've reproduced that error message - on Fedora it's hidden by default
simply because autofs is built in - and the silly compatibiility patch
fixes it for me. I don't actually have any autofs users, so I only
verified that the module is loaded, and the warning is gone.
I would suggest the systemd people fix their problem by
- making that "load_module()" helper function in
src/modules-load/modules-load.c available more widely (right now there
are basically three versions of it)
- make kmod_setup() use that function
- change the module list in kmod_table[] to have "autofs" instead of
"autofs4" as the module name, which is not only the new (and old)
name, but also allows you to test that the module alias handling works
now that you've fixed kmod_setup().
in the meantime, kernel commit d02d21ea007b ("autofs: rename 'autofs'
module back to 'autofs4'") fixes the problem with existing broken
systemd setups.
Linus
|
It turns out that systemd has a bug: it wants to load the autofs module early because of some initialization ordering with udev, and it doesn't do that correctly. Everywhere else it does the proper "look up module name" that does the proper alias resolution, but in that early code, it just uses a hardcoded "autofs4" for the module name. The result of that is that as of commit a2225d9 ("autofs: remove left-over autofs4 stubs"), you get systemd[1]: Failed to insert module 'autofs4': No such file or directory in the system logs, and a lack of module loading. All this despite the fact that we had very clearly marked 'autofs4' as an alias for this module. What's so ridiculous about this is that literally everything else does the module alias handling correctly, including really old versions of systemd (that just used 'modprobe' to do this), and even all the other systemd module loading code. Only that special systemd early module load code is broken, hardcoding the module names for not just 'autofs4', but also "ipv6", "unix", "ip_tables" and "virtio_rng". Very annoying. Instead of creating an _additional_ separate compatibility 'autofs4' module, just rely on the fact that everybody else gets this right, and just call the module 'autofs4' for compatibility reasons, with 'autofs' as the alias name. That will allow the systemd people to fix their bugs, adding the proper alias handling, and maybe even fix the name of the module to be just "autofs" (so that they can _test_ the alias handling). And eventually, we can revert this silly compatibility hack. See also systemd/systemd#9501 https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=902946 for the systemd bug reports upstream and in the Debian bug tracker respectively. Fixes: a2225d9 ("autofs: remove left-over autofs4 stubs") Reported-by: Ben Hutchings <ben@decadent.org.uk> Reported-by: Michael Biebl <biebl@debian.org> Cc: Ian Kent <raven@themaw.net> Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
|
On Thu, 2018-07-05 at 10:35 -0700, Linus Torvalds wrote:
On Thu, Jul 5, 2018 at 1:56 AM Ian Kent ***@***.***> wrote:
>
> Then I realized that my system (f27) is using systemd and autofs
> as a module without any autofs4 cruft and this must have been the
> case with at least one kernel build during my testing.
Looking at the systemd sources, it looks like the only place that
cares about "autofs4" is kmod_setup(), which has a list of modules it
wants to load.
That code uses the kmod library - which *can* handle aliases, but you
need to actually ask it to look up the module using
kmod_module_new_from_lookup() and then walk the (possible) list of
results.
That's not what systemd does, it just does
r = kmod_module_new_from_name(...)
...
r = kmod_module_probe_insert_module(..)
and if that fails, I think you should find something like
Failed to insert module 'autofs4'
in your logs.
Yes, I saw all of this and guessed there would be a log message.
I just couldn't get journalctl to find it, more likely me that's
the problem with that.
> There may have been a message from systemd I didn't see and can't
> find but the system booted, mounted binfmt_misc and the autofs
> module it needs in spite of the reported problem.
I wonder if you had a later auto-module load that just worked, because
I think the only place this special "just load the name blindly
without any aliases" exists seems to be that special early
kmod_setup() phase.
I'm not even certain why systemd wants to do that early. It only does
it for a very few special modules. The comment says:
/* auto-loading on use doesn't work before udev is up */
so there seems to be some initialization dependency with udev, but I'm
not sure why udev would need autofs.
Looking through the systemd history for this code, the comment *used*
to be slightly more verbose:
/* This one we need to load explicitly, since
* auto-loading on use doesn't work before udev
* created the ghost device nodes, and we need it
* earlier than that. */
and the whole thing was added long long ago in 2010 by commit
11c3a4e ("kmod: automatically load a few kernel modules we need for
normal operation before udev is active") and before that systemd used
to just manually do "modprobe" when it started up its own automount
daemon.
Of course, the old "modprobe" way would have worked fine, since
modprobe does the whole alias thing.
Anyway, it is *possible* that the early loading is not even necessary,
and that the
m->dev_autofs_fd = open("/dev/autofs", O_CLOEXEC|O_RDONLY);
by the automount daemon just causes the module to be auto-loaded.
Maybe the udev-vs-autofs dependency doesn't exist any more? Or maybe
it only exists in Debian, not in Fedora?
Yes, that should auto-load the module.
I didn't go nearly as far into the systemd code as you have.
I'll have a look and see if I can work out when systemd automount
starts, and try and work out if this really is a module load
problem or just log noise.
TBH I hadn't investigated the auto-loading of autofs4 for a long
time, it didn't work long ago (because of the naming) so I assumed
there would likely be some sort of problem remaining.
…
Very annoying. This lack of alias handling really seems to be for a
hacky special case inside systemd. The *normal* systemd module loading
actually uses the proper kmod_module_new_from_lookup(), so it's
literally just this broken hack that really is broken garbage.
systemd even has a *convenience* function called "load_module()" that
does all that proper module name lookup for you. Of course, it's
static to src/modules-load/modules-load.c, so kmod_setup() can't use
it directly, but still.. This bug literally seems to be because of a
nasty and broken hack.
Linus
|
This allows aliases to be used for the basic modules we load from pid1 before udev is started. In systemd#9501 the kernel renamed autofs4 to autofs, with "autofs4" as alias, but we wouldn't load the module, because we didn't follow aliases. The kernel change was reverted, but it's probably better to support aliases.
|
On Wed, 2018-07-04 at 18:25 -0700, Linus Torvalds wrote:
On Wed, Jul 4, 2018 at 6:17 PM Linus Torvalds
***@***.***> wrote:
>
> We'll make the kernel do an autofs4 module. We don't break user space.
> Because unlike systemd, we actually care.
>
> Does the attached patch "just fix it"? Can somebody who uses this check?
Ok, that didn't work, because github doesn't actuall seem to react to
attachments.
So explicitly including Ben Hutchings and Michael Biebl in the email,
since they were active on the original Debian bugzilla entry, and thus
hopefully can test the kernel patch.
Trivial - but entirely untested, and this quite probably broken -
patch attached once more.
Ben, Michael? Does this make things work for you guys again?
Works for me.
It's just a shame you felt the need to attack the systemd project in
the process.
Ben.
…--
Ben Hutchings
It is impossible to make anything foolproof
because fools are so ingenious.
|
This allows aliases to be used for the basic modules we load from pid1 before udev is started. In systemd#9501 the kernel renamed autofs4 to autofs, with "autofs4" as alias, but we wouldn't load the module, because we didn't follow aliases. The kernel change was reverted, but it's probably better to support aliases.
This allows aliases to be used for the basic modules we load from pid1 before udev is started. In systemd#9501 the kernel renamed autofs4 to autofs, with "autofs4" as alias, but we wouldn't load the module, because we didn't follow aliases. The kernel change was reverted, but it's probably better to support aliases.
This allows aliases to be used for the basic modules we load from pid1 before udev is started. In systemd#9501 the kernel renamed autofs4 to autofs, with "autofs4" as alias, but we wouldn't load the module, because we didn't follow aliases. The kernel change was reverted, but it's probably better to support aliases. (cherry picked from commit 81d7c69)
This allows aliases to be used for the basic modules we load from pid1 before udev is started. In systemd#9501 the kernel renamed autofs4 to autofs, with "autofs4" as alias, but we wouldn't load the module, because we didn't follow aliases. The kernel change was reverted, but it's probably better to support aliases. (cherry picked from commit 81d7c69)
systemd version the issue has been seen with
v239
Used distribution
Debian sid
Filed as downstream bug report https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=902946
Starting with Linux 4.18, the autofs4 kernel module is renamed
to autofs, but retaining an 'autofs4' alias.
Despite the presence of the alias, systemd fails to load the
autofs module:
Opened as 'serious' since I think autofs supports critical
functionality in systemd.
The text was updated successfully, but these errors were encountered: